Copy page

Core Concepts

What are Pipelines?

A pipeline is the core orchestration unit in Monad. It defines how data is ingested, processed, and delivered by connecting components together into a directed graph. Every pipeline starts with a single input, flows through optional transforms and enrichments, and terminates at one or more outputs. Data routing controls how records move between components using conditional edges.

This guide is a conceptual tour of what a pipeline is made of and how data moves through it. Each component has its own reference documentation, linked inline.

Pipeline Structure

Pipelines are composed of two primitives: nodes and edges.

• Nodes are instances of components (inputs, transforms, enrichments, or outputs).
• Edges are the connections between nodes that define data flow and can include conditional routing logic.

Together they form a directed acyclic graph (DAG) — a tree-like structure where data always flows forward, never in circles.

Code
  ┌──────────┐
  │  Input   │
  └────┬─────┘
       │
  ┌────▼──────┐
  │Transform  │
  └──┬──────┬─┘
     │      │
┌────▼───┐ ┌▼────────┐
│Output A│ │Output B │
└────────┘ └─────────┘

Component Roles

Each node in a pipeline must follow these rules based on its position in the graph:

Position	Allowed Types	Description
Root (no incoming edges)	Input	Every pipeline must have exactly one root node, and it must be an input
Middle (incoming and outgoing edges)	Transform, Enrichment	Intermediate processing nodes that receive data and pass it along
Leaf (no outgoing edges)	Output	Every branch must terminate at an output node

The components of a pipeline

Every node in a pipeline is one of a few component types, each with a distinct job:

• Inputs are the entry point — each connects to a specific upstream source and pulls its data into the pipeline, handling authentication and initial formatting. See Inputs.
• Transforms reshape each record as it passes through, one record at a time — cleaning, normalizing, and restructuring it. See Transforms.
• Enrichments add context to a record mid-flight: they look up a value (a join key) and write the result back onto the record. See Enrichments.
• Conditionals and routing decide which records travel down which edge, based on their content. See Data Routing and Conditionals.
• Outputs are the exit point — each delivers processed records to a downstream destination such as a SIEM, data lake, or object store. See Outputs.

For enrichment, normalization, and routing in more depth, see Enrichment, Normalization, and Routing.

How data flows through a pipeline

Data always flows forward through the graph. A node has a single incoming edge but can fan out to multiple downstream nodes, which gives a pipeline its tree shape and makes data lineage easy to trace.

Every outgoing edge is evaluated for every record, independently of the others — so a single record can match multiple edges and be delivered to several destinations at once, for example routing high-severity findings to a SIEM while archiving everything to cloud storage.

Pipeline Rules

Monad validates every pipeline against the following rules before it can be deployed. Understanding these constraints will help you build valid pipelines on the first try.

Single Root Input

Every pipeline must have exactly one root node, and it must be an input. The root node is the entry point where data enters the pipeline.

No Circular Paths

Pipelines must be acyclic. Data always flows forward through the graph — no edges may create loops back to previously visited nodes. This ensures predictable processing and prevents infinite loops.

All Branches Must Terminate at an Output

Every path through the pipeline must end at an output node. If you add a transform or enrichment, it must eventually connect to an output downstream.

Single Incoming Edge Per Node

Each node can receive data from only one upstream source. However, a node can fan out to multiple downstream nodes through multiple outgoing edges. This creates a tree structure that makes data lineage easy to trace.

No Outgoing Edges From Outputs

Output nodes are always terminal. They cannot have outgoing edges.

Node Limit

A single pipeline supports up to 50 nodes. For workloads that require more processing steps, consider splitting the work across multiple pipelines.

Node and pipeline states

Monad reports a single status for a pipeline and for each node inside it, drawn from the same set of values. A pipeline's status summarizes its nodes: it shows the most significant state among them — an error on any node surfaces as Erroring — reads Running only once every enabled node is running, and reads Disabled when the whole pipeline is turned off.

• Pending — enabled, but not yet reporting a running state. This is the brief state before a node's components begin processing; it resolves to Running.
• Initializing — starting up but not yet processing data. For push-based inputs (HTTP, OTLP, Splunk HEC, syslog), ingest requests return 503 Service Unavailable with pipeline is not ready and a Retry-After header until the node is running — no data is lost. This is expected briefly after a pipeline is created or redeployed.
• Running — actively processing data. This is the desired steady state.
• Paused — not processing, but incoming edges stay open, so data buffers upstream and drains once the node is resumed. The buffer is self-limiting: as it fills, the node enters Throttled. Monad does not drop buffered data.
• Throttled — back-pressure from high data volume has halted ingestion to keep the buffer bounded. Pull-based inputs pause and resume automatically; push-based inputs receive 429 Too Many Requests. Buffered data is retained — records are lost only if the source's own retention window is exceeded before ingestion resumes, or a push client ignores the 429.
• Erroring — an error occurred while processing: an input error (credentials, misconfiguration, network, or upstream throttling), a transformation error (a wrong operation for a key, or a data-format mismatch), or an output error (a schema or format mismatch at the destination). Check the node logs for details.
• Disabled — turned off; it will not process until re-enabled. A disabled node closes its incoming edges, so data is dropped rather than buffered — the key difference from Paused, which buffers it.
• Unknown — the status could not be determined. If a status stays unknown for more than five minutes, contact support@monad.com.

Paused buffers data while Disabled drops it. For the durability and delivery guarantees behind these states — burst back-pressure, durable buffering, and no-loss failover — see Reliability & Delivery.

The full reference, including sync types, is on the Pipeline Status Indicators page.

Related

• Pipelines
• Inputs · Transforms · Enrichments · Outputs
• Data Routing · Conditionals
• Enrichment, Normalization, and Routing
• Monitoring and Alerting
• Pipeline Status Indicators