Skip to main content

Pipelines

A pipeline is the core orchestration unit in Monad. It defines how data is ingested, processed, and delivered by connecting components together into a directed graph. Every pipeline starts with a single input, flows through optional transforms and enrichments, and terminates at one or more outputs. Data routing controls how records move between components using conditional edges.

Pipeline Structure​

Pipelines are composed of two primitives: nodes and edges.

  • Nodes are instances of components (inputs, transforms, enrichments, or outputs).
  • Edges are the connections between nodes that define data flow and can include conditional routing logic.

Together they form a directed acyclic graph (DAG) — a tree-like structure where data always flows forward, never in circles.

  ┌──────────┐
  │  Input   │
  └────┬─────┘
       │
  ┌────▼──────┐
  │Transform  │
  └──┬──────┬─┘
     │      │
┌────▼───┐ ┌▼────────┐
│Output A│ │Output B │
└────────┘ └─────────┘

Component Roles​

Each node in a pipeline must follow these rules based on its position in the graph:

PositionAllowed TypesDescription
Root (no incoming edges)InputEvery pipeline must have exactly one root node, and it must be an input
Middle (incoming and outgoing edges)Transform, EnrichmentIntermediate processing nodes that receive data and pass it along
Leaf (no outgoing edges)OutputEvery branch must terminate at an output node

Pipeline Rules​

Monad validates every pipeline against the following rules before it can be deployed. Understanding these constraints will help you build valid pipelines on the first try.

Single Root Input​

Every pipeline must have exactly one root node, and it must be an input. The root node is the entry point where data enters the pipeline.

No Circular Paths​

Pipelines must be acyclic. Data always flows forward through the graph — no edges may create loops back to previously visited nodes. This ensures predictable processing and prevents infinite loops.

All Branches Must Terminate at an Output​

Every path through the pipeline must end at an output node. If you add a transform or enrichment, it must eventually connect to an output downstream.

Single Incoming Edge Per Node​

Each node can receive data from only one upstream source. However, a node can fan out to multiple downstream nodes through multiple outgoing edges. This creates a tree structure that makes data lineage easy to trace.

No Outgoing Edges From Outputs​

Output nodes are always terminal. They cannot have outgoing edges.

Node Limit​

A single pipeline supports up to 50 nodes. For workloads that require more processing steps, consider splitting the work across multiple pipelines.

Pipeline Configuration​

A pipeline configuration consists of metadata, a set of nodes, and a set of edges that connect them.

Nodes​

Each node references a component and defines its role in the pipeline:

FieldDescription
SlugUnique identifier for the node within the pipeline
Component TypeOne of: input, transform, enrichment, output
Component IDReference to the configured component
EnabledWhether this individual node is active

Edges​

Edges connect nodes and optionally define routing conditions:

FieldDescription
From NodeThe source node slug or ID
To NodeThe destination node slug or ID
ConditionsOptional conditional routing rules that filter which records pass through

For detailed documentation on configuring edge conditions, see Data Routing.

Building a Pipeline​

Basic Pipeline​

The simplest pipeline connects an input directly to an output:

  1. Add an input node to ingest data from a source
  2. Add an output node for the destination
  3. Create an edge connecting the input to the output

Adding Processing​

To transform or enrich data before delivery:

  1. Add a transform or enrichment node between the input and output
  2. Create an edge from the input to the processing node
  3. Create an edge from the processing node to the output

Fan-Out to Multiple Destinations​

To send data to multiple outputs (optionally with different filters):

  1. Start with an input node
  2. Create multiple outgoing edges to different output nodes
  3. Configure conditions on each edge to control which records go where

For example, you could route high-severity alerts to a SIEM while archiving all records to cloud storage.

Pipeline Lifecycle​

Status Indicators​

Pipelines and their individual nodes report health and operational status. For a complete guide to pipeline statuses, see Status Indicators.

Enabling and Disabling​

Pipelines can be enabled or disabled at two levels:

  • Pipeline level: Disabling a pipeline stops all data processing across all nodes
  • Node level: Individual nodes can be disabled while the rest of the pipeline continues to operate. this will lead to data building up in your pipelines stream and can cause your pipeline to throttle
  • Edge level: Individual edges can be disabled to stop data flow between specific nodes without affecting the rest of the pipeline