# Pipelines

A pipeline is the core orchestration unit in Monad. It defines how data is ingested, processed, and delivered by connecting components together into a directed graph. Every pipeline starts with a single [input](/docs/inputs), flows through optional [transforms](/docs/transforms) and [enrichments](/docs/enrichments), and terminates at one or more [outputs](/docs/outputs). [Data routing](/docs/routing) controls how records move between components using conditional edges.

## Pipeline Structure

Pipelines are composed of two primitives: **nodes** and **edges**.

- **Nodes** are instances of components (inputs, transforms, enrichments, or outputs).
- **Edges** are the connections between nodes that define data flow and can include [conditional routing logic](/docs/conditionals).

Together they form a **directed acyclic graph (DAG)** — a tree-like structure where data always flows forward, never in circles.

```
  ┌──────────┐
  │  Input   │
  └────┬─────┘
       │
  ┌────▼──────┐
  │Transform  │
  └──┬──────┬─┘
     │      │
┌────▼───┐ ┌▼────────┐
│Output A│ │Output B │
└────────┘ └─────────┘
```

## Component Roles

Each node in a pipeline must follow these rules based on its position in the graph:

| Position | Allowed Types | Description |
|----------|--------------|-------------|
| **Root** (no incoming edges) | [Input](/docs/inputs) | Every pipeline must have exactly one root node, and it must be an input |
| **Middle** (incoming and outgoing edges) | [Transform](/docs/transforms), [Enrichment](/docs/enrichments) | Intermediate processing nodes that receive data and pass it along |
| **Leaf** (no outgoing edges) | [Output](/docs/outputs) | Every branch must terminate at an output node |

## Pipeline Rules

Monad validates every pipeline against the following rules before it can be deployed. Understanding these constraints will help you build valid pipelines on the first try.

### Single Root Input

Every pipeline must have exactly **one root node**, and it must be an [input](/docs/inputs). The root node is the entry point where data enters the pipeline.

### No Circular Paths

Pipelines must be acyclic. Data always flows forward through the graph — no edges may create loops back to previously visited nodes. This ensures predictable processing and prevents infinite loops.

### All Branches Must Terminate at an Output

Every path through the pipeline must end at an [output](/docs/outputs) node. If you add a transform or enrichment, it must eventually connect to an output downstream.

### Single Incoming Edge Per Node

Each node can receive data from only **one** upstream source. However, a node can fan out to **multiple** downstream nodes through multiple outgoing edges. This creates a tree structure that makes data lineage easy to trace.

### No Outgoing Edges From Outputs

[Output](/docs/outputs) nodes are always terminal. They cannot have outgoing edges.

### Node Limit

A single pipeline supports up to **50 nodes**. For workloads that require more processing steps, consider splitting the work across multiple pipelines.

## Pipeline Configuration

A pipeline configuration consists of metadata, a set of nodes, and a set of edges that connect them.

### Nodes

Each node references a component and defines its role in the pipeline:

| Field | Description |
|-------|-------------|
| **Slug** | Unique identifier for the node within the pipeline |
| **Component Type** | One of: `input`, `transform`, `enrichment`, `output` |
| **Component ID** | Reference to the configured component |
| **Enabled** | Whether this individual node is active |

### Edges

Edges connect nodes and optionally define routing conditions:

| Field | Description |
|-------|-------------|
| **From Node** | The source node slug or ID |
| **To Node** | The destination node slug or ID |
| **Conditions** | Optional [conditional routing](/docs/routing) rules that filter which records pass through |

For detailed documentation on configuring edge conditions, see [Data Routing](/docs/routing).

## Building a Pipeline

### Basic Pipeline

The simplest pipeline connects an input directly to an output:

1. Add an **input** node to ingest data from a source
2. Add an **output** node for the destination
3. Create an edge connecting the input to the output

### Adding Processing

To transform or enrich data before delivery:

1. Add a **transform** or **enrichment** node between the input and output
2. Create an edge from the input to the processing node
3. Create an edge from the processing node to the output

### Fan-Out to Multiple Destinations

To send data to multiple outputs (optionally with different filters):

1. Start with an input node
2. Create multiple outgoing edges to different output nodes
3. Configure [conditions](/docs/conditionals) on each edge to control which records go where

For example, you could route high-severity alerts to a SIEM while archiving all records to cloud storage.

## Pipeline Lifecycle

### Status Indicators

Pipelines and their individual nodes report health and operational status. For a complete guide to pipeline statuses, see [Status Indicators](/docs/pipeline_status_indicators).

### Enabling, Pausing, and Disabling

Pipelines can be controlled at three levels:

- **Pipeline level**: Disabling a pipeline stops all data processing across all nodes.
- **Node level**: Each non-input node has a **Status** of `Enabled`, `Paused`, or `Disabled`, set from the node's detail panel in the pipeline editor:
  - **Enabled** — the node processes data normally.
  - **Paused** — the node stops processing, but its incoming edges stay open, so **data buffers upstream** and is delivered once the node is re-enabled. Use `Paused` when you want to hold data and replay it later. The buffer is bounded: as it fills, the pipeline throttles (halting input ingestion), so it cannot grow until disk is exhausted, and buffered data is retained rather than dropped. Data is only lost if a node is left paused longer than the source retains it — see [Status Indicators](/docs/pipeline_status_indicators).
  - **Disabled** — the node's incoming edges are closed, so **data is dropped** instead of buffered. On re-enable, the node resumes from live data with no backlog. Use `Disabled` when you want to stop a node without accumulating a backlog. (Not available for input nodes, which have no incoming edges.)
- **Edge level**: Individual edges can be disabled to stop data flow between specific nodes without affecting the rest of the pipeline.

## Related Documentation

- [Inputs](/docs/inputs) - Data ingestion components
- [Transforms](/docs/transforms) - Data processing and manipulation
- [Enrichments](/docs/enrichments) - Contextual data enhancement
- [Outputs](/docs/outputs) - Data delivery destinations
- [Data Routing](/docs/routing) - Conditional edge configuration
- [Conditionals](/docs/conditionals) - Condition types and operators
- [Status Indicators](/docs/pipeline_status_indicators) - Pipeline health monitoring