Status Indicators

Monad reports a status for every pipeline and for each node inside it, so you can see at a glance whether data is flowing or something needs attention. The same set of statuses applies to a pipeline and to its nodes — a pipeline's status summarizes the state of the nodes it contains.

Status values

Monad reports a single status for a pipeline and for each node inside it, drawn from the same set of values. A pipeline's status summarizes its nodes: it shows the most significant state among them — an error on any node surfaces as Erroring — reads Running only once every enabled node is running, and reads Disabled when the whole pipeline is turned off.

• Pending — enabled, but not yet reporting a running state. This is the brief state before a node's components begin processing; it resolves to Running.
• Initializing — starting up but not yet processing data. For push-based inputs (HTTP, OTLP, Splunk HEC, syslog), ingest requests return 503 Service Unavailable with pipeline is not ready and a Retry-After header until the node is running — no data is lost. This is expected briefly after a pipeline is created or redeployed.
• Running — actively processing data. This is the desired steady state.
• Paused — not processing, but incoming edges stay open, so data buffers upstream and drains once the node is resumed. The buffer is self-limiting: as it fills, the node enters Throttled. Monad does not drop buffered data.
• Throttled — back-pressure from high data volume has halted ingestion to keep the buffer bounded. Pull-based inputs pause and resume automatically; push-based inputs receive 429 Too Many Requests. Buffered data is retained — records are lost only if the source's own retention window is exceeded before ingestion resumes, or a push client ignores the 429.
• Erroring — an error occurred while processing: an input error (credentials, misconfiguration, network, or upstream throttling), a transformation error (a wrong operation for a key, or a data-format mismatch), or an output error (a schema or format mismatch at the destination). Check the node logs for details.
• Disabled — turned off; it will not process until re-enabled. A disabled node closes its incoming edges, so data is dropped rather than buffered — the key difference from Paused, which buffers it.
• Unknown — the status could not be determined. If a status stays unknown for more than five minutes, contact support@monad.com.

Paused buffers data while Disabled drops it. For the durability and delivery guarantees behind these states — burst back-pressure, durable buffering, and no-loss failover — see Reliability & Delivery.

Sync Types

Monad input components use different synchronization methods to retrieve and process data. Understanding these sync types is important for proper expectations of pipeline behavior:

1. Incremental

• Components with incremental sync maintain state and continue data processing from where they left off in the previous run.
• Incremental syncs use techniques like timestamps, watermarks, or other tracking mechanisms to identify and process only new or changed data.
• Characteristics: More efficient resource usage, reduced data transfer, and faster processing times.
• These are commonly used in components handling continuous data streams or large datasets.

2. Full Synchronisation

• Components with full synchronisation ingest ALL of the data each time they run.
• These syncs completely refresh the destination data, replacing previous data with a fresh copy.
• Full synchronisation is usually controlled via a cron job or scheduled trigger to limit resource usage.
• Characteristics: Ensures complete data consistency but requires more resources per run.
• This approach is typically used when the source system doesn't support bookmarking features or other state-tracking mechanisms.

The sync type is an inherent property of each component and cannot be changed by the end user. Each component's documentation specifies which sync type it uses. Understanding a component's sync type helps to set appropriate expectations for data processing behavior and resource usage.

Summary

• Status: A pipeline and each of its nodes report one status — Pending, Initializing, Running, Paused, Throttled, Erroring, Disabled, or Unknown. A pipeline shows the most significant state among its nodes, and Disabled when it is turned off. Paused buffers data while Disabled drops it.
• Sync Types: (Incremental, Full Synchronisation) define how components retrieve and process data over time.

Monitoring a pipeline's status and understanding its components' sync types gives you a clear picture of its behavior and any actions you may need to take.