Metrics & Monitoring
Every Monad component exposes Prometheus metrics, and so does every pipeline pod Monad creates on your behalf. This page lists the metrics that are worth watching, explains what each one represents, and gives worked alert rules you can drop into your own monitoring stack.
For a map of the components referenced here, see Platform Architecture. For symptom-by-symptom debugging, see Troubleshooting.
How metrics are exposed
Monad services and pipeline pods serve Prometheus text-format metrics on port 2112 at /metrics. The same port serves /readyz for readiness probes.
Every pod that exposes metrics carries the standard discovery annotations:
Code
This includes the pipeline pods the Operator creates when you enable a pipeline, which are not known ahead of time and so cannot be covered by a static scrape config.
Annotation discovery is not automatic in kube-prometheus-stack
The Prometheus Operator scrapes targets described by ServiceMonitor and PodMonitor resources. It does not read prometheus.io/* annotations. To pick up Monad's pipeline pods you need either a PodMonitor that selects them, or an additionalScrapeConfigs entry that does annotation-based discovery. See Scraping Monad into your own stack below.
Scraping Monad into your own stack
The Monad chart does not create monitors for its own workloads, so you add them yourself. One PodMonitor covers the services, and a second covers the pipeline pods, which need a label selector rather than a static config because their names are generated per pipeline:
Code
Two things about this are easy to get wrong.
The release: kube-prometheus-stack label is required. By default the chart's Prometheus only selects monitors carrying its own release label, so a monitor without it is created and then silently ignored, with no error anywhere.
Selecting by port name rather than number is deliberate. A pipeline pod can run up to three containers that each expose a port named metrics, and matching on the name picks up all of them:
| Port | Container | What it serves |
|---|---|---|
2112 | The pipeline stage itself | Everything in Pipeline metrics below. |
2113 | schema-detector | Schema drift detection, present when the pipeline has it enabled. See Schema drift detection. |
2114 | network-metrics | Network byte and packet counters for the pod, split by direction and by whether the traffic left the cluster. Present on long-running stages only, not on scheduled cron-style inputs. |
The Monad chart can create monitors for its bundled dependencies as well. These are off by default so the chart installs cleanly on a cluster with no Prometheus Operator CRDs:
Code
Each of these fails the render with a clear message if you enable it without the corresponding CRD installed, rather than installing a resource that does nothing.
Labels on Monad metrics
Every metric emitted by a pipeline pod carries the full identity of the pipeline node that emitted it. These labels are what make the metrics useful, because they let you go from "throughput dropped" to "throughput dropped on this output of this pipeline in this organization" in one query.
| Label | Meaning |
|---|---|
org_id, org_name | The organization that owns the pipeline. |
pipeline_id, pipeline_name | The pipeline. |
node_id, node_slug | The individual stage within the pipeline. |
component_type | The kind of stage: input, enrichment, transform, or output. |
component_subtype | The specific connector, for example aws-cloudtrail or s3. |
component_id | The saved connector configuration the stage runs. |
component_house | The vendor the connector belongs to, where the connector has one. |
pod | The pod that emitted the sample. Pipeline stages scale horizontally, so counters are per-pod and must be summed before use. |
billing_type | On inputs and outputs, whether the volume counts as billable. |
The pod label is the one that catches people out. A pipeline stage with three replicas produces three independent counter series, so always sum() across pods before computing a rate, and never compare a raw counter value between two scrapes of a scaling deployment.
Pipeline metrics
These are emitted by every input, enrichment, transform, and output pod. They are the core of any Monad dashboard.
| Metric | Type | What it represents |
|---|---|---|
monad_status | Gauge | The current state of the stage, as a number. See the mapping below. |
monad_enabled | Gauge | 1 if the stage is enabled, 0 if not. |
monad_records_ingested_total | Counter | Records the stage has taken in. |
monad_records_published_total | Counter | Records the stage has emitted downstream, or delivered to the destination for an output. |
monad_bytes_ingested_total | Counter | Bytes taken in, measured at the size the source sent them. |
monad_bytes_published_total | Counter | Bytes emitted downstream or delivered. |
monad_records_in_flight | Gauge | Records currently being processed by the stage. |
monad_errors_total | Counter | Records that caused an error in this stage. |
monad_process_time_seconds | Histogram | Duration of one iteration of the stage's main processing loop. |
monad_pipeline_duration_seconds | Histogram | End-to-end pipeline latency, from the moment a record was ingested to the moment an output emitted it. Observed once per record, at the output stage only. |
monad_last_record_processed_time | Gauge | Unix timestamp, in seconds, of the last record the stage successfully published. |
monad_record_byte_size_ingested | Gauge | Size of the most recent record ingested. |
monad_record_byte_size_published | Gauge | Size of the most recent record published. |
monad_last_time_ingested_timestamp | Gauge | Inputs only. Unix timestamp of the last record ingested. Compare against time() to detect a source that has gone quiet. |
All four byte metrics count uncompressed record bytes. Records are compressed in transit between stages, but the counters are recorded before a record is compressed on publish and after it is decompressed on consume, so they measure the data itself rather than what goes over the wire. Expect them to read higher than the queue's size on disk or a cloud egress bill.
Reading monad_status
| Value | State | What it means |
|---|---|---|
0 | Unknown | No status reported yet. |
1 | Not ready | Created but not yet started. |
2 | Initializing | Starting up, connecting to its source or destination. |
3 | Running | Healthy. |
4 | Disabled | Deliberately turned off. Not a fault. |
5 | Erroring | Failing to process records. |
6 | Throttled | Applying back-pressure because a downstream stage or destination cannot keep up. |
Values 5 and 6 are the two worth alerting on. Throttling is not itself data loss, since Monad buffers rather than drops, but a pipeline that stays throttled for a long time can outlive the retention window of a pull-based source. See Reliability & Delivery for what these states guarantee.
Back-pressure and queue depth
Input pods observe the message queue backing each pipeline and publish its depth. These are pipeline-scoped rather than pod-scoped, so replicas write the same value idempotently and you should not sum them.
| Metric | Type | What it represents |
|---|---|---|
monad_pipeline_stream_bytes | Gauge | Total bytes buffered in the queue backing this pipeline. |
monad_pipeline_message_count | Gauge | Total messages buffered for this pipeline. |
monad_pipeline_node_message_count | Gauge | Messages pending for one specific consumer stage. Labeled with node_id and node_slug. |
monad_pipeline_node_message_count is how you find where a backlog is forming. Back-pressure propagates upstream from whichever stage is slow, so the highest pending count identifies the stage actually responsible.
Publishing to the message queue
Emitted by any stage that publishes records onward.
| Metric | Type | What it represents |
|---|---|---|
monad_batch_publisher_overflowed_messages_total | Counter | Records dropped because they exceeded the queue's maximum message size. |
monad_batch_publisher_overflowed_messages_total is the one unambiguous data-loss signal Monad emits. Any increase means records were discarded because they were individually too large to enqueue. Alert on it at a threshold of zero.
The limit is 8 MB per record, and it is applied to a single record on its own, not to the batch a stage publishes. The size counted is the uncompressed record, measured before compression, so a record that would compress under the limit is still dropped. A record over the limit is discarded rather than retried, and the drop is also written to the pipeline's log with a monad_message_too_large code, so you can find the offending records in the UI. The fix is upstream: either the source is producing oversized records, or a transform is inflating them past the limit.
Control plane metrics
API
The REST control plane behind the UI and the public API.
| Metric | Type | What it represents |
|---|---|---|
monad_api_request_duration_seconds | Histogram | HTTP request latency, labeled by method, path, code, and status. |
monad_api_request_size_bytes | Histogram | Request body size. |
monad_api_response_size_bytes | Histogram | Response body size. |
monad_api_grpc_request_duration_seconds | Histogram | gRPC request latency, labeled by grpc_method and code. |
monad_api_pipeline_cache | Counter | Pipeline config cache lookups, labeled hit. |
monad_api_permissions_cache | Counter | Permission cache lookups, labeled hit. |
The path label is the registered route pattern rather than the resolved URL, so its cardinality is bounded by the number of routes and it is safe to group by.
The API also consumes internal events and handles requests over the message queue:
| Metric | Type | What it represents |
|---|---|---|
monad_eventhandler_events_total | Counter | Events processed, labeled by handler. |
monad_eventhandler_event_errors_total | Counter | Errors raised by a handler. |
monad_eventhandler_event_duration_seconds | Histogram | Time to process one event. |
monad_eventhandler_retries_total | Counter | Retries attempted by a handler. |
monad_eventhandler_handlers | Gauge | Handlers currently active. |
monad_natsservice_nats_request_duration_seconds | Histogram | Time to serve a request over the message queue, by subject. |
monad_natsservice_nats_request_errors_total | Counter | Errors serving those requests, by subject and code. |
The pipeline-node-heartbeat-handler is the handler to watch. It is how pipeline pods report their liveness back to the control plane, so if it stops producing events while pipelines are running, the status you see in the UI stops reflecting reality.
Pipeline data ingest
The entry point for push-based inputs. HTTP, Splunk HEC, and OTLP arrive here:
| Metric | Type | What it represents |
|---|---|---|
monad_ingest_records_published_total | Counter | Records the target pipeline reported publishing, labeled by transport (http, splunk_hec, otlp_http, otlp_grpc). |
monad_ingest_forward_duration_seconds | Histogram | Time to forward a request to the pipeline's input pod and receive its reply, labeled by transport and result. |
The result label takes the values ok, throttled, disabled, invalid, unauthorized, internal, unspecified, and transport_error. Splitting on it separates a client sending bad data (invalid, unauthorized) from a pipeline under back-pressure (throttled) from a genuine platform fault (internal, transport_error).
Syslog and other TLS-terminated stream inputs arrive on a separate path:
| Metric | Type | What it represents |
|---|---|---|
monad_l4_input_connections_total | Counter | Connections handled, labeled by source (how the pipeline was identified) and result (how the connection ended). |
monad_l4_input_active_connections | Gauge | Connections currently open. |
monad_l4_input_connection_duration_seconds | Histogram | Connection lifetime. Syslog clients hold connections open for hours, so the buckets extend to a day. |
monad_l4_input_accept_errors_total | Counter | Errors accepting a connection. A connection lost here never appears in connections_total. |
monad_l4_input_records_total | Counter | Record publish attempts, labeled by result. |
monad_l4_input_record_size_bytes | Histogram | Size of individual records read off connections. |
monad_l4_input_publish_duration_seconds | Histogram | Round-trip time to publish a batch, labeled by outcome. |
The result values handshake_error, unresolved, no_config, and no_handler all mean the connection never reached a pipeline. In practice they point at TLS configuration, certificate identity, or a pipeline that is not enabled, rather than at anything wrong with the platform.
Alert service
Evaluates the alert rules you configure inside Monad.
| Metric | Type | What it represents |
|---|---|---|
monad_alerts_alert_triggered_total | Counter | Alerts fired, labeled by org_id, alert_id, and rule_type. |
monad_alerts_alert_resolved_total | Counter | Alerts resolved. |
monad_alerts_active_incidents | Gauge | Alert incidents currently open. |
monad_alerts_evaluation_duration_seconds | Histogram | Time to evaluate one rule. |
monad_alerts_evaluation_errors_total | Counter | Evaluation failures, labeled by error_type (timeout or evaluation). |
monad_alerts_last_evaluation_unix | Gauge | Unix timestamp of the last completed evaluation cycle. |
monad_alerts_alert_chan_dropped_total | Counter | Alerts dropped because the delivery channel was full. |
monad_alerts_nats_publish_errors_total | Counter | Publish failures. A non-zero value means alerts fired but were not delivered to their subscribers. |
monad_alerts_last_evaluation_unix is the health check for this service. Compare it against time(): a growing gap means the evaluation loop is stuck, and while it is stuck no Monad alert can fire, so nothing else will tell you.
Operator
Reconciles your saved pipelines into running workloads.
| Metric | Type | What it represents |
|---|---|---|
pipeline_operator_watcher_message_total | Counter | Pipeline events consumed, labeled by subject. |
pipeline_operator_watcher_error_total | Counter | Errors handling an event, labeled by subject and error_type. |
pipeline_operator_watcher_message_duration_seconds | Histogram | Time to handle one event. |
The Operator is built with controller-runtime, so it also exposes the standard controller_runtime_reconcile_total, controller_runtime_reconcile_errors_total, controller_runtime_reconcile_time_seconds, and workqueue_* metrics. A sustained high reconcile rate on a quiet cluster usually indicates a reconcile loop rather than real work.
Recommended alerts
These are the rules Monad runs against its own deployments, generalized. They are written in PromQL and work unmodified in Prometheus, Victoria Metrics, and Grafana-managed alerting.
A pipeline is throttled
Back-pressure is normal in bursts. Sustained throttling means a destination cannot keep up, and a pull-based source may age out data before the pipeline drains.
Code
Fire after 5m. Group by pipeline_name and org_name so a single struggling destination does not page once per pod.
A pipeline is erroring
Code
Fire after 10m. Below that, treat it as a transient, since a stage that recovers on its own clears the state.
Records are arriving but nothing is being delivered
The most useful pipeline alert, because it catches a stage that is up, reporting healthy, and silently doing nothing.
Code
Fire after 15m.
Delivery has gone stale
A simpler variant, keyed on the last successful publish rather than on rates. Useful for low-volume pipelines where a 15-minute rate window is too short to be meaningful.
Code
Tune the threshold to the pipeline's expected cadence. An input on an hourly schedule will trip a one-hour threshold routinely.
The error ratio is high
Code
Fire after 10m. Grouping by node_slug points straight at the stage responsible.
Records are being dropped for being too large
Unambiguous data loss. Alert at any increase.
Code
Fire immediately.
End-to-end latency has degraded
Code
Fire after 15m. Set the threshold from the freshness your downstream consumers actually need, not from a round number.
The backlog is growing without draining
Code
Fire after 30m. Requiring both a high absolute depth and a positive slope avoids paging on a large but draining backlog.
The heartbeat handler has stopped
Scope this to namespaces that actually have pipelines. Written as an unconditional absent(), it fires forever on an idle deployment.
Code
Fire after 5m.
Alert evaluation is stuck
Code
Fire after 5m. While this is firing, the alerts you configured inside Monad are not being evaluated, so this rule is the only thing covering them.
Alerts fired but were not delivered
Code
Fire immediately.
Push ingest is being rejected
Code
Fire after 10m. Keeping result in the output tells you immediately whether this is a client problem (invalid, unauthorized), back-pressure (throttled), or a platform fault (internal, transport_error).
Useful queries for dashboards
Throughput per pipeline, in records per second:
Code
Ingest volume per organization over a day, in bytes:
Code
Median and p99 end-to-end latency:
Code
Which stage is slowest, by p95 loop time:
Code
Count of pipeline stages by state, which makes a good single-panel overview:
Code
API error ratio by route:
Code
Infrastructure monitoring
The metrics on this page describe Monad, not the cluster underneath it. Most of what goes wrong in a self-managed deployment is an ordinary Kubernetes failure: a pod that will not schedule, a node under memory pressure, a volume that filled up. Monad does not ship anything that covers those.
Run kube-prometheus-stack alongside Monad. It is the reference deployment Monad tests against, and we recommend keeping the alert rules it ships with rather than writing your own. Those defaults catch the cluster-level failures above, and they will usually tell you something is wrong before any Monad metric does.
The bundled Victoria Metrics is not your observability stack
The chart deploys a small Victoria Metrics cluster. Monad services push a subset of pipeline metrics into it, and the API reads them back to render pipeline volume, throughput, and health in the UI, and to evaluate the alert rules you configure in Monad.
It exists to power product features. It has a short retention window (one month by default, victoriametrics.retentionPeriod), it does not scrape anything, and it holds none of the infrastructure metrics you need to operate a cluster. Run your own Prometheus-compatible stack alongside it and scrape the /metrics endpoints directly.
The metrics pushed into the bundled store are the pipeline throughput and health counters (monad_records_*, monad_bytes_*, monad_errors_total, monad_status, monad_last_record_processed_time, monad_record_byte_size_*), plus the batch publisher and back-pressure gauges. Everything else is scrape-only.
Getting help
If a metric is telling you something is wrong but you cannot work out what, capture the relevant queries and pod logs and reach out to Monad support at support@monad.com.