# Metrics & Monitoring

Every Monad component exposes Prometheus metrics, and so does every pipeline pod Monad creates on your behalf. This page lists the metrics that are worth watching, explains what each one represents, and gives worked alert rules you can drop into your own monitoring stack.

For a map of the components referenced here, see [Platform Architecture](/docs/guides/architecture). For symptom-by-symptom debugging, see [Troubleshooting](/docs/guides/troubleshooting).

## How metrics are exposed

Monad services and pipeline pods serve Prometheus text-format metrics on **port `2112`** at **`/metrics`**. The same port serves `/readyz` for readiness probes.

Every pod that exposes metrics carries the standard discovery annotations:

```yaml
prometheus.io/scrape: "true"
prometheus.io/path: /metrics
prometheus.io/port: metrics
```

This includes the pipeline pods the Operator creates when you enable a pipeline, which are not known ahead of time and so cannot be covered by a static scrape config.

:::info[Annotation discovery is not automatic in kube-prometheus-stack]
The Prometheus Operator scrapes targets described by `ServiceMonitor` and `PodMonitor` resources. It does **not** read `prometheus.io/*` annotations. To pick up Monad's pipeline pods you need either a `PodMonitor` that selects them, or an `additionalScrapeConfigs` entry that does annotation-based discovery. See [Scraping Monad into your own stack](#scraping-monad-into-your-own-stack) below.
:::

## Scraping Monad into your own stack

The Monad chart does not create monitors for its own workloads, so you add them yourself. One `PodMonitor` covers the services, and a second covers the pipeline pods, which need a label selector rather than a static config because their names are generated per pipeline:

```yaml
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: monad-pipelines
  namespace: <namespace>
  labels:
    release: kube-prometheus-stack
spec:
  namespaceSelector:
    matchNames:
      - <namespace>
  selector:
    matchExpressions:
      - key: monad.com/pipeline-id
        operator: Exists
  podMetricsEndpoints:
    - port: metrics
      path: /metrics
```

Two things about this are easy to get wrong.

The `release: kube-prometheus-stack` label is required. By default the chart's Prometheus only selects monitors carrying its own release label, so a monitor without it is created and then silently ignored, with no error anywhere.

Selecting by port **name** rather than number is deliberate. A pipeline pod can run up to three containers that each expose a port named `metrics`, and matching on the name picks up all of them:

| Port | Container | What it serves |
| --- | --- | --- |
| `2112` | The pipeline stage itself | Everything in [Pipeline metrics](#pipeline-metrics) below. |
| `2113` | `schema-detector` | Schema drift detection, present when the pipeline has it enabled. See [Schema drift detection](/docs/guides/schema-detection). |
| `2114` | `network-metrics` | Network byte and packet counters for the pod, split by direction and by whether the traffic left the cluster. Present on long-running stages only, not on scheduled cron-style inputs. |

The Monad chart can create monitors for its bundled dependencies as well. These are off by default so the chart installs cleanly on a cluster with no Prometheus Operator CRDs:

```yaml
victoriametrics:
  monitoring:
    enabled: true    # ServiceMonitors for vminsert, vmselect, vmstorage
postgresql:
  cnpg:
    monitoring:
      enabled: true  # PodMonitor for the PostgreSQL cluster
dragonfly:
  serviceMonitor:
    enabled: true    # ServiceMonitor for the cache
```

Each of these fails the render with a clear message if you enable it without the corresponding CRD installed, rather than installing a resource that does nothing.

## Labels on Monad metrics

Every metric emitted by a pipeline pod carries the full identity of the pipeline node that emitted it. These labels are what make the metrics useful, because they let you go from "throughput dropped" to "throughput dropped on this output of this pipeline in this organization" in one query.

| Label | Meaning |
| --- | --- |
| `org_id`, `org_name` | The organization that owns the pipeline. |
| `pipeline_id`, `pipeline_name` | The pipeline. |
| `node_id`, `node_slug` | The individual stage within the pipeline. |
| `component_type` | The kind of stage: `input`, `enrichment`, `transform`, or `output`. |
| `component_subtype` | The specific connector, for example `aws-cloudtrail` or `s3`. |
| `component_id` | The saved connector configuration the stage runs. |
| `component_house` | The vendor the connector belongs to, where the connector has one. |
| `pod` | The pod that emitted the sample. Pipeline stages scale horizontally, so counters are per-pod and must be summed before use. |
| `billing_type` | On inputs and outputs, whether the volume counts as billable. |

The `pod` label is the one that catches people out. A pipeline stage with three replicas produces three independent counter series, so always `sum()` across pods before computing a rate, and never compare a raw counter value between two scrapes of a scaling deployment.

## Pipeline metrics

These are emitted by every input, enrichment, transform, and output pod. They are the core of any Monad dashboard.

| Metric | Type | What it represents |
| --- | --- | --- |
| `monad_status` | Gauge | The current state of the stage, as a number. See the mapping below. |
| `monad_enabled` | Gauge | `1` if the stage is enabled, `0` if not. |
| `monad_records_ingested_total` | Counter | Records the stage has taken in. |
| `monad_records_published_total` | Counter | Records the stage has emitted downstream, or delivered to the destination for an output. |
| `monad_bytes_ingested_total` | Counter | Bytes taken in, measured at the size the source sent them. |
| `monad_bytes_published_total` | Counter | Bytes emitted downstream or delivered. |
| `monad_records_in_flight` | Gauge | Records currently being processed by the stage. |
| `monad_errors_total` | Counter | Records that caused an error in this stage. |
| `monad_process_time_seconds` | Histogram | Duration of one iteration of the stage's main processing loop. |
| `monad_pipeline_duration_seconds` | Histogram | End-to-end pipeline latency, from the moment a record was ingested to the moment an output emitted it. Observed once per record, at the output stage only. |
| `monad_last_record_processed_time` | Gauge | Unix timestamp, in seconds, of the last record the stage successfully published. |
| `monad_record_byte_size_ingested` | Gauge | Size of the most recent record ingested. |
| `monad_record_byte_size_published` | Gauge | Size of the most recent record published. |
| `monad_last_time_ingested_timestamp` | Gauge | Inputs only. Unix timestamp of the last record ingested. Compare against `time()` to detect a source that has gone quiet. |

All four byte metrics count **uncompressed** record bytes. Records are compressed in transit between stages, but the counters are recorded before a record is compressed on publish and after it is decompressed on consume, so they measure the data itself rather than what goes over the wire. Expect them to read higher than the queue's size on disk or a cloud egress bill.

### Reading `monad_status`

| Value | State | What it means |
| --- | --- | --- |
| `0` | Unknown | No status reported yet. |
| `1` | Not ready | Created but not yet started. |
| `2` | Initializing | Starting up, connecting to its source or destination. |
| `3` | Running | Healthy. |
| `4` | Disabled | Deliberately turned off. Not a fault. |
| `5` | Erroring | Failing to process records. |
| `6` | Throttled | Applying back-pressure because a downstream stage or destination cannot keep up. |

Values `5` and `6` are the two worth alerting on. Throttling is not itself data loss, since Monad buffers rather than drops, but a pipeline that stays throttled for a long time can outlive the retention window of a pull-based source. See [Reliability & Delivery](/docs/guides/reliability) for what these states guarantee.

### Back-pressure and queue depth

Input pods observe the message queue backing each pipeline and publish its depth. These are pipeline-scoped rather than pod-scoped, so replicas write the same value idempotently and you should not sum them.

| Metric | Type | What it represents |
| --- | --- | --- |
| `monad_pipeline_stream_bytes` | Gauge | Total bytes buffered in the queue backing this pipeline. |
| `monad_pipeline_message_count` | Gauge | Total messages buffered for this pipeline. |
| `monad_pipeline_node_message_count` | Gauge | Messages pending for one specific consumer stage. Labeled with `node_id` and `node_slug`. |

`monad_pipeline_node_message_count` is how you find *where* a backlog is forming. Back-pressure propagates upstream from whichever stage is slow, so the highest pending count identifies the stage actually responsible.

### Publishing to the message queue

Emitted by any stage that publishes records onward.

| Metric | Type | What it represents |
| --- | --- | --- |
| `monad_batch_publisher_overflowed_messages_total` | Counter | Records dropped because they exceeded the queue's maximum message size. |

`monad_batch_publisher_overflowed_messages_total` is the one unambiguous data-loss signal Monad emits. Any increase means records were discarded because they were individually too large to enqueue. Alert on it at a threshold of zero.

The limit is **8 MB per record**, and it is applied to a single record on its own, not to the batch a stage publishes. The size counted is the uncompressed record, measured before compression, so a record that would compress under the limit is still dropped. A record over the limit is discarded rather than retried, and the drop is also written to the pipeline's log with a `monad_message_too_large` code, so you can find the offending records in the UI. The fix is upstream: either the source is producing oversized records, or a transform is inflating them past the limit.

## Control plane metrics

### API

The REST control plane behind the UI and the public API.

| Metric | Type | What it represents |
| --- | --- | --- |
| `monad_api_request_duration_seconds` | Histogram | HTTP request latency, labeled by `method`, `path`, `code`, and `status`. |
| `monad_api_request_size_bytes` | Histogram | Request body size. |
| `monad_api_response_size_bytes` | Histogram | Response body size. |
| `monad_api_grpc_request_duration_seconds` | Histogram | gRPC request latency, labeled by `grpc_method` and `code`. |
| `monad_api_pipeline_cache` | Counter | Pipeline config cache lookups, labeled `hit`. |
| `monad_api_permissions_cache` | Counter | Permission cache lookups, labeled `hit`. |

The `path` label is the registered route pattern rather than the resolved URL, so its cardinality is bounded by the number of routes and it is safe to group by.

The API also consumes internal events and handles requests over the message queue:

| Metric | Type | What it represents |
| --- | --- | --- |
| `monad_eventhandler_events_total` | Counter | Events processed, labeled by `handler`. |
| `monad_eventhandler_event_errors_total` | Counter | Errors raised by a handler. |
| `monad_eventhandler_event_duration_seconds` | Histogram | Time to process one event. |
| `monad_eventhandler_retries_total` | Counter | Retries attempted by a handler. |
| `monad_eventhandler_handlers` | Gauge | Handlers currently active. |
| `monad_natsservice_nats_request_duration_seconds` | Histogram | Time to serve a request over the message queue, by `subject`. |
| `monad_natsservice_nats_request_errors_total` | Counter | Errors serving those requests, by `subject` and `code`. |

The `pipeline-node-heartbeat-handler` is the handler to watch. It is how pipeline pods report their liveness back to the control plane, so if it stops producing events while pipelines are running, the status you see in the UI stops reflecting reality.

### Pipeline data ingest

The entry point for push-based inputs. HTTP, Splunk HEC, and OTLP arrive here:

| Metric | Type | What it represents |
| --- | --- | --- |
| `monad_ingest_records_published_total` | Counter | Records the target pipeline reported publishing, labeled by `transport` (`http`, `splunk_hec`, `otlp_http`, `otlp_grpc`). |
| `monad_ingest_forward_duration_seconds` | Histogram | Time to forward a request to the pipeline's input pod and receive its reply, labeled by `transport` and `result`. |

The `result` label takes the values `ok`, `throttled`, `disabled`, `invalid`, `unauthorized`, `internal`, `unspecified`, and `transport_error`. Splitting on it separates a client sending bad data (`invalid`, `unauthorized`) from a pipeline under back-pressure (`throttled`) from a genuine platform fault (`internal`, `transport_error`).

Syslog and other TLS-terminated stream inputs arrive on a separate path:

| Metric | Type | What it represents |
| --- | --- | --- |
| `monad_l4_input_connections_total` | Counter | Connections handled, labeled by `source` (how the pipeline was identified) and `result` (how the connection ended). |
| `monad_l4_input_active_connections` | Gauge | Connections currently open. |
| `monad_l4_input_connection_duration_seconds` | Histogram | Connection lifetime. Syslog clients hold connections open for hours, so the buckets extend to a day. |
| `monad_l4_input_accept_errors_total` | Counter | Errors accepting a connection. A connection lost here never appears in `connections_total`. |
| `monad_l4_input_records_total` | Counter | Record publish attempts, labeled by `result`. |
| `monad_l4_input_record_size_bytes` | Histogram | Size of individual records read off connections. |
| `monad_l4_input_publish_duration_seconds` | Histogram | Round-trip time to publish a batch, labeled by outcome. |

The `result` values `handshake_error`, `unresolved`, `no_config`, and `no_handler` all mean the connection never reached a pipeline. In practice they point at TLS configuration, certificate identity, or a pipeline that is not enabled, rather than at anything wrong with the platform.

### Alert service

Evaluates the alert rules you configure inside Monad.

| Metric | Type | What it represents |
| --- | --- | --- |
| `monad_alerts_alert_triggered_total` | Counter | Alerts fired, labeled by `org_id`, `alert_id`, and `rule_type`. |
| `monad_alerts_alert_resolved_total` | Counter | Alerts resolved. |
| `monad_alerts_active_incidents` | Gauge | Alert incidents currently open. |
| `monad_alerts_evaluation_duration_seconds` | Histogram | Time to evaluate one rule. |
| `monad_alerts_evaluation_errors_total` | Counter | Evaluation failures, labeled by `error_type` (`timeout` or `evaluation`). |
| `monad_alerts_last_evaluation_unix` | Gauge | Unix timestamp of the last completed evaluation cycle. |
| `monad_alerts_alert_chan_dropped_total` | Counter | Alerts dropped because the delivery channel was full. |
| `monad_alerts_nats_publish_errors_total` | Counter | Publish failures. A non-zero value means alerts fired but were not delivered to their subscribers. |

`monad_alerts_last_evaluation_unix` is the health check for this service. Compare it against `time()`: a growing gap means the evaluation loop is stuck, and while it is stuck no Monad alert can fire, so nothing else will tell you.

### Operator

Reconciles your saved pipelines into running workloads.

| Metric | Type | What it represents |
| --- | --- | --- |
| `pipeline_operator_watcher_message_total` | Counter | Pipeline events consumed, labeled by `subject`. |
| `pipeline_operator_watcher_error_total` | Counter | Errors handling an event, labeled by `subject` and `error_type`. |
| `pipeline_operator_watcher_message_duration_seconds` | Histogram | Time to handle one event. |

The Operator is built with controller-runtime, so it also exposes the standard `controller_runtime_reconcile_total`, `controller_runtime_reconcile_errors_total`, `controller_runtime_reconcile_time_seconds`, and `workqueue_*` metrics. A sustained high reconcile rate on a quiet cluster usually indicates a reconcile loop rather than real work.

## Recommended alerts

These are the rules Monad runs against its own deployments, generalized. They are written in PromQL and work unmodified in Prometheus, Victoria Metrics, and Grafana-managed alerting.

### A pipeline is throttled

Back-pressure is normal in bursts. Sustained throttling means a destination cannot keep up, and a pull-based source may age out data before the pipeline drains.

```promql
monad_status == 6
```

Fire after `5m`. Group by `pipeline_name` and `org_name` so a single struggling destination does not page once per pod.

### A pipeline is erroring

```promql
monad_status == 5
```

Fire after `10m`. Below that, treat it as a transient, since a stage that recovers on its own clears the state.

### Records are arriving but nothing is being delivered

The most useful pipeline alert, because it catches a stage that is up, reporting healthy, and silently doing nothing.

```promql
  sum by (pipeline_id, pipeline_name, org_name) (
    rate(monad_records_ingested_total{component_type="input"}[15m])
  ) > 0
unless
  sum by (pipeline_id, pipeline_name, org_name) (
    rate(monad_records_published_total{component_type="output"}[15m])
  ) > 0
```

Fire after `15m`.

### Delivery has gone stale

A simpler variant, keyed on the last successful publish rather than on rates. Useful for low-volume pipelines where a 15-minute rate window is too short to be meaningful.

```promql
time() - max by (pipeline_id, pipeline_name) (
  monad_last_record_processed_time{component_type="output"}
) > 3600
```

Tune the threshold to the pipeline's expected cadence. An input on an hourly schedule will trip a one-hour threshold routinely.

### The error ratio is high

```promql
  sum by (pipeline_id, pipeline_name, node_slug) (rate(monad_errors_total[10m]))
/
  sum by (pipeline_id, pipeline_name, node_slug) (rate(monad_records_ingested_total[10m]))
> 0.05
```

Fire after `10m`. Grouping by `node_slug` points straight at the stage responsible.

### Records are being dropped for being too large

Unambiguous data loss. Alert at any increase.

```promql
increase(monad_batch_publisher_overflowed_messages_total[10m]) > 0
```

Fire immediately.

### End-to-end latency has degraded

```promql
histogram_quantile(0.99,
  sum by (le, pipeline_id, pipeline_name) (
    rate(monad_pipeline_duration_seconds_bucket[10m])
  )
) > 300
```

Fire after `15m`. Set the threshold from the freshness your downstream consumers actually need, not from a round number.

### The backlog is growing without draining

```promql
  sum by (pipeline_id, pipeline_name) (monad_pipeline_message_count) > 100000
and
  deriv(sum by (pipeline_id, pipeline_name) (monad_pipeline_message_count)[30m:1m]) > 0
```

Fire after `30m`. Requiring both a high absolute depth and a positive slope avoids paging on a large but draining backlog.

### The heartbeat handler has stopped

Scope this to namespaces that actually have pipelines. Written as an unconditional `absent()`, it fires forever on an idle deployment.

```promql
  count by (namespace) (monad_status) > 0
unless
  count by (namespace) (monad_eventhandler_events_total{handler="pipeline-node-heartbeat-handler"})
```

Fire after `5m`.

### Alert evaluation is stuck

```promql
time() - max by (org_id) (monad_alerts_last_evaluation_unix) > 900
```

Fire after `5m`. While this is firing, the alerts you configured inside Monad are not being evaluated, so this rule is the only thing covering them.

### Alerts fired but were not delivered

```promql
increase(monad_alerts_nats_publish_errors_total[10m]) > 0
```

Fire immediately.

### Push ingest is being rejected

```promql
  sum by (transport, result) (
    rate(monad_ingest_forward_duration_seconds_count{result!="ok"}[5m])
  )
/ ignoring(result) group_left
  sum by (transport) (rate(monad_ingest_forward_duration_seconds_count[5m]))
> 0.1
```

Fire after `10m`. Keeping `result` in the output tells you immediately whether this is a client problem (`invalid`, `unauthorized`), back-pressure (`throttled`), or a platform fault (`internal`, `transport_error`).

## Useful queries for dashboards

Throughput per pipeline, in records per second:

```promql
sum by (pipeline_name, component_type) (rate(monad_records_published_total[5m]))
```

Ingest volume per organization over a day, in bytes:

```promql
sum by (org_name) (increase(monad_bytes_ingested_total{component_type="input"}[1d]))
```

Median and p99 end-to-end latency:

```promql
histogram_quantile(0.50, sum by (le) (rate(monad_pipeline_duration_seconds_bucket[5m])))
histogram_quantile(0.99, sum by (le) (rate(monad_pipeline_duration_seconds_bucket[5m])))
```

Which stage is slowest, by p95 loop time:

```promql
topk(10,
  histogram_quantile(0.95,
    sum by (le, pipeline_name, node_slug) (rate(monad_process_time_seconds_bucket[5m]))
  )
)
```

Count of pipeline stages by state, which makes a good single-panel overview:

```promql
count by (component_type) (monad_status == 3)
count by (component_type) (monad_status == 5)
count by (component_type) (monad_status == 6)
```

API error ratio by route:

```promql
  sum by (path) (rate(monad_api_request_duration_seconds_count{code=~"5.."}[5m]))
/ sum by (path) (rate(monad_api_request_duration_seconds_count[5m]))
```

## Infrastructure monitoring

The metrics on this page describe Monad, not the cluster underneath it. Most of what goes wrong in a self-managed deployment is an ordinary Kubernetes failure: a pod that will not schedule, a node under memory pressure, a volume that filled up. Monad does not ship anything that covers those.

Run [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) alongside Monad. It is the reference deployment Monad tests against, and we recommend keeping the alert rules it ships with rather than writing your own. Those defaults catch the cluster-level failures above, and they will usually tell you something is wrong before any Monad metric does.

## The bundled Victoria Metrics is not your observability stack

The chart deploys a small Victoria Metrics cluster. Monad services **push** a subset of pipeline metrics into it, and the API reads them back to render pipeline volume, throughput, and health in the UI, and to evaluate the alert rules you configure in Monad.

It exists to power product features. It has a short retention window (one month by default, `victoriametrics.retentionPeriod`), it does not scrape anything, and it holds none of the infrastructure metrics you need to operate a cluster. Run your own Prometheus-compatible stack alongside it and scrape the `/metrics` endpoints directly.

The metrics pushed into the bundled store are the pipeline throughput and health counters (`monad_records_*`, `monad_bytes_*`, `monad_errors_total`, `monad_status`, `monad_last_record_processed_time`, `monad_record_byte_size_*`), plus the batch publisher and back-pressure gauges. Everything else is scrape-only.

## Getting help

If a metric is telling you something is wrong but you cannot work out what, capture the relevant queries and pod logs and reach out to Monad support at support@monad.com.