Troubleshooting
How to diagnose common issues in a self-managed Monad deployment. For a map of the components referenced here and how data flows between them, see Platform Architecture.
Because Monad runs as ordinary Kubernetes workloads, most issues can be diagnosed with standard kubectl commands in the namespace where Monad is installed. Throughout this page, <namespace> is the namespace Monad was installed into.
Start with pod health
A component in CrashLoopBackOff or stuck Pending usually points at the problem:
Code
Read component logs
Each service logs in structured JSON. To follow the logs for a component:
Code
Swap api for ui, operator or another component's Deployment as needed. The chart names them without the release name.
Common issues
The UI won't load or shows errors
Check the UI and API pods first, then the gateway and load balancer routing.
A pipeline won't start or update after you save it
Check the Operator logs, since it reconciles pipelines into workloads.
Also check the pipeline's Deployment and its pods. If the cluster cannot schedule a pod (for example, insufficient CPU or memory, or no matching node), that shows up here, and until the pod actually starts the pipeline's status never advances to running. Use the following to see scheduling events:
Code
A pipeline's pods are running but no data moves
Check the input's source connection and credentials. Then check the pipeline's logs, which are surfaced in the UI, or read the pod logs directly with kubectl logs for the full, unfiltered output.
Data is backing up or a pipeline shows as throttled
Use the UI to find which pipeline node is building up: its back-pressure size shows where the backlog is forming. Then check that node's metrics and pod logs to see whether it is running slowly or erroring.
Back-pressure usually originates at a slow or unavailable output and propagates upstream. See Reliability & Delivery for how this behaves and what the status indicators mean.
Push data (HTTP, OTLP, syslog) is rejected
Check the logs of the HTTP input, the L4 input (for TCP-terminated inputs like syslog), and the pipeline's own input pod for details on why data was rejected. If none of them show any logs for the traffic, the request likely never reached them, which points at routing or the gateway (including TLS).
Config changes won't save
Check the API and PostgreSQL.
Metrics, dashboards, or alerts look wrong
Check Victoria Metrics and the Alert Service. Also check the API logs for errors talking to the metrics database, since the API queries Victoria Metrics to serve the metrics shown in the UI.
Check that dependencies are healthy
If several components are failing at once, a shared dependency is often the cause. Confirm the database, message queue, and cache pods are Running:
Code
Recovering when NATS runs out of disk
NATS JetStream is the shared buffer between pipeline stages, and it stops accepting writes at about 90% of its volume. A full store shows up as many components crash-looping at once, with JetStream errors in the logs such as no response from stream, no responders available for request, or context deadline exceeded.
Confirm it with disk usage on a NATS pod and the JetStream report from the nats-box pod. These commands use the release name monad. The report needs the NATS system account. Without NATS authentication, the chart's system user is sys with the password sys:
Code
With NATS authentication, the system account has no password. Issue a credentials file for its sys user from your nsc store, pass it to the nats-box pod, and remove it afterwards:
Code
If File storage is at or near max_file_store on each server, the store is full. Recover in place rather than wiping the data, since a wipe loses your streams. Only wipe if support recommends it or you have PVC-level backups.
- Grow the NATS volumes. Helm cannot change the volume size of an existing install, so follow Grow the JetStream volume, which needs a StorageClass that allows volume expansion; if yours does not, reach out to support.
max_file_storefollows the new size, at 90% of the volume. - Restart all NATS pods so a new leader is elected and the streams catch up.
- Once healthy, the backlog returns until the pipeline throttles, so investigate the stalled stage and disable pipelines if needed.
On large stores a node can take 30 to 45 minutes to rebuild on startup and exceed the default startup probe budget of about 15 minutes. Raise it before restarting by increasing nats.container.merge.startupProbe.failureThreshold. If a stream still won't elect a leader once there is free space, capture the report and contact support before using targeted recovery commands.
To avoid this, size the NATS volume for peak load before the first install; see Size the JetStream volume before you install. Each pipeline buffers up to about 2GB before throttling and they share one store, so plan for your peak backed-up pipelines at roughly 2GB each plus headroom (around 10Gi for dev, 100Gi for small clusters, 1000Gi or more for production) and monitor store usage.
Getting help
If you can identify the affected component but not the cause, capture the relevant pod logs and reach out to Monad support at support@monad.com.