Troubleshooting
How to diagnose common issues in a self-managed Monad deployment. For a map of the components referenced here and how data flows between them, see Platform Architecture.
Because Monad runs as ordinary Kubernetes workloads, most issues can be diagnosed with standard kubectl commands in the namespace where Monad is installed. Throughout this page, <namespace> is the namespace Monad was installed into.
Start with pod health
A component in CrashLoopBackOff or stuck Pending usually points at the problem:
Code
Read component logs
Each service logs in structured JSON. To follow the logs for a component:
Code
Swap monad-api for monad-ui, the operator, or another Deployment as needed.
Common issues
The UI won't load or shows errors
Check the UI and API pods first, then the gateway and load balancer routing.
A pipeline won't start or update after you save it
Check the Operator logs, since it reconciles pipelines into workloads.
Also check the pipeline's Deployment and its pods. If the cluster cannot schedule a pod (for example, insufficient CPU or memory, or no matching node), that shows up here, and until the pod actually starts the pipeline's status never advances to running. Use the following to see scheduling events:
Code
A pipeline's pods are running but no data moves
Check the input's source connection and credentials. Then check the pipeline's logs, which are surfaced in the UI, or read the pod logs directly with kubectl logs for the full, unfiltered output.
Data is backing up or a pipeline shows as throttled
Use the UI to find which pipeline node is building up: its back-pressure size shows where the backlog is forming. Then check that node's metrics and pod logs to see whether it is running slowly or erroring.
Back-pressure usually originates at a slow or unavailable output and propagates upstream. See Reliability & Delivery for how this behaves and what the status indicators mean.
Push data (HTTP, OTLP, syslog) is rejected
Check the logs of the HTTP input, the L4 input (for TCP-terminated inputs like syslog), and the pipeline's own input pod for details on why data was rejected. If none of them show any logs for the traffic, the request likely never reached them, which points at routing or the gateway (including TLS).
Config changes won't save
Check the API and PostgreSQL.
Metrics, dashboards, or alerts look wrong
Check Victoria Metrics and the Alert Service. Also check the API logs for errors talking to the metrics database, since the API queries Victoria Metrics to serve the metrics shown in the UI.
Check that dependencies are healthy
If several components are failing at once, a shared dependency is often the cause. Confirm the database, message queue, and cache pods are Running:
Code
Getting help
If you can identify the affected component but not the cause, capture the relevant pod logs and reach out to Monad support at support@monad.com.