Installation With Helm
Overview
This guide documents the process for installing Monad into a Kubernetes cluster using Helm. The primary installation method uses the Kubernetes Gateway API for ingress. If your cluster uses a traditional ingress controller instead, see Alternative: Ingress Controller at the end of this document.
Phase 1: Prerequisites
These must be installed before the Monad Helm chart can be deployed.
Authentication Provider
Monad supports local authentication, Auth0, and AWS Cognito, with Auth0 being the default.
Required Operators/Controllers
-
Victoria Metrics Operator
- Required for internal metrics and dashboards, deployed by Custom Resources from within the chart. This does not replace an observability stack like Prometheus, and all components expose endpoints for scraping by an observability platform.
- Installation: https://docs.victoriametrics.com/operator/
-
CloudNativePG (CNPG) - PostgreSQL operator
- Required unless you're using an external PostgreSQL instance
- Installation: https://cloudnative-pg.io/documentation/current/installation_upgrade/
Gateway or Ingress Controller
- Monad recommends using the Kubernetes Gateway API for ingress and will create HTTPRoute and TCPRoute resources automatically. You need a Gateway API implementation installed in your cluster (e.g., Traefik, Istio, Envoy Gateway, kgateway, or any other conformant implementation).
- Your implementation must support the experimental Gateway API channel, which includes
TCPRoute. - Installation varies by implementation. Refer to your implementation's documentation.
- This guide assumes the use of Gateway API, though alternatives for using an ingress controller are provided at the end of the document.
Create Namespace
Code
Phase 2: Create Required Secrets
These secrets must exist before installing the Helm chart. All secrets are created in the monad namespace.
1. License Secret
The Monad license is a TLS certificate. You should have received a license.crt file from Monad.
Code
2. Encryption Key Secret
Used for encrypting sensitive data within Monad. This is a base64-encoded random 32-byte key.
Code
3. Key Encryption Key (KEK)
The Key Encryption Key (KEK) is the master key that wraps every organization's Data Encryption Key (DEK). All historical KEK versions must be preserved — older versions stay in the Secret because previously-wrapped DEKs still need them to decrypt. Losing the KEK is unrecoverable: Monad has no way to recover wrapped DEKs without it, and every byte of data wrapped under a lost version becomes permanently inaccessible. You are the sole custodian.
Each Monad organization is issued its own DEK, which is wrapped by the active KEK before being stored. Decryption requires the same KEK version that performed the wrap, so the Secret holds every version that has ever been used. The KEK never leaves the cluster.
The Helm chart consumes the KEK from a Kubernetes Secret named monad-master-encryption-key in the monad namespace. Each data field is a numeric version label (1, 2, 3, …) holding a base64-encoded 32-byte random key. The highest numeric field is the active version; new versions are appended on rotation, and old versions are never deleted or overwritten. When a new version appears, Monad automatically re-wraps every organization's DEK to the new active KEK — no manual re-encryption step is required.
Recommended: sync from your secrets manager
In production, the KEK should originate in the secrets manager you already operate — HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, 1Password, etc. — and be reconciled into the cluster via External Secrets Operator (ESO), so that backups, audit, and rotation flow through the same controls as the rest of your secrets.
Store an item in your secrets manager (e.g. named monad-key-encryption-key) with one numeric field per version, each value a base64-encoded 32-byte random key (openssl rand 32 | base64 or any equivalent CSPRNG). Then enable the chart's ExternalSecret in your values-override.yaml:
Code
The extract directive syncs every field of the upstream item into the K8s Secret, so adding a new numeric version upstream needs no chart or values change.
You are the sole custodian of the KEK. Back up every version in your secrets manager, and back up that secrets manager. Never delete or overwrite a numeric field once it has been used to wrap data. Treat KEK exfiltration as a P0 security incident.
Without an external secret store
For air-gapped clusters or environments without ESO (or an equivalent bridge like SealedSecrets / SOPS), you can create the Secret directly with kubectl. Generate and back up the key material in something durable before applying it — once it lives only in the cluster, K8s becomes the source of truth, which is fragile.
Code
Rotations are performed by kubectl patch-ing in a new numeric field, or by re-creating the Secret with all existing versions plus the new one.
Verify
After install, confirm the api and operator pods see the mount:
Code
4. Image Pull Secret
You should have received credentials from Monad support for accessing the images in Docker Hub. Configure those here as default-pull-secret (required name).
Code
5. Authentication Secrets
Using External Secrets Operator
If you're using an external secrets store, you'll need to either save your secrets with the property names found in values.yaml or update the values.yaml to match your secret property names.
Code
Not Using External Secrets Operator
If you're not using an External Secrets operator, you need to create secrets for authentication backend credentials. Below is an example file of key/value environment variable pairs that you can create a secret from.
Note: Some of these variables exist in two forms. Monad is migrating from variable names like AUTH0_CLIENT_ID to MONAD_AUTH_AUTH0_CLIENT_ID. The MONAD_-prefixed variables are the new names, and the old names will be deprecated in a future release. For current and future compatibility, use both until informed that the old names can be removed.
api.env
Code
Create the Secret api from the file:
Code
ui.env
Code
Create the secret ui from the file:
Code
6. NATS Credentials (optional)
Monad's internal message bus runs without authentication by default, and no secret is needed for it. If your environment requires the bus to be authenticated as well, see NATS Authentication. It has to be set up before the first install, so read that section now rather than after the platform is running.
Phase 3: Configure TLS
For the remainder of this guide we will be using monad.example.com as our example domain.
Monad requires two TLS certificates:
1. Gateway Certificate
Used by the Gateway to terminate HTTPS for all web traffic. This is a standard certificate for your Monad hostname (e.g., monad.example.com).
The Secret must be created in the same namespace as your Gateway resource (not the monad namespace). This is a Gateway API requirement: certificate secrets must be co-located with the Gateway that references them.
See TLS with cert-manager for instructions on creating this certificate automatically.
2. HTTP Input TLS Certificate
Monad's HTTP Input workload handles inputs such as Syslog that require direct TLS termination at the Pod level. This certificate is separate from the Gateway certificate and must be named http-input-tls in the monad namespace.
Syslog and other TCP-terminated inputs use the hostname with SNI for routing, with names like cef85707-4b6e-405a-aea9-3237d520e805.l4.monad.example.com. It should answer to both l4.monad.example.com and the wildcard domain *.l4.monad.example.com. You will also need a DNS record pointing *.l4.monad.example.com to the load balancer address that handles TCP traffic into your cluster.
The full FQDN that you use for L4 traffic doesn't have to be connected to the hostname that you use for Monad itself (such as *.l4.monad.example.com and monad.example.com). As long as the FQDN you choose lands on the Gateway and is routed to the correct Service, the Pod that receives it performs a handshake, retrieves the requested FQDN from SNI, and then uses the hostname portion to route to the corresponding pipeline.
See TLS with cert-manager for instructions on creating this certificate automatically.
If you want each pipeline to have its own ingest hostname, this certificate needs one more name on it. See Per-Pipeline Ingest Host.
Configure Your Gateway
Your Gateway resource needs an HTTPS listener that references the Gateway certificate Secret. The exact configuration depends on your implementation, but the Gateway API spec looks like:
Code
Refer to your Gateway implementation's documentation for how to configure listeners.
Per-pipeline ingest hostnames need one additional HTTPS listener. See Per-Pipeline Ingest Host.
Phase 4: Configure values.yaml Overrides
Create a values-override.yaml file with the following configurations.
With Gateway API enabled, Monad's chart generates HTTPRoute resources automatically for all components. You do not need to configure ingress per-component — setting hostnames and routing at the top level is sufficient for all components to be reachable.
Code
The hostnames value replaces what previously required per-component BACKEND_URL environment variables and per-component ingress host configuration. Setting it once here propagates to all components automatically.
Phase 5: Install Monad
Log in to OCI Registry
Before you can pull the Helm chart, authenticate to the Docker registry:
Code
Note: Credentials are provided by Monad support. The login persists in ~/.docker/config.json.
Install with Custom Values
Code
You can perform upgrades of Monad using the same command above. Simply remove the --install directive to perform an upgrade.
Verify Installation
Some pods will initially come up in an Error state as they wait for the database to be ready. They should all be Running (or Completed) within a few minutes.
Code
Phase 6: Post-Installation
At this point, Monad should be up and running. Access it at your designated hostname.
For a map of the components now running in your cluster, how data flows between them, and where to look when troubleshooting, see Platform Architecture.
Alternative Installation Options
TLS with cert-manager
If you're using cert-manager for certificate management, create a ClusterIssuer and two Certificate resources as described below.
ClusterIssuer
Code
Gateway Certificate
Create this in your Gateway's namespace (e.g., kube-system for Traefik on k3s):
Code
HTTP Input TLS Certificate
Create this in the monad namespace:
Code
Per-Pipeline Ingest Host
Monad can serve each pipeline its own ingest hostname, <pipeline-id>.data.monad.example.com. Senders POST to the root of that host with no path, and the hostname itself identifies the pipeline, so there is nothing per-sender to configure beyond the URL. Syslog clients use the same name on port 6514, where the pipeline ID is read from SNI.
This is optional. Without it, HTTP senders post to https://monad.example.com/api/v2/http/send/<pipeline-id> and syslog clients connect to <pipeline-id>.l4.monad.example.com:6514.
Setting it up takes a DNS record, a certificate change, a Gateway listener, and one values override.
1. DNS
Point a wildcard record at the load balancer that fronts your Gateway:
Code
HTTP ingest (443) and syslog (6514) both use this name, so it has to resolve to a load balancer that carries both listeners.
2. Certificates
The wildcard name has to appear on two certificates, because HTTP ingest and syslog terminate TLS in different places:
- Gateway certificate. The Gateway terminates TLS for HTTP ingest, so add
*.data.monad.example.comto the certificate your HTTPS listener references, or issue a separate certificate for the data listener. http-input-tls. The HTTP Input Pod terminates TLS itself for syslog, and this is the certificate a syslog client validates, so it needs the same name.
With cert-manager, that is one more entry in each dnsNames list:
Code
A wildcard SAN cannot be issued through an HTTP-01 solver. If you use Let's Encrypt, the issuer for these certificates needs a DNS-01 solver.
3. Gateway listener
Add an HTTPS listener for the wildcard name to the Gateway you configured in Phase 3:
Code
The tcp listener on 6514 needs no change. A Gateway API TCP listener carries no hostname, so it already accepts syslog connections for any name that resolves to it.
4. values.yaml override
Add a route on the http-input component that claims the wildcard hostname and sends everything under it to the ingest Service:
Code
To accept OTLP on the same hostname, add a second route for the OTLP listeners. Both ports land on the ingest Service's single OTLP port:
Code
Define this under http-input, not at the top level
A route defined at the top level of values-override.yaml is inherited by every component, so each one would claim *.data.monad.example.com and ingest requests could land on the UI or the API instead.
A new route name also inherits nothing from routes.default, so enabled, type, pathType, hostnames, parentRefs, and rules all have to be set. Leaving out type: http renders no route at all and reports no error.
5. Verify
Code
A 200 response with {"status":"success","count":1} means the DNS record, certificate, listener, and route are all wired up.
External Postgres
If you want to use an external Postgres instance, you can disable the CloudNativePG operator and provide the connection details in the monad-db-app Secret.
Below is an example file you can create a Secret from. It uses the following values:
dbname: monaduser: monadpassword: somereallylongandcomplexpasswordhost: monad-db-rw.postgres / monad-db-rw.postgres.svc.cluster.local (default CNPG service structure for a database in thepostgresnamespace)port: 5432
Code
Create the Secret monad-db-app from the file:
Code
Disable the CloudNativePG operator in your values-override.yaml:
Code
AWS IAM Access for Connectors
Skip this section if you do not use any AWS connectors. If you do — S3, CloudTrail, GuardDuty, Security Hub, Inspector, the S3 output, and so on — you have a choice of two authentication methods, and only one of them needs anything from your Helm install:
- Static credentials. You give the connector an access key and secret key, stored as Monad secrets. Nothing in this section applies.
- IAM role assumption. The connector pod assumes a role in your account. This requires that the pod has an AWS identity of its own, which a self-hosted install has to provide. That is what this section covers.
The connector-side setup — which fields to fill in, and the trust policy templates — is documented in AWS Connectors. This section is only about the cluster half.
How role assumption works
A connector configured with a Role ARN points at the target role: the one holding the permissions for the data you want. How Monad gets to it depends on a single environment variable, MONAD_SECURE_ACCESS_DELEGATE_ROLE:
MONAD_SECURE_ACCESS_DELEGATE_ROLE | Behavior |
|---|---|
| Empty or unset (self-hosted default) | Single hop. The pod's own identity assumes the connector's Role ARN directly. |
| Set to an IAM role ARN | Two hop. The pod assumes that delegate role first, then assumes the connector's Role ARN using the delegate's credentials. |
Self-hosted installs normally want the single hop, which is the default — leave the variable unset and the pods use whatever AWS identity you have given them. The two-hop path exists for Monad Cloud, where a Monad-owned delegate role sits between the workload and your account; it is documented here only so the variable is not a mystery if you see it.
In both cases Monad passes your organization ID as the ExternalId when assuming the target role, so that role's trust policy must expect it. Your organization ID is the UUID in the Monad UI's address bar while you are viewing your organization (/organizations/<organization-id>); it is also shown on the detail panel of any pipeline node.
Give the pods an AWS identity
Monad does not create AWS identity for you. On EKS, the usual route is IRSA: annotate the Kubernetes service accounts with a role ARN and let the pods federate into it.
Three service accounts make AWS calls, and they are created by different things:
| Service account | Why it needs an AWS identity | Created by |
|---|---|---|
api | Connection tests run in the API process | The Helm chart, via api.serviceAccount.annotations |
inputs, outputs | Every AWS connector pod runs as one of these, whether it is a long-running deployment or a cron-scheduled job | The pipeline operator, at runtime — see the warning below |
The operator creates the data plane service accounts (inputs, outputs, transforms, enrichments, alerts) only if they do not already exist, and it creates them with no annotations and never updates them afterward. There is no chart value for these — the connector pods are spawned by the operator, not rendered by the chart. Create the ones you need yourself, annotated, and the operator will leave them alone.
Create the connector service accounts in the Monad namespace:
Code
Then annotate the api service account in your values-override.yaml:
Code
IRSA credentials are injected when a pod is admitted, so annotating a service account does nothing for pods that are already running. If you are adding these annotations to a live install, delete the affected connector pods afterward so they are recreated with the credentials in place.
IAM setup in your account
For a single-hop install you create two kinds of role:
- A web-identity role — the one you put in the annotations above. Its trust policy federates your cluster's OIDC provider, conditioned on the service account subjects (
system:serviceaccount:monad:api,system:serviceaccount:monad:inputs,system:serviceaccount:monad:outputs) and thests.amazonaws.comaudience. If you do not already have an IAM OIDC provider for the cluster, create that first. AWS Connectors has the full trust policy template. - The connector target roles — one per connector, whatever granularity you want. Each trusts the web-identity role above and requires your Monad organization ID as the
ExternalId. These hold the actual data permissions, and their ARNs are what you paste into the connector's Role ARN field in the Monad UI.
A target role's trust policy looks like this:
Code
The web-identity role also needs permission to assume those target roles. This is mandatory when a target role lives in a different account from the web-identity role, and harmless when they share one:
Code
If your cluster runs the Crossplane AWS provider, the chart can instead emit PodIdentityAssociation resources via the per-component podIdentityAssociations value. Most self-hosted installs will not have that provider, so the IRSA annotations above are the general path.
Using a delegate role
If Monad support asks you to route through a delegate role, set it on both the api and the operator. The api uses it directly for connection tests; the operator copies it into the input and output pods and the cron jobs it spawns, so setting it in those two places covers the whole data plane:
Code
Verify
Check that the annotation reached the connector service accounts, and that a running connector pod picked up the projected credentials:
Code
If a connector fails with an STS AccessDenied, the error names the hop that failed: failed to assume Role A is the delegate hop, failed to assume Role B is the target role. A failure on the target role usually means its trust policy is missing the web-identity role principal, or the ExternalId condition does not match your organization ID.
Local Authentication
Local authentication creates an admin user of admin@monad.local and a random password. To activate this, remove the Auth0 and Cognito configuration from the api and ui secrets, and set MONAD_AUTH_TYPE to local for both components in your values-override.yaml:
Code
After completing the installation, retrieve the Secret with the username and password:
Code
This secret is only used to deliver credentials after installation and can be deleted after retrieval.
Local authentication does not allow the creation of more users than the local admin. If you wish to have multiple users, log in as the admin user and set up SSO from the Settings menu.
NATS Authentication
Monad uses NATS as its internal message bus. The chart deploys the cluster with no authentication, and every component connects to it anonymously. The cluster is only reachable from inside the Kubernetes cluster, so this is a reasonable default, and nothing below is required to run Monad.
If your environment requires the message bus itself to be authenticated, NATS can be run in operator mode instead. Every connection then has to present a credentials file signed by an operator you generate and hold. This section covers generating that trust hierarchy with nsc and wiring it into the chart.
Enable this at install time, not on an upgrade
Authentication cannot be turned on for a deployment that is already running. Turning it on moves the platform onto a new NATS account, and JetStream keeps its data per account: a cluster that has been running unauthenticated holds everything under the anonymous global account, and a cluster in operator mode has no way to reach it.
There is no migration for this and it is not a supported change. Enable it as part of a first install, before any pipelines have run.
Prerequisites
nsc, the NATS credentials tool:brew install nsc- A chart version of 2.77.1 or later
jq, used below to read account identifiers out ofnsc
1. Generate the operator, accounts, and credentials
NATS operator mode builds a chain of trust: an operator signs accounts, and an account signs the users that services authenticate as. Two accounts are enough for Monad, and they match what the platform expects:
SYS, the system account, used for server monitoring and administrationmonad, the account every Monad component connects to
Run the following on a machine you control. None of it touches the cluster.
Code
JetStream is off by default on a new account
The nsc edit account step is not optional. A new account in operator mode carries no JetStream limits at all, and a client that tries to use JetStream against it gets a request timeout rather than a permissions error, which is a hard failure to recognize. Monad relies on JetStream for everything, so the account needs the limits above before the platform can start.
nsc keeps its keys under ~/.nsc and ~/.nkeys by default. Back both up and treat them the way you treat any other signing key. They are what lets you issue further credentials later; if you lose them you cannot add a credential to this cluster without generating a new operator, which means reinstalling.
2. Add the trust configuration to your values
The NATS servers need the operator's public identity, the system account, and the account JWTs. This command writes the block to add to your values-override.yaml:
Code
Everything it prints is public key material. JWTs are signed assertions rather than secrets, so this block belongs in your values file alongside the rest of your configuration. The private keys stay in your nsc store, and the only secret that reaches the cluster is the credentials file from step 1.
Two parts of that block are worth understanding:
patchremoves the staticaccountsblock the chart ships for the unauthenticated setup. NATS refuses to start when a configuration carries bothoperatorandaccounts, and it is not caught ahead of time:nats-server -treports the combination as valid, and the server only fails once it is running.resolver: MEMORYserves exactly the accounts listed inresolver_preloadand learns no others. That keeps the deployed trust configuration identical to what is in your values file, with no JWT store to back up or push to. The tradeoff is that adding an account later is a chart upgrade.
3. Create the credentials Secret
The credentials file goes into a Secret in the Monad namespace. The key must be nats.creds, which is what the components look for:
Code
Then point the chart at it in your values-override.yaml:
Code
That one value covers the whole platform. Every component that talks to NATS mounts the Secret at /etc/nats-creds, and the pipeline operator passes it on to the pods it creates for your pipelines, your scheduled inputs, and alerting. Components that never open a NATS connection, such as the UI, are left alone.
A pod will not start while the Secret is missing or while its nats.creds key is absent. That is deliberate. The alternative is a pod that starts, connects anonymously, and is refused by NATS on every attempt, which is far harder to diagnose.
4. Install
Install as described in Phase 5. The values from steps 2 and 3 are part of the same values-override.yaml as the rest of your configuration.
5. Verify
Every pod should reach Running as usual:
Code
A component that cannot authenticate logs nats: Authorization Violation and retries, so a pod that is up but reporting that has a credentials problem rather than a connectivity one. The usual causes are a Secret built from the wrong file or one whose key is not named nats.creds.
To check the cluster directly, give the bundled NATS box the same credentials:
Code
Code
An anonymous connection should now be refused. A throwaway pod with no credentials shows this:
Code
Code
Ingress Controller
If your cluster uses a traditional Kubernetes Ingress resource rather than Gateway API, use the following values-override.yaml instead of the one in Phase 4. All other phases remain the same, except:
- The TLS certificate Secret should be created in the
monadnamespace instead of the Gateway namespace - Use
kubectl get ingress -n monadinstead ofkubectl get httproute -n monadto verify routing
Remove the routing and routes keys from the earlier example of values-override.yaml and add the following:
Code
The hostnames value at the top automatically configures backend URLs and the UI origin for all components. Per-component ingress.hosts configuration is handled by the chart defaults and does not need to be set explicitly.