NATS
Monad's components pass records and events to each other through NATS JetStream, which also buffers pipeline data between stages. Read this page before the first install if you need to size the message bus or require it to authenticate. Both are much harder to change once Monad is running.
The commands on this page use the release name monad in the namespace monad. The chart names its NATS objects after the release, such as <release>-nats.
The chart's NATS
The chart runs a NATS cluster in the release namespace: a StatefulSet of NATS servers, each with its own JetStream volume, and a nats-box Deployment that holds the nats CLI for administration. It is the only NATS Monad supports; Monad does not run against a NATS you operate. By default it has no authentication, and every component connects anonymously. Its Services are ClusterIP, so it is reachable only from inside the Kubernetes cluster, by any pod unless a NetworkPolicy prevents it.
| Setting | Key | Default | After the first install |
|---|---|---|---|
| NATS servers | nats.config.cluster.replicas | 3 | Keep at least 3: Monad creates every stream with three replicas. |
| JetStream volume per server | nats.config.jetstream.fileStore.pvc.size | 10Gi | Fixed. See Grow the JetStream volume. |
| StorageClass of that volume | nats.config.jetstream.fileStore.pvc.storageClassName | Unset: the cluster's default StorageClass | Fixed. An existing volume keeps its class. |
| JetStream store limit | nats.config.jetstream.fileStore.maxSize | 90% of the volume size | Follows the volume size unless you set it. |
Resources, node placement and priority for the NATS pods are on Scheduling and Resources. The NATS images and how to pull them from your own registry are on Images and Registries.
Size the JetStream volume before you install
The volume size cannot change through Helm
The JetStream volume size is part of the NATS StatefulSet's volume template, which Kubernetes does not let you change after the StatefulSet exists. Choose the size before the first install. Growing it later is the manual procedure below.
10Gi suits local and development clusters, 100Gi small clusters, and 1000Gi or more production. Cluster Sizing covers how much your pipelines need. Every example values file already sets the size, so change it there and add a StorageClass beside it if you need one, rather than adding a second nats: key:
Code
storageClassName is optional. Without it the volume uses the cluster's default StorageClass.
JetStream stops accepting new messages when its store reaches the limit, 90% of the volume, and reports that to publishers as backpressure. The remaining 10% is room to recover: filling the volume itself disables JetStream on that server. If it happens, see Recovering when NATS runs out of disk.
Grow the JetStream volume
helm upgrade with a larger pvc.size fails with updates to statefulset spec for fields other than ... are forbidden. Grow the volumes by hand instead, then let Helm recreate the StatefulSet with the new size. You need a StorageClass that allows volume expansion. The volumes are named <release>-nats-js-<release>-nats-<n>, one per server.
If Argo CD, Flux or another tool applies your release, pause its sync for the release until the last step, so it does not recreate the StatefulSet with the old size.
-
Check that the volumes' StorageClass allows expansion. The second command prints
truewhen it does:Code -
Request the new size on each server's volume, here
200Gifor three servers:Code -
Delete the StatefulSet but leave its pods running:
Code -
Set
nats.config.jetstream.fileStore.pvc.sizeto the new size in your values file and run your usualhelm upgrade, or resume your GitOps tool's sync. The new StatefulSet takes over the running pods, and the JetStream limit rises to 90% of the new size. -
Restart the servers one at a time, so each picks up the new limit and finishes any filesystem resize the storage driver leaves for a restart:
CodeA server with a large store can take a long time to start. Recovering when NATS runs out of disk shows how to give it longer.
-
Check the new capacity of each volume and the free space each server sees:
Code
NATS authentication
The chart's NATS is reachable only from inside the cluster, so running it without authentication is a reasonable default, and nothing in this section is required to run Monad. If your environment requires the message bus itself to authenticate, run NATS in operator mode: every connection then presents a credentials file signed by an operator you generate and hold.
Turn authentication on at the first install
Authentication cannot be turned on for an install that is already running. It moves Monad onto a new NATS account, and JetStream keeps its data per account: an install that has run without authentication holds its data under NATS's anonymous global account, which the new account cannot reach. There is no migration, and turning it on later is not supported. Set it up before the first install, before any pipeline has run.
Prerequisites
nsc, the NATS credentials tool, on a machine you control. On macOS:brew install nsc.jq, which the commands below use to read account identifiers out ofnsc.- Monad chart 2.77.2 or later.
1. Generate the operator, accounts and credentials
NATS operator mode builds a chain of trust: an operator signs accounts, and an account signs the users that services authenticate as. Monad needs two accounts:
SYS, the system account, for server monitoring and administrationmonad, the account every Monad component connects to
Run these commands on your machine. None of them touch the cluster.
Code
A new account has JetStream turned off
Do not skip the nsc edit account step. A new account in operator mode has no JetStream limits, and every JetStream request against it fails with JetStream not enabled for account. Monad relies on JetStream throughout, so the account needs the limits above before Monad can start.
nsc keeps the operator's and accounts' private keys on this machine; nsc env shows where. Back them up and protect them like any other signing key. You need them to issue further credentials, and without them the only way to issue one is a new operator and new accounts, which is the same unsupported change as turning authentication on.
2. Write the authentication values
The NATS servers need the operator's JWT, the system account's public key and the account JWTs. This command writes them, with the other two settings authentication needs, to a values file of its own, nats-auth.yaml:
Code
Keep it as its own file and pass it to Helm after your main values file: Helm merges the two, while pasting it into a file that already has global: and nats: keys would replace them. Everything in it is public: JWTs are signed assertions, not secrets, so it can live in Git with the rest of your values. The private keys stay in your nsc store, and the only secret that reaches the cluster is the credentials file.
What each part does:
global.monad.natsCredsSecretnames the Secret holding the credentials. Every component that talks to NATS mounts it at/etc/nats-creds, and the operator mounts it in the pods it creates for your pipelines, your scheduled inputs and alerting. Components that never open a NATS connection, such as the ui, do not mount it.nats.config.patchremoves the staticaccountsblock the chart ships for running without authentication. NATS refuses to start with bothoperatorandaccounts, and nothing catches this earlier:helm templateandnats-server -tboth accept the combination, and the server fails only when it starts.resolver: MEMORYserves exactly the accounts listed inresolver_preloadand learns no others. The deployed trust configuration is then exactly what is in your values, with no JWT store to back up. Adding an account later is a chart upgrade.nats.natsBox.contexts.default.credsgives thenatsCLI in thenats-boxpod the same credentials, for administration and for the checks below.
3. Create the credentials Secret
Put the credentials file in a Secret in the release namespace. The key must be nats.creds:
Code
You can also create it with your usual secret tooling, such as External Secrets Operator, as long as the Secret is named monad-nats-creds and the key is nats.creds.
A pod that talks to NATS does not start while the Secret or its nats.creds key is missing. That is deliberate: the alternative is a pod that starts, connects anonymously and is refused on every attempt.
4. Install
Install as described in Install, with nats-auth.yaml after your main values file:
Code
With Argo CD or Flux, add nats-auth.yaml to the values files the release reads, after your main one.
5. Verify
Every pod should reach Running as usual:
Code
A component that cannot authenticate logs nats: Authorization Violation and retries. A pod that is running but logs that has a credentials problem, not a connectivity one: usually a Secret built from the wrong file, or one whose key is not nats.creds.
With credentials, the nats CLI in nats-box lists Monad's streams:
Code
Without them, the connection is refused:
Code
Code