# NATS Monad's components pass records and events to each other through NATS JetStream, which also buffers pipeline data between stages. Read this page before the first install if you need to size the message bus or require it to authenticate. Both are much harder to change once Monad is running. The commands on this page use the release name `monad` in the namespace `monad`. The chart names its NATS objects after the release, such as `-nats`. ## The chart's NATS The chart runs a NATS cluster in the release namespace: a StatefulSet of NATS servers, each with its own JetStream volume, and a `nats-box` Deployment that holds the `nats` CLI for administration. It is the only NATS Monad supports; Monad does not run against a NATS you operate. By default it has no authentication, and every component connects anonymously. Its Services are `ClusterIP`, so it is reachable only from inside the Kubernetes cluster, by any pod unless a NetworkPolicy prevents it. | Setting | Key | Default | After the first install | | --- | --- | --- | --- | | NATS servers | `nats.config.cluster.replicas` | `3` | Keep at least 3: Monad creates every stream with three replicas. | | JetStream volume per server | `nats.config.jetstream.fileStore.pvc.size` | `10Gi` | Fixed. See [Grow the JetStream volume](#grow-the-jetstream-volume). | | StorageClass of that volume | `nats.config.jetstream.fileStore.pvc.storageClassName` | Unset: the cluster's default StorageClass | Fixed. An existing volume keeps its class. | | JetStream store limit | `nats.config.jetstream.fileStore.maxSize` | 90% of the volume size | Follows the volume size unless you set it. | Resources, node placement and priority for the NATS pods are on [Scheduling and Resources](/docs/guides/self-managed/scheduling). The NATS images and how to pull them from your own registry are on [Images and Registries](/docs/guides/self-managed/images). ### Size the JetStream volume before you install :::warning[The volume size cannot change through Helm] The JetStream volume size is part of the NATS StatefulSet's volume template, which Kubernetes does not let you change after the StatefulSet exists. Choose the size before the first install. Growing it later is the manual procedure below. ::: `10Gi` suits local and development clusters, `100Gi` small clusters, and `1000Gi` or more production. [Cluster Sizing](/docs/guides/cluster-sizing) covers how much your pipelines need. Every example values file already sets the size, so change it there and add a StorageClass beside it if you need one, rather than adding a second `nats:` key: ```yaml nats: config: jetstream: fileStore: pvc: size: 100Gi storageClassName: ``` `storageClassName` is optional. Without it the volume uses the cluster's default StorageClass. JetStream stops accepting new messages when its store reaches the limit, 90% of the volume, and reports that to publishers as backpressure. The remaining 10% is room to recover: filling the volume itself disables JetStream on that server. If it happens, see [Recovering when NATS runs out of disk](/docs/guides/troubleshooting#recovering-when-nats-runs-out-of-disk). ### Grow the JetStream volume `helm upgrade` with a larger `pvc.size` fails with `updates to statefulset spec for fields other than ... are forbidden`. Grow the volumes by hand instead, then let Helm recreate the StatefulSet with the new size. You need a StorageClass that allows volume expansion. The volumes are named `-nats-js--nats-`, one per server. If Argo CD, Flux or another tool applies your release, pause its sync for the release until the last step, so it does not recreate the StatefulSet with the old size. 1. Check that the volumes' StorageClass allows expansion. The second command prints `true` when it does: ```bash kubectl get pvc monad-nats-js-monad-nats-0 -n monad -o jsonpath='{.spec.storageClassName}' kubectl get storageclass -o jsonpath='{.allowVolumeExpansion}' ``` 2. Request the new size on each server's volume, here `200Gi` for three servers: ```bash for n in 0 1 2; do kubectl patch pvc "monad-nats-js-monad-nats-$n" -n monad \ -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}' done ``` 3. Delete the StatefulSet but leave its pods running: ```bash kubectl delete statefulset monad-nats -n monad --cascade=orphan ``` 4. Set `nats.config.jetstream.fileStore.pvc.size` to the new size in your values file and run your usual `helm upgrade`, or resume your GitOps tool's sync. The new StatefulSet takes over the running pods, and the JetStream limit rises to 90% of the new size. 5. Restart the servers one at a time, so each picks up the new limit and finishes any filesystem resize the storage driver leaves for a restart: ```bash kubectl rollout restart statefulset monad-nats -n monad kubectl rollout status statefulset monad-nats -n monad ``` A server with a large store can take a long time to start. [Recovering when NATS runs out of disk](/docs/guides/troubleshooting#recovering-when-nats-runs-out-of-disk) shows how to give it longer. 6. Check the new capacity of each volume and the free space each server sees: ```bash kubectl get pvc -n monad kubectl exec -n monad monad-nats-0 -c nats -- df -h /data ``` ## NATS authentication The chart's NATS is reachable only from inside the cluster, so running it without authentication is a reasonable default, and nothing in this section is required to run Monad. If your environment requires the message bus itself to authenticate, run NATS in operator mode: every connection then presents a credentials file signed by an operator you generate and hold. :::warning[Turn authentication on at the first install] Authentication cannot be turned on for an install that is already running. It moves Monad onto a new NATS account, and JetStream keeps its data per account: an install that has run without authentication holds its data under NATS's anonymous global account, which the new account cannot reach. There is no migration, and turning it on later is not supported. Set it up before the first install, before any pipeline has run. ::: ### Prerequisites - [`nsc`](https://github.com/nats-io/nsc), the NATS credentials tool, on a machine you control. On macOS: `brew install nsc`. - `jq`, which the commands below use to read account identifiers out of `nsc`. - Monad chart 2.77.2 or later. ### 1. Generate the operator, accounts and credentials NATS operator mode builds a chain of trust: an operator signs accounts, and an account signs the users that services authenticate as. Monad needs two accounts: - `SYS`, the system account, for server monitoring and administration - `monad`, the account every Monad component connects to Run these commands on your machine. None of them touch the cluster. ```bash # Create the operator, with the SYS system account and its sys user nsc add operator --name monad --sys # Create the account Monad connects to nsc add account --name monad # Turn JetStream on for that account, with no limits nsc edit account --name monad \ --js-tier 0 \ --js-mem-storage -1 \ --js-disk-storage -1 \ --js-streams -1 \ --js-consumer -1 # Issue the user Monad runs as, and write its credentials file nsc add user --account monad --name monad-platform nsc generate creds --account monad --name monad-platform > nats.creds ``` :::warning[A new account has JetStream turned off] Do not skip the `nsc edit account` step. A new account in operator mode has no JetStream limits, and every JetStream request against it fails with `JetStream not enabled for account`. Monad relies on JetStream throughout, so the account needs the limits above before Monad can start. ::: `nsc` keeps the operator's and accounts' private keys on this machine; `nsc env` shows where. Back them up and protect them like any other signing key. You need them to issue further credentials, and without them the only way to issue one is a new operator and new accounts, which is the same unsupported change as turning authentication on. ### 2. Write the authentication values The NATS servers need the operator's JWT, the system account's public key and the account JWTs. This command writes them, with the other two settings authentication needs, to a values file of its own, `nats-auth.yaml`: ```bash cat > nats-auth.yaml < \ --namespace monad \ --values values.yaml \ --values nats-auth.yaml ``` With Argo CD or Flux, add `nats-auth.yaml` to the values files the release reads, after your main one. ### 5. Verify Every pod should reach `Running` as usual: ```bash kubectl get pods -n monad ``` A component that cannot authenticate logs `nats: Authorization Violation` and retries. A pod that is running but logs that has a credentials problem, not a connectivity one: usually a Secret built from the wrong file, or one whose key is not `nats.creds`. With credentials, the `nats` CLI in `nats-box` lists Monad's streams: ```bash kubectl exec -n monad deploy/monad-nats-box -- nats stream ls ``` Without them, the connection is refused: ```bash kubectl exec -n monad deploy/monad-nats-box -- \ nats --no-context --server nats://monad-nats:4222 pub test hello ``` ``` nats: error: nats: Authorization Violation ```