# Backup and Restore This page lists what in a self-managed Monad install holds state, what you lose without each part, and how to back up and restore the parts the chart runs. Read it before you go to production: Postgres backups need settings in your values file and somewhere to store the backups. The commands on this page use the release name `monad` in the namespace `monad`, and the chart's default Postgres Cluster name, `monad-db`. ## What holds state | What | Holds | If you lose it | Protect it with | | --- | --- | --- | --- | | Postgres | Your configuration: organizations, users, pipelines, connectors and settings, and the secrets you store in Monad, encrypted. With local authentication, the Kratos database too. | That configuration. | [CloudNativePG backups](#back-up-postgres) for the chart's Postgres. For your own Postgres, your database's own backups, including the `kratos` database if local authentication uses it. | | The `monad-master-encryption-key` Secret | The key encryption key (KEK), which unwraps each organization's data encryption key | Everything encrypted under a lost KEK version is permanently unreadable, in the live database and in every backup of it. Monad cannot recover it. | Your secrets manager. See [Key encryption key](/docs/guides/self-managed/before-you-install#key-encryption-key). | | NATS JetStream volumes | Records buffered between pipeline stages, and events between Monad's components that are not yet processed | Those records and events. Monad's components recreate their streams and buckets when they next start. | [Volume snapshots](#back-up-nats), or accept the loss. | | VictoriaMetrics vmstorage volumes | Pipeline metrics, such as volume, throughput and health, for `victoriametrics.retentionPeriod` (one month by default) | The metrics history the console shows. New metrics arrive as pipelines run. | [Replication](#victoriametrics), volume snapshots, or accept the loss. | | Other Secrets | License, identity provider settings, Kratos and NATS credentials | Depends on the Secret | See [Secrets](#secrets). | Dragonfly, the cache, keeps nothing on disk, so there is nothing to back up. A Postgres backup is useful only with the KEK versions that wrapped the keys in it. This is why no KEK version is ever deleted: restoring an older backup needs the versions that were active when it was taken. ## Back up Postgres With the chart's Postgres, CloudNativePG can take a base backup on a schedule and archive the write-ahead log (WAL) continuously to object storage. Together they let you restore the database to its latest state or to any point in time the archive covers. Backups are off by default. :::warning[Backups need CloudNativePG 1.30 or earlier] The chart's backups use CloudNativePG's native Barman Cloud support (`spec.backup.barmanObjectStore`), which CloudNativePG has deprecated and announced it will remove in 1.31.0. Keep the operator between 1.25, the chart's minimum, and 1.30 while backups are on. On 1.30, CloudNativePG returns a deprecation warning whenever the chart applies the Cluster; the backups still run. ::: | Key | Default | What it sets | | --- | --- | --- | | `postgresql.cnpg.backup.enabled` | `false` | Turns on scheduled base backups and WAL archiving. | | `postgresql.cnpg.backup.objectStore.destinationPath` | Empty | Where backups go: `s3:///`, `https://.blob.core.windows.net//` or `gs:///`. Required when backups are on; the render fails without it. | | `postgresql.cnpg.backup.objectStore.provider` | `s3` | `s3`, `azure` or `gcs`. | | `postgresql.cnpg.backup.schedule` | `"0 0 0 * * *"` | When the base backup runs: daily at 00:00. CloudNativePG schedules have six fields, the first for seconds. | | `postgresql.cnpg.backup.retentionPolicy` | `30d` | How long backups are kept. | | `postgresql.cnpg.backup.objectStore.wal.compression` | `bzip2` | Compression for archived WAL files. | | `postgresql.cnpg.backup.objectStore.wal.maxParallel` | `8` | WAL files archived or restored in parallel. | CloudNativePG writes each Cluster's backups under a folder named after it, such as `/monad-db/`. ### Give Postgres access to the object store The chart stores no object-store keys. The Postgres pods reach the object store with the workload identity of their ServiceAccount, which CloudNativePG names after the Cluster: `monad-db`, unless you set `postgresql.cnpg.name`. The chart has no setting for access keys or for a custom S3 endpoint. Set up the identity for your object store, then add its block to your values file. Use one block. If your file already has a `postgresql` key, as `examples/private-cluster.yaml` does, merge the block into it. CloudNativePG's [object store reference](https://cloudnative-pg.io/docs/1.30/appendixes/object_stores) has more detail on each provider. #### Amazon S3 Use IAM roles for service accounts. Create an IAM role that trusts your EKS cluster's OIDC provider for the subject `system:serviceaccount:monad:monad-db`, and that can read, write, list and delete objects under the backup path. Delete lets the retention policy remove old backups. ```yaml postgresql: cnpg: backup: enabled: true objectStore: provider: s3 destinationPath: s3:/// spec: serviceAccountTemplate: metadata: annotations: eks.amazonaws.com/role-arn: arn:aws:iam:::role/ ``` #### Azure Blob Storage Use Azure Workload Identity, on an AKS cluster with the OIDC issuer and workload identity turned on. Create a managed identity with a federated credential for the subject `system:serviceaccount:monad:monad-db`, and give it the Storage Blob Data Contributor role on the container. The pods also need the `azure.workload.identity/use` label, which `inheritedMetadata` adds. ```yaml postgresql: cnpg: backup: enabled: true objectStore: provider: azure destinationPath: https://.blob.core.windows.net// spec: serviceAccountTemplate: metadata: annotations: azure.workload.identity/client-id: inheritedMetadata: labels: azure.workload.identity/use: "true" ``` #### Google Cloud Storage Use Workload Identity Federation for GKE. Either grant a role that can read, write, list and delete objects in the bucket, such as Storage Object Admin, to the ServiceAccount's principal, `principal://iam.googleapis.com/projects//locations/global/workloadIdentityPools/.svc.id.goog/subject/ns/monad/sa/monad-db`, and leave out `spec` below; or grant it to a Google service account and annotate the ServiceAccount with it, as here: ```yaml postgresql: cnpg: backup: enabled: true objectStore: provider: gcs destinationPath: gs:/// spec: serviceAccountTemplate: metadata: annotations: iam.gke.io/gcp-service-account: @.iam.gserviceaccount.com ``` ### Take the first backup WAL archiving starts as soon as the upgrade applies, but the first scheduled base backup waits for the schedule, and a restore needs a base backup. Take one now: ```bash kubectl apply -f - </ serverName: monad-db s3Credentials: inheritFromIAMRole: true ``` With `targetTime`, CloudNativePG starts from the closest base backup completed before that time and replays the archive up to it. The recovery fails if the archive holds no transaction after that time. 2. In your main values file, change `postgresql.cnpg.backup.objectStore.destinationPath` to a new, empty path under the one the backup identity can already write to, such as `s3:////restored-`. CloudNativePG then writes to `/restored-/monad-db/`, apart from the old `/monad-db/`. The restored Cluster has the same name as the old one, so on the old path its backups would land in the folder you are restoring from, and CloudNativePG refuses to archive into a folder that already holds another cluster's WAL. With Argo CD or Flux, make the change in the values file in Git and commit it. It takes effect at step 4; until it is in Git, every sync puts the old path back. 3. Delete the Cluster. This also deletes its volumes and the `monad-db-app` Secret, and Monad has no database until the restore finishes: ```bash kubectl delete clusters.postgresql.cnpg.io monad-db -n monad ``` 4. Run your usual `helm upgrade`, at the chart version you already run, with `restore.yaml` after your main values file: ```bash helm upgrade --install monad oci://registry-1.docker.io/monadinc/monad \ --version \ --namespace monad \ --values values.yaml \ --values restore.yaml ``` With Argo CD, which has no Helm release to upgrade, commit `restore.yaml` beside your main values file instead, add it to the Application's `helm.valueFiles` after the main file, such as `$values/monad/restore.yaml`, and sync the Application once. With Flux, commit `restore.yaml` beside your main values file instead. Add it to the `monad-values` ConfigMap generator from [Install with Flux](/docs/guides/self-managed/install#flux), and to the HelmRelease's `valuesFrom` after the main values: ```yaml configMapGenerator: - name: monad-values files: - values.yaml - restore.yaml ``` ```yaml valuesFrom: - kind: ConfigMap name: monad-values - kind: ConfigMap name: monad-values valuesKey: restore.yaml ``` Once Flux has applied the commit, this prints the file: ```bash kubectl get configmap monad-values -n monad -o jsonpath='{.data.restore\.yaml}' ``` Only then resume the HelmRelease, which upgrades the release with both files. Resuming earlier creates an empty Cluster that a later restore cannot replace without deleting it again: ```bash flux resume helmrelease monad -n monad ``` 5. Wait until `STATUS` reads `Cluster in healthy state`. A large database takes a while to restore: ```bash kubectl get clusters.postgresql.cnpg.io monad-db -n monad --watch ``` 6. Restart Monad's components. They read the database password when they start, and the restored Cluster has a new one in `monad-db-app`: ```bash kubectl rollout restart deployment -n monad -l app.kubernetes.io/name=monad ``` 7. Take a base backup into the new path, as in [Take the first backup](#take-the-first-backup), using a new Backup name. 8. Leave `restore.yaml` out of later upgrades, and resume any sync you paused. With Argo CD, remove `restore.yaml` from `helm.valueFiles` and turn automated sync back on. With Flux, remove `restore.yaml` from the ConfigMap generator and from `valuesFrom` in the same commit. CloudNativePG reads `bootstrap` only when it creates a Cluster, so the restored Cluster is unaffected. ## Back up NATS NATS holds only data in flight, so many installs accept losing it. To keep it, take Kubernetes [VolumeSnapshots](https://kubernetes.io/docs/concepts/storage/volume-snapshots/) of the JetStream volumes. They need: 1. A [CSI driver](https://kubernetes.io/docs/concepts/storage/volumes/#csi) that supports volume snapshots, such as the EBS CSI driver on EKS. 2. A snapshot controller, which carries out VolumeSnapshots through the CSI driver. On EKS, for example, this is the [CSI snapshot controller](https://docs.aws.amazon.com/eks/latest/userguide/csi-snapshot-controller.html) add-on. 3. The VolumeSnapshot CustomResourceDefinitions, usually installed with the snapshot controller, and a VolumeSnapshotClass for your CSI driver. The `nats` CLI can also back up every stream over the network with `nats backup account`, using the `monad` account's credentials and room for a full copy of the store. ### Take the snapshots Snapshot every server's volume at about the same time. The volumes are named `-nats-js--nats-`. For each one: ```yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: monad-nats-js-monad-nats-0-20261113 namespace: monad spec: volumeSnapshotClassName: source: persistentVolumeClaimName: monad-nats-js-monad-nats-0 ``` A snapshot of a running server captures its volume as if the server had stopped abruptly. ### Restore from the snapshots You replace each server's volume with one created from its snapshot while NATS is stopped. If a GitOps tool applies your release, pause its sync first, or it scales NATS back up. 1. Stop the NATS servers, and wait until their pods are gone: ```bash kubectl scale statefulset monad-nats -n monad --replicas=0 ``` 2. Delete the volumes: ```bash kubectl delete pvc -n monad \ monad-nats-js-monad-nats-0 monad-nats-js-monad-nats-1 monad-nats-js-monad-nats-2 ``` 3. Create each volume again from its snapshot, with the same name, size and StorageClass: ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: monad-nats-js-monad-nats-0 namespace: monad spec: accessModes: - ReadWriteOnce storageClassName: resources: requests: storage: 100Gi dataSource: apiGroup: snapshot.storage.k8s.io kind: VolumeSnapshot name: monad-nats-js-monad-nats-0-20261113 ``` 4. Start the servers again, with the number you run, and resume your GitOps tool's sync: ```bash kubectl scale statefulset monad-nats -n monad --replicas=3 ``` 5. Check that the streams are back: ```bash kubectl exec -n monad deploy/monad-nats-box -- nats stream ls ``` ## VictoriaMetrics vmstorage writes each sample to `victoriametrics.replicationFactor` nodes, two by default, so losing one node's volume loses no metrics. Keep `victoriametrics.vmstorage.replicaCount` at least that high. The chart has no backup setting for VictoriaMetrics. The VictoriaMetrics operator's built-in backup, `vmBackup` on the VMCluster, runs `vmbackupmanager`, which needs VictoriaMetrics Enterprise, and the chart does not expose it. The open-source [`vmbackup`](https://docs.victoriametrics.com/victoriametrics/vmbackup/) has to run beside each vmstorage node with access to its volume, which the chart does not set up. That leaves VolumeSnapshots of the vmstorage volumes, taken as for NATS; `kubectl get pvc -n monad` lists them. VictoriaMetrics protects its storage from corruption on an unclean shutdown, so a snapshot of a running node is usable. Most installs accept the loss instead: the metrics are a month of history, and new ones arrive as soon as pipelines run. ## Secrets | Secret | Back it up? | | --- | --- | | `monad-master-encryption-key` | Yes, every version. Nothing can recreate it. See [What holds state](#what-holds-state). | | `monad-license` | Keep the `license.crt` file Monad gave you. | | `api` and `ui` | Keep their values with your other secrets. They hold your identity provider settings. | | `monad-kratos-secrets` (local authentication) | Yes. It holds the Kratos database password and the secrets Kratos signs and encrypts with. When the chart generates it, the chart keeps it across upgrades and uninstalls, and generates new values if it is deleted. | | `monad-nats-creds` (NATS authentication) | Back up your `nsc` store rather than the Secret: it can issue the credentials again, and nothing else can. See [NATS authentication](/docs/guides/self-managed/nats#nats-authentication). | | `monad-db-app` | With the chart's Postgres, no: CloudNativePG creates it, and a restore creates a new one. With your own Postgres, keep its values with your other secrets. |