Backup and Restore
This page lists what in a self-managed Monad install holds state, what you lose without each part, and how to back up and restore the parts the chart runs. Read it before you go to production: Postgres backups need settings in your values file and somewhere to store the backups.
The commands on this page use the release name monad in the namespace monad, and the chart's default Postgres Cluster name, monad-db.
What holds state
| What | Holds | If you lose it | Protect it with |
|---|---|---|---|
| Postgres | Your configuration: organizations, users, pipelines, connectors and settings, and the secrets you store in Monad, encrypted. With local authentication, the Kratos database too. | That configuration. | CloudNativePG backups for the chart's Postgres. For your own Postgres, your database's own backups, including the kratos database if local authentication uses it. |
The monad-master-encryption-key Secret | The key encryption key (KEK), which unwraps each organization's data encryption key | Everything encrypted under a lost KEK version is permanently unreadable, in the live database and in every backup of it. Monad cannot recover it. | Your secrets manager. See Key encryption key. |
| NATS JetStream volumes | Records buffered between pipeline stages, and events between Monad's components that are not yet processed | Those records and events. Monad's components recreate their streams and buckets when they next start. | Volume snapshots, or accept the loss. |
| VictoriaMetrics vmstorage volumes | Pipeline metrics, such as volume, throughput and health, for victoriametrics.retentionPeriod (one month by default) | The metrics history the console shows. New metrics arrive as pipelines run. | Replication, volume snapshots, or accept the loss. |
| Other Secrets | License, identity provider settings, Kratos and NATS credentials | Depends on the Secret | See Secrets. |
Dragonfly, the cache, keeps nothing on disk, so there is nothing to back up.
A Postgres backup is useful only with the KEK versions that wrapped the keys in it. This is why no KEK version is ever deleted: restoring an older backup needs the versions that were active when it was taken.
Back up Postgres
With the chart's Postgres, CloudNativePG can take a base backup on a schedule and archive the write-ahead log (WAL) continuously to object storage. Together they let you restore the database to its latest state or to any point in time the archive covers. Backups are off by default.
Backups need CloudNativePG 1.30 or earlier
The chart's backups use CloudNativePG's native Barman Cloud support (spec.backup.barmanObjectStore), which CloudNativePG has deprecated and announced it will remove in 1.31.0. Keep the operator between 1.25, the chart's minimum, and 1.30 while backups are on. On 1.30, CloudNativePG returns a deprecation warning whenever the chart applies the Cluster; the backups still run.
| Key | Default | What it sets |
|---|---|---|
postgresql.cnpg.backup.enabled | false | Turns on scheduled base backups and WAL archiving. |
postgresql.cnpg.backup.objectStore.destinationPath | Empty | Where backups go: s3://<bucket>/<path>, https://<account>.blob.core.windows.net/<container>/<path> or gs://<bucket>/<path>. Required when backups are on; the render fails without it. |
postgresql.cnpg.backup.objectStore.provider | s3 | s3, azure or gcs. |
postgresql.cnpg.backup.schedule | "0 0 0 * * *" | When the base backup runs: daily at 00:00. CloudNativePG schedules have six fields, the first for seconds. |
postgresql.cnpg.backup.retentionPolicy | 30d | How long backups are kept. |
postgresql.cnpg.backup.objectStore.wal.compression | bzip2 | Compression for archived WAL files. |
postgresql.cnpg.backup.objectStore.wal.maxParallel | 8 | WAL files archived or restored in parallel. |
CloudNativePG writes each Cluster's backups under a folder named after it, such as <destinationPath>/monad-db/.
Give Postgres access to the object store
The chart stores no object-store keys. The Postgres pods reach the object store with the workload identity of their ServiceAccount, which CloudNativePG names after the Cluster: monad-db, unless you set postgresql.cnpg.name. The chart has no setting for access keys or for a custom S3 endpoint.
Set up the identity for your object store, then add its block to your values file. Use one block. If your file already has a postgresql key, as examples/private-cluster.yaml does, merge the block into it. CloudNativePG's object store reference has more detail on each provider.
Amazon S3
Use IAM roles for service accounts. Create an IAM role that trusts your EKS cluster's OIDC provider for the subject system:serviceaccount:monad:monad-db, and that can read, write, list and delete objects under the backup path. Delete lets the retention policy remove old backups.
Code
Azure Blob Storage
Use Azure Workload Identity, on an AKS cluster with the OIDC issuer and workload identity turned on. Create a managed identity with a federated credential for the subject system:serviceaccount:monad:monad-db, and give it the Storage Blob Data Contributor role on the container. The pods also need the azure.workload.identity/use label, which inheritedMetadata adds.
Code
Google Cloud Storage
Use Workload Identity Federation for GKE. Either grant a role that can read, write, list and delete objects in the bucket, such as Storage Object Admin, to the ServiceAccount's principal, principal://iam.googleapis.com/projects/<project-number>/locations/global/workloadIdentityPools/<project-id>.svc.id.goog/subject/ns/monad/sa/monad-db, and leave out spec below; or grant it to a Google service account and annotate the ServiceAccount with it, as here:
Code
Take the first backup
WAL archiving starts as soon as the upgrade applies, but the first scheduled base backup waits for the schedule, and a restore needs a base backup. Take one now:
Code
Take one the same way before any change you might want to undo, such as a chart upgrade.
Check the backups
Code
A backup stuck in failed, or archiving that is not True, usually means the identity cannot reach the object store. kubectl describe on the Backup or the Cluster shows the error.
Restore Postgres
CloudNativePG does not restore into a running Cluster. You delete the Cluster and the chart creates it again, this time from the backup. The restored Cluster has the same name, so Monad finds it without other changes.
You need a completed base backup, and the object-store identity from Give Postgres access to the object store. If Argo CD, Flux or another tool applies your release, pause its sync for the release until the last step: once you delete the Cluster, a sync would create a new, empty one. With Argo CD, turn off the Application's automated sync. With Flux, suspend the HelmRelease: flux suspend helmrelease monad -n monad. Argo CD and Flux then follow their own variants of steps 2, 4 and 8, which make every change in Git rather than with Helm, so the values they apply afterwards are the ones the restore used.
-
Write the recovery settings to a values file of their own,
restore.yaml.destinationPathis the path the backups were written to, andserverNamethe name of the Cluster they came from. The credentials block matches your provider:s3Credentials.inheritFromIAMRole: truefors3,azureCredentials.inheritFromAzureAD: trueforazure, orgoogleCredentials.gkeEnvironment: trueforgcs.CodeWith
targetTime, CloudNativePG starts from the closest base backup completed before that time and replays the archive up to it. The recovery fails if the archive holds no transaction after that time. -
In your main values file, change
postgresql.cnpg.backup.objectStore.destinationPathto a new, empty path under the one the backup identity can already write to, such ass3://<your-bucket>/<path>/restored-<date>. CloudNativePG then writes to<path>/restored-<date>/monad-db/, apart from the old<path>/monad-db/. The restored Cluster has the same name as the old one, so on the old path its backups would land in the folder you are restoring from, and CloudNativePG refuses to archive into a folder that already holds another cluster's WAL.With Argo CD or Flux, make the change in the values file in Git and commit it. It takes effect at step 4; until it is in Git, every sync puts the old path back.
-
Delete the Cluster. This also deletes its volumes and the
monad-db-appSecret, and Monad has no database until the restore finishes:Code -
Run your usual
helm upgrade, at the chart version you already run, withrestore.yamlafter your main values file:CodeWith Argo CD, which has no Helm release to upgrade, commit
restore.yamlbeside your main values file instead, add it to the Application'shelm.valueFilesafter the main file, such as$values/monad/restore.yaml, and sync the Application once.With Flux, commit
restore.yamlbeside your main values file instead. Add it to themonad-valuesConfigMap generator from Install with Flux, and to the HelmRelease'svaluesFromafter the main values:CodeCodeOnce Flux has applied the commit, this prints the file:
CodeOnly then resume the HelmRelease, which upgrades the release with both files. Resuming earlier creates an empty Cluster that a later restore cannot replace without deleting it again:
Code -
Wait until
STATUSreadsCluster in healthy state. A large database takes a while to restore:Code -
Restart Monad's components. They read the database password when they start, and the restored Cluster has a new one in
monad-db-app:Code -
Take a base backup into the new path, as in Take the first backup, using a new Backup name.
-
Leave
restore.yamlout of later upgrades, and resume any sync you paused. With Argo CD, removerestore.yamlfromhelm.valueFilesand turn automated sync back on. With Flux, removerestore.yamlfrom the ConfigMap generator and fromvaluesFromin the same commit. CloudNativePG readsbootstraponly when it creates a Cluster, so the restored Cluster is unaffected.
Back up NATS
NATS holds only data in flight, so many installs accept losing it. To keep it, take Kubernetes VolumeSnapshots of the JetStream volumes. They need:
- A CSI driver that supports volume snapshots, such as the EBS CSI driver on EKS.
- A snapshot controller, which carries out VolumeSnapshots through the CSI driver. On EKS, for example, this is the CSI snapshot controller add-on.
- The VolumeSnapshot CustomResourceDefinitions, usually installed with the snapshot controller, and a VolumeSnapshotClass for your CSI driver.
The nats CLI can also back up every stream over the network with nats backup account, using the monad account's credentials and room for a full copy of the store.
Take the snapshots
Snapshot every server's volume at about the same time. The volumes are named <release>-nats-js-<release>-nats-<n>. For each one:
Code
A snapshot of a running server captures its volume as if the server had stopped abruptly.
Restore from the snapshots
You replace each server's volume with one created from its snapshot while NATS is stopped. If a GitOps tool applies your release, pause its sync first, or it scales NATS back up.
-
Stop the NATS servers, and wait until their pods are gone:
Code -
Delete the volumes:
Code -
Create each volume again from its snapshot, with the same name, size and StorageClass:
Code -
Start the servers again, with the number you run, and resume your GitOps tool's sync:
Code -
Check that the streams are back:
Code
VictoriaMetrics
vmstorage writes each sample to victoriametrics.replicationFactor nodes, two by default, so losing one node's volume loses no metrics. Keep victoriametrics.vmstorage.replicaCount at least that high.
The chart has no backup setting for VictoriaMetrics. The VictoriaMetrics operator's built-in backup, vmBackup on the VMCluster, runs vmbackupmanager, which needs VictoriaMetrics Enterprise, and the chart does not expose it. The open-source vmbackup has to run beside each vmstorage node with access to its volume, which the chart does not set up.
That leaves VolumeSnapshots of the vmstorage volumes, taken as for NATS; kubectl get pvc -n monad lists them. VictoriaMetrics protects its storage from corruption on an unclean shutdown, so a snapshot of a running node is usable. Most installs accept the loss instead: the metrics are a month of history, and new ones arrive as soon as pipelines run.
Secrets
| Secret | Back it up? |
|---|---|
monad-master-encryption-key | Yes, every version. Nothing can recreate it. See What holds state. |
monad-license | Keep the license.crt file Monad gave you. |
api and ui | Keep their values with your other secrets. They hold your identity provider settings. |
monad-kratos-secrets (local authentication) | Yes. It holds the Kratos database password and the secrets Kratos signs and encrypts with. When the chart generates it, the chart keeps it across upgrades and uninstalls, and generates new values if it is deleted. |
monad-nats-creds (NATS authentication) | Back up your nsc store rather than the Secret: it can issue the credentials again, and nothing else can. See NATS authentication. |
monad-db-app | With the chart's Postgres, no: CloudNativePG creates it, and a restore creates a new one. With your own Postgres, keep its values with your other secrets. |