Scheduling and Resources
Read this page if your cluster decides where pods run by node labels, taints, affinity or PriorityClasses, or limits namespaces with quotas. It covers where each of Monad's pods runs, the PriorityClasses the chart creates, and the requests and limits of every workload. Any of these settings can change with an upgrade.
Merge snippets into your values file
Each snippet gives full paths from the top-level key. Your values file already has some of these keys, such as global and nats in every example file. Add a snippet's settings under the key you already have: a second copy of a top-level key silently replaces the first. See Install.
Place pods on your nodes
Set the chart-wide placement under global:
Code
global.affinity takes a standard Kubernetes affinity the same way.
These values reach Monad's components (api, ui, operator and the others in the first rows of Defaults for every workload), the NATS servers and the nats-box pod, Postgres, VictoriaMetrics, the pods the operator creates, and the Karmada box, bootstrap and register pods if Karmada is enabled. Dragonfly and Kratos don't read them; see Workloads that ignore global.
Unless you set global.affinity, NATS and Postgres prefer nodes labelled dedicated=stateful. Unless you set global.tolerations, they tolerate a dedicated=stateful:NoSchedule taint.
Quote node selector values
Quote a node selector value that YAML reads as a boolean or a number, such as "true" or "1". The chart refuses to render an unquoted one in global.nodeSelector or operator.managedPods.nodeSelector, and the Kubernetes API rejects one anywhere else when you install.
Give one workload its own value
A workload's own value and the global value combine like this:
- A workload's own
nodeSelector,tolerationsoraffinityreplaces theglobalone whole. The two are not merged, so a workload with its ownnodeSelectorneeds every label it should match, not only the extra one. falseremoves the field for that workload, for exampleapi.tolerations: false.- An empty
{}or[]counts as unset, so theglobalvalue applies. - Pull secrets add up: a workload gets
global.imagePullSecretsplus its own list. Images and Registries lists the pull secrets each pod gets.
Where each workload's own values go:
| Workload | nodeSelector, tolerations, affinity | PriorityClass |
|---|---|---|
| Monad components | Under the component, such as api.nodeSelector | <component>.priorityClassName |
| NATS servers | nats.podTemplate.merge.spec | The same spec |
nats-box | nats.natsBox.podTemplate.merge.spec | The same spec; none by default |
| Postgres | postgresql.cnpg.nodeSelector, tolerations, affinity | postgresql.cnpg.priorityClassName |
| VictoriaMetrics | Under each of victoriametrics.vmstorage, vmselect, vminsert and vmalert | Under each of the four |
| Pods the operator creates | operator.managedPods | operator.managedPods.priorityClassName |
| Karmada box, bootstrap and register pods, if enabled | Under each of karmada.box, karmada.bootstrap and karmada.register | Under each of the three |
NATS takes its own values inside its pod template. For example, to put the NATS servers and nats-box on different nodes:
Code
For Postgres, postgresql.cnpg.spec.affinity replaces everything the chart generates from postgresql.cnpg.nodeSelector, tolerations and affinity.
Workloads that ignore global
These don't read the global nodeSelector, tolerations or affinity. Set their own keys. Dragonfly and Kratos don't read global.imagePullSecrets either; see Pull secrets.
| Workload | Its own keys |
|---|---|
| Dragonfly | dragonfly.nodeSelector, tolerations, affinity, imagePullSecrets, priorityClassName and resources |
| Kratos | kratos.deployment.nodeSelector, tolerations and affinity for the server; kratos.statefulSet.nodeSelector, tolerations and affinity for the courier; kratos.imagePullSecrets and kratos.priorityClassName for both |
| Karmada's own components, if enabled | Each component's own block, such as karmada.apiServer.nodeSelector |
Top-level keys for Monad components
The top-level nodeSelector, tolerations, affinity and imagePullSecrets, outside global, apply to Monad's components only. When nodeSelector, tolerations or affinity is set there, Monad's components use it instead of the global value, while NATS, Postgres, VictoriaMetrics and the pods the operator creates keep reading global. The top-level imagePullSecrets adds to global.imagePullSecrets instead.
A component's own value combines with these top-level keys differently:
- A map merges key by key.
api.nodeSelectoradds its labels to the top-levelnodeSelector, and a label set in both takes the component's value. - A list replaces the top-level list. Setting
api.imagePullSecretsdrops the top-levelimagePullSecretsforapi, thoughglobal.imagePullSecretsstill applies.
Use global unless you need this behavior.
Spread across zones and nodes
The NATS servers spread across zones and nodes, and the Postgres instances across zones; CloudNativePG itself prefers to keep instances on separate nodes. The spreads are preferences (ScheduleAnyway), so a single-zone or single-node cluster still schedules every replica. To make a spread a requirement, add these settings under your existing nats and postgresql keys:
Code
Do this only where your zones can always satisfy it; otherwise a replica stays Pending. Each map is keyed by topology key. A key you add becomes another constraint and needs its own maxSkew and whenUnsatisfiable: Kubernetes has no default for either and rejects the constraint without them. Setting the map to {} removes nothing. To replace the Postgres constraints entirely, write the list under postgresql.cnpg.spec.topologySpreadConstraints.
Pods the operator creates
The operator creates the pods that run your pipelines, the controllers for scheduled inputs, each organization's alert service, and a Dragonfly cache for each pipeline that needs one. Their settings go under operator.managedPods. Their environment goes under operator.pipelineEnv:
Code
nodeSelector,tolerationsandaffinityfall back to theirglobalvalues. The top-level keys outsideglobaldon't reach these pods.priorityClassNamehas no fallback, so these pods have no PriorityClass unless you set one.- Their pull secrets, and the
default-pull-secretfallback, are in Pull secrets. pipelineEnvsets environment variables on the main container of each pipeline pod:allfor every pipeline pod, orinput,output,transformandenrichmentfor one component type, which wins overall. Values are plain strings, so quote numbers and booleans, and don't put secrets there.
The operator reads these settings at startup, and a change restarts it. To see what it applies, read its ConfigMap:
Code
Priority classes
The chart creates two PriorityClasses so that application pods cannot preempt the datastores:
| Tier | Default value | Pods |
|---|---|---|
app | 10000 | Monad's components, VictoriaMetrics vmselect, vminsert and vmalert, and the Karmada box, bootstrap and register pods if Karmada is enabled |
critical | 10000000 | The NATS servers, Postgres and VictoriaMetrics vmstorage |
The classes are named monad-<namespace>-<value> and monad-critical-<namespace>-<value>, so several installs can share a cluster. Because the value is part of the name, changing global.monad.priorityClasses.app.value or critical.value creates a new class and rolls the pods onto it.
nats-box, Dragonfly, Kratos and the pods the operator creates are in neither tier. Set dragonfly.priorityClassName, kratos.priorityClassName and operator.managedPods.priorityClassName to give them one. A workload's own priorityClassName, such as api.priorityClassName, wins over its tier.
To use PriorityClasses you already have, name them and turn off creation, under your existing global key. The install then needs no permission to create PriorityClasses:
Code
Give your critical class the higher value. The chart checks the order only for classes it creates, and fails the render if app is not below critical. With create: false, a tier without a name gets no PriorityClass.
Resources
Set a component's requests and limits under its resources key. Each key merges into the chart's default for that workload, so setting requests.cpu keeps the default memory request and limit. The top-level resources fills only the fields a component's own defaults leave unset, which in practice is limits.cpu; set requests and memory limits under each component.
Code
Defaults for every workload
| Workload | Resources key | Requests (CPU, memory) | Limits |
|---|---|---|---|
api | api.resources | 50m, 64Mi | 512Mi memory |
api migrations (init container) | api.initContainers | 10m, 80Mi | 256Mi memory |
api-grpc | api-grpc.resources | 50m, 64Mi | 512Mi memory |
eventhandler | eventhandler.resources | 50m, 64Mi | 512Mi memory |
http-input | http-input.resources | 50m, 80Mi | 1Gi memory |
l4-input | l4-input.resources | 50m, 80Mi | 1Gi memory |
mcp | mcp.resources | 20m, 64Mi | 256Mi memory |
docs | docs.resources | 10m, 100Mi | 256Mi memory |
ui | ui.resources | 20m, 64Mi | 256Mi memory |
operator | operator.resources | 10m, 160Mi | 256Mi memory |
partition-manager (CronJob) | partition-manager.cronjobs.partition-manager.resources | 10m, 80Mi | 256Mi memory |
quota-service-cron | quota-service-cron.resources | 5m, 16Mi | 128Mi memory |
cisa-kev-data-ingest | cisa-kev-data-ingest.resources | 10m, 64Mi | 128Mi memory |
gadb-data-ingest | gadb-data-ingest.resources | 100m, 128Mi | 640Mi memory |
community-transform-ingest | community-transform-ingest.resources | 10m, 64Mi | 128Mi memory |
| Postgres, each instance | postgresql.cnpg.resources | 10m, 512Mi | None |
| NATS server | nats.container.merge.resources | None | 2 CPU |
| NATS config reloader | nats.reloader.merge.resources | None | None |
nats-box | nats.natsBox.container.merge.resources | None | None |
VictoriaMetrics vmstorage, vmselect, vminsert, vmalert | victoriametrics.<component>.resources | The VictoriaMetrics operator's defaults | The operator's defaults |
| Dragonfly | dragonfly.resources | 100m, 1Gi | 1 CPU, 1Gi |
| Kratos server and courier | kratos.deployment.resources | None | None |
| Kratos migrations (init container) | kratos.kratos.automigration.resources | None | None |
Notes on some of these:
http-inputandl4-inputsetGOMEMLIMITto 900MiB. Keep<component>.env.GOMEMLIMIT.valuebelow the memory limit when you change it.api.initContainersis a list, so setting it replaces the whole list. Copy themigrationsentry from the chart's values (helm show values) and change itsresources.kratos.deployment.resourcessizes the courier as well as the server.kratos.statefulSet.resourceshas no effect.- With
resourcesunset, the VictoriaMetrics operator applies its own defaults. See its configuration reference for the values your operator version uses.
Most of Monad's containers set a memory limit but no CPU limit, and a few containers set nothing. If your namespace has a ResourceQuota on CPU or memory, Kubernetes rejects a pod whose containers don't set the resource the quota covers. Set those values here, or give the namespace a LimitRange with defaults.
Replicas and autoscaling
api, api-grpc, docs, http-input, l4-input, mcp and ui autoscale between 2 and 10 replicas. http-input and l4-input add a replica at an average of 400m CPU per pod; the others scale on CPU at 80% of their request, so raising their CPU request also raises the load at which they scale out. Change the range with autoscaling.minReplicas and autoscaling.maxReplicas, for every component or under one component. The components with autoscaling off run a fixed number of replicas, set by their replicaCount.
Pipeline pods
You do not set requests or limits for the pods the operator creates. The operator sets them, to defaults sized from production usage:
| Pod | Requests (CPU, memory) | Limits |
|---|---|---|
| Input | 80m, 100Mi | 4 CPU, 1Gi |
| Output | 10m, 150Mi | 2 CPU, 1Gi |
| Enrichment | 20m, 70Mi | 2 CPU, 1Gi |
| Transform | 10m, 10Mi | 1 CPU, 1Gi |
| Schema detection, a sidecar on pipeline pods | 200m, 80Mi | 1 CPU, 256Mi |
| A pipeline's Dragonfly cache | 25m, 32Mi | 1 CPU, 1536Mi |
| Scheduled-input controller | 10m, 128Mi | 200m CPU, 128Mi |
| Alert service, one per organization | 3m, 20Mi | 1 CPU, 512Mi |
A few connectors request more or less than their type's default. For how many pipeline pods to expect, see Cluster Sizing. To have the operator fit each pod's requests to its own usage, see Right-size requests with VerticalPodAutoscalers.
Right-size requests with VerticalPodAutoscalers
The operator can create a VerticalPodAutoscaler (VPA) for each of these Deployments:
- Each pipeline node: inputs, transforms, enrichments and outputs. Scheduled inputs and the scheduled-input controller run as Jobs and get none.
- Each organization's alert service.
- The Dragonfly cache of each pipeline that has one.
A VPA changes requests only, so limits stay as in the table above. It resizes a running pod in place where it can, and recreates the pod otherwise (update mode InPlaceOrRecreate).
- Pipeline nodes and alert services: a request follows the pod's usage, below or above its default, up to the container's limit and to what the largest node matching the pods' node selector can hold, less room for system pods. The schema detection sidecar keeps its default requests.
- Dragonfly: a request stays between its default and its limit.
The cluster needs:
- The Vertical Pod Autoscaler 1.5.0 or later, where
InPlaceOrRecreateis on by default: its CRDs, the Metrics Server it reads usage from, and all three of its components. The recommender sizes requests from usage, the updater resizes running pods, and the admission controller sets the requests on each new pod. - In-place pod resize (
InPlacePodVerticalScaling), on by default from Kubernetes 1.33.
Turn it on under operator. The operator also needs to manage VPAs in the release namespace and to read nodes, which it lists to cap each request:
Code
role.extraRules adds to the operator's Role. clusterrole gives the operator a ClusterRole and ClusterRoleBinding, so the account that runs helm install also needs permission to create both and to read nodes itself. If an Argo CD AppProject restricts cluster-scoped resources, allow both kinds.
Install the VPA first
Without the VPA's CRDs or this RBAC, the operator's reconciles fail: pipeline status stops updating, and pipelines that use a Dragonfly cache stop picking up changes.
Setting MONAD_MANAGED_VPA back to "false" leaves the VPAs in place, and they keep resizing pods. Delete them yourself, such as with kubectl delete vpa --all -n monad when nothing else in the namespace uses a VPA.
Storage
The volume sizes and StorageClasses for NATS, Postgres and VictoriaMetrics are listed in the storage table. Set them before the first install.