# Scheduling and Resources Read this page if your cluster decides where pods run by node labels, taints, affinity or PriorityClasses, or limits namespaces with quotas. It covers where each of Monad's pods runs, the PriorityClasses the chart creates, and the requests and limits of every workload. Any of these settings can change with an upgrade. :::note[Merge snippets into your values file] Each snippet gives full paths from the top-level key. Your values file already has some of these keys, such as `global` and `nats` in every example file. Add a snippet's settings under the key you already have: a second copy of a top-level key silently replaces the first. See [Install](/docs/guides/self-managed/install). ::: ## Place pods on your nodes Set the chart-wide placement under `global`: ```yaml global: nodeSelector: node-pool: monad tolerations: - key: dedicated operator: Equal value: monad effect: NoSchedule ``` `global.affinity` takes a standard Kubernetes affinity the same way. These values reach Monad's components (`api`, `ui`, `operator` and the others in the first rows of [Defaults for every workload](#defaults-for-every-workload)), the NATS servers and the `nats-box` pod, Postgres, VictoriaMetrics, the [pods the operator creates](#pods-the-operator-creates), and the Karmada box, bootstrap and register pods if Karmada is enabled. Dragonfly and Kratos don't read them; see [Workloads that ignore global](#workloads-that-ignore-global). Unless you set `global.affinity`, NATS and Postgres prefer nodes labelled `dedicated=stateful`. Unless you set `global.tolerations`, they tolerate a `dedicated=stateful:NoSchedule` taint. :::warning[Quote node selector values] Quote a node selector value that YAML reads as a boolean or a number, such as `"true"` or `"1"`. The chart refuses to render an unquoted one in `global.nodeSelector` or `operator.managedPods.nodeSelector`, and the Kubernetes API rejects one anywhere else when you install. ::: ### Give one workload its own value A workload's own value and the `global` value combine like this: - A workload's own `nodeSelector`, `tolerations` or `affinity` replaces the `global` one whole. The two are not merged, so a workload with its own `nodeSelector` needs every label it should match, not only the extra one. - `false` removes the field for that workload, for example `api.tolerations: false`. - An empty `{}` or `[]` counts as unset, so the `global` value applies. - Pull secrets add up: a workload gets `global.imagePullSecrets` plus its own list. [Images and Registries](/docs/guides/self-managed/images#pull-secrets) lists the pull secrets each pod gets. Where each workload's own values go: | Workload | `nodeSelector`, `tolerations`, `affinity` | PriorityClass | | --- | --- | --- | | Monad components | Under the component, such as `api.nodeSelector` | `.priorityClassName` | | NATS servers | `nats.podTemplate.merge.spec` | The same `spec` | | `nats-box` | `nats.natsBox.podTemplate.merge.spec` | The same `spec`; none by default | | Postgres | `postgresql.cnpg.nodeSelector`, `tolerations`, `affinity` | `postgresql.cnpg.priorityClassName` | | VictoriaMetrics | Under each of `victoriametrics.vmstorage`, `vmselect`, `vminsert` and `vmalert` | Under each of the four | | Pods the operator creates | `operator.managedPods` | `operator.managedPods.priorityClassName` | | Karmada box, bootstrap and register pods, if enabled | Under each of `karmada.box`, `karmada.bootstrap` and `karmada.register` | Under each of the three | NATS takes its own values inside its pod template. For example, to put the NATS servers and `nats-box` on different nodes: ```yaml nats: podTemplate: merge: spec: nodeSelector: node-pool: stateful natsBox: podTemplate: merge: spec: nodeSelector: node-pool: monad ``` For Postgres, `postgresql.cnpg.spec.affinity` replaces everything the chart generates from `postgresql.cnpg.nodeSelector`, `tolerations` and `affinity`. ### Workloads that ignore global These don't read the `global` `nodeSelector`, `tolerations` or `affinity`. Set their own keys. Dragonfly and Kratos don't read `global.imagePullSecrets` either; see [Pull secrets](/docs/guides/self-managed/images#pull-secrets). | Workload | Its own keys | | --- | --- | | Dragonfly | `dragonfly.nodeSelector`, `tolerations`, `affinity`, `imagePullSecrets`, `priorityClassName` and `resources` | | Kratos | `kratos.deployment.nodeSelector`, `tolerations` and `affinity` for the server; `kratos.statefulSet.nodeSelector`, `tolerations` and `affinity` for the courier; `kratos.imagePullSecrets` and `kratos.priorityClassName` for both | | Karmada's own components, if enabled | Each component's own block, such as `karmada.apiServer.nodeSelector` | ### Top-level keys for Monad components The top-level `nodeSelector`, `tolerations`, `affinity` and `imagePullSecrets`, outside `global`, apply to Monad's components only. When `nodeSelector`, `tolerations` or `affinity` is set there, Monad's components use it instead of the `global` value, while NATS, Postgres, VictoriaMetrics and the pods the operator creates keep reading `global`. The top-level `imagePullSecrets` adds to `global.imagePullSecrets` instead. A component's own value combines with these top-level keys differently: - A map merges key by key. `api.nodeSelector` adds its labels to the top-level `nodeSelector`, and a label set in both takes the component's value. - A list replaces the top-level list. Setting `api.imagePullSecrets` drops the top-level `imagePullSecrets` for `api`, though `global.imagePullSecrets` still applies. Use `global` unless you need this behavior. ### Spread across zones and nodes The NATS servers spread across zones and nodes, and the Postgres instances across zones; CloudNativePG itself prefers to keep instances on separate nodes. The spreads are preferences (`ScheduleAnyway`), so a single-zone or single-node cluster still schedules every replica. To make a spread a requirement, add these settings under your existing `nats` and `postgresql` keys: ```yaml nats: podTemplate: topologySpreadConstraints: topology.kubernetes.io/zone: whenUnsatisfiable: DoNotSchedule postgresql: cnpg: topologySpreadConstraints: topology.kubernetes.io/zone: whenUnsatisfiable: DoNotSchedule ``` Do this only where your zones can always satisfy it; otherwise a replica stays `Pending`. Each map is keyed by topology key. A key you add becomes another constraint and needs its own `maxSkew` and `whenUnsatisfiable`: Kubernetes has no default for either and rejects the constraint without them. Setting the map to `{}` removes nothing. To replace the Postgres constraints entirely, write the list under `postgresql.cnpg.spec.topologySpreadConstraints`. ## Pods the operator creates The operator creates the pods that run your pipelines, the controllers for scheduled inputs, each organization's alert service, and a Dragonfly cache for each pipeline that needs one. Their settings go under `operator.managedPods`. Their environment goes under `operator.pipelineEnv`: ```yaml operator: managedPods: nodeSelector: node-pool: pipelines priorityClassName: pipelineEnv: all: MONAD_INTERNAL_HOSTS: "10.0.0.0/8" ``` - `nodeSelector`, `tolerations` and `affinity` fall back to their `global` values. The top-level keys outside `global` don't reach these pods. - `priorityClassName` has no fallback, so these pods have no PriorityClass unless you set one. - Their pull secrets, and the `default-pull-secret` fallback, are in [Pull secrets](/docs/guides/self-managed/images#pull-secrets). - `pipelineEnv` sets environment variables on the main container of each pipeline pod: `all` for every pipeline pod, or `input`, `output`, `transform` and `enrichment` for one component type, which wins over `all`. Values are plain strings, so quote numbers and booleans, and don't put secrets there. The operator reads these settings at startup, and a change restarts it. To see what it applies, read its ConfigMap: ```bash kubectl get configmap operator-config -n monad -o yaml ``` ## Priority classes The chart creates two PriorityClasses so that application pods cannot preempt the datastores: | Tier | Default value | Pods | | --- | --- | --- | | `app` | 10000 | Monad's components, VictoriaMetrics `vmselect`, `vminsert` and `vmalert`, and the Karmada box, bootstrap and register pods if Karmada is enabled | | `critical` | 10000000 | The NATS servers, Postgres and VictoriaMetrics `vmstorage` | The classes are named `monad--` and `monad-critical--`, so several installs can share a cluster. Because the value is part of the name, changing `global.monad.priorityClasses.app.value` or `critical.value` creates a new class and rolls the pods onto it. `nats-box`, Dragonfly, Kratos and the pods the operator creates are in neither tier. Set `dragonfly.priorityClassName`, `kratos.priorityClassName` and `operator.managedPods.priorityClassName` to give them one. A workload's own `priorityClassName`, such as `api.priorityClassName`, wins over its tier. To use PriorityClasses you already have, name them and turn off creation, under your existing `global` key. The install then needs no permission to create PriorityClasses: ```yaml global: monad: priorityClasses: create: false app: name: critical: name: ``` Give your critical class the higher value. The chart checks the order only for classes it creates, and fails the render if `app` is not below `critical`. With `create: false`, a tier without a `name` gets no PriorityClass. ## Resources Set a component's requests and limits under its `resources` key. Each key merges into the chart's default for that workload, so setting `requests.cpu` keeps the default memory request and limit. The top-level `resources` fills only the fields a component's own defaults leave unset, which in practice is `limits.cpu`; set requests and memory limits under each component. ```yaml api: resources: requests: cpu: 500m memory: 512Mi limits: memory: 1Gi victoriametrics: vmstorage: resources: requests: cpu: 250m memory: 1Gi limits: memory: 2Gi dragonfly: resources: requests: cpu: 500m memory: 2Gi limits: memory: 2Gi ``` ### Defaults for every workload | Workload | Resources key | Requests (CPU, memory) | Limits | | --- | --- | --- | --- | | `api` | `api.resources` | 50m, 64Mi | 512Mi memory | | `api` migrations (init container) | `api.initContainers` | 10m, 80Mi | 256Mi memory | | `api-grpc` | `api-grpc.resources` | 50m, 64Mi | 512Mi memory | | `eventhandler` | `eventhandler.resources` | 50m, 64Mi | 512Mi memory | | `http-input` | `http-input.resources` | 50m, 80Mi | 1Gi memory | | `l4-input` | `l4-input.resources` | 50m, 80Mi | 1Gi memory | | `mcp` | `mcp.resources` | 20m, 64Mi | 256Mi memory | | `docs` | `docs.resources` | 10m, 100Mi | 256Mi memory | | `ui` | `ui.resources` | 20m, 64Mi | 256Mi memory | | `operator` | `operator.resources` | 10m, 160Mi | 256Mi memory | | `partition-manager` (CronJob) | `partition-manager.cronjobs.partition-manager.resources` | 10m, 80Mi | 256Mi memory | | `quota-service-cron` | `quota-service-cron.resources` | 5m, 16Mi | 128Mi memory | | `cisa-kev-data-ingest` | `cisa-kev-data-ingest.resources` | 10m, 64Mi | 128Mi memory | | `gadb-data-ingest` | `gadb-data-ingest.resources` | 100m, 128Mi | 640Mi memory | | `community-transform-ingest` | `community-transform-ingest.resources` | 10m, 64Mi | 128Mi memory | | Postgres, each instance | `postgresql.cnpg.resources` | 10m, 512Mi | None | | NATS server | `nats.container.merge.resources` | None | 2 CPU | | NATS config reloader | `nats.reloader.merge.resources` | None | None | | `nats-box` | `nats.natsBox.container.merge.resources` | None | None | | VictoriaMetrics `vmstorage`, `vmselect`, `vminsert`, `vmalert` | `victoriametrics..resources` | The VictoriaMetrics operator's defaults | The operator's defaults | | Dragonfly | `dragonfly.resources` | 100m, 1Gi | 1 CPU, 1Gi | | Kratos server and courier | `kratos.deployment.resources` | None | None | | Kratos migrations (init container) | `kratos.kratos.automigration.resources` | None | None | Notes on some of these: - `http-input` and `l4-input` set `GOMEMLIMIT` to 900MiB. Keep `.env.GOMEMLIMIT.value` below the memory limit when you change it. - `api.initContainers` is a list, so setting it replaces the whole list. Copy the `migrations` entry from the chart's values (`helm show values`) and change its `resources`. - `kratos.deployment.resources` sizes the courier as well as the server. `kratos.statefulSet.resources` has no effect. - With `resources` unset, the VictoriaMetrics operator applies its own defaults. See its [configuration reference](https://docs.victoriametrics.com/operator/configuration/) for the values your operator version uses. Most of Monad's containers set a memory limit but no CPU limit, and a few containers set nothing. If your namespace has a ResourceQuota on CPU or memory, Kubernetes rejects a pod whose containers don't set the resource the quota covers. Set those values here, or give the namespace a LimitRange with defaults. ### Replicas and autoscaling `api`, `api-grpc`, `docs`, `http-input`, `l4-input`, `mcp` and `ui` autoscale between 2 and 10 replicas. `http-input` and `l4-input` add a replica at an average of 400m CPU per pod; the others scale on CPU at 80% of their request, so raising their CPU request also raises the load at which they scale out. Change the range with `autoscaling.minReplicas` and `autoscaling.maxReplicas`, for every component or under one component. The components with autoscaling off run a fixed number of replicas, set by their `replicaCount`. ### Pipeline pods You do not set requests or limits for the pods the operator creates. The operator sets them, to defaults sized from production usage: | Pod | Requests (CPU, memory) | Limits | | --- | --- | --- | | Input | 80m, 100Mi | 4 CPU, 1Gi | | Output | 10m, 150Mi | 2 CPU, 1Gi | | Enrichment | 20m, 70Mi | 2 CPU, 1Gi | | Transform | 10m, 10Mi | 1 CPU, 1Gi | | Schema detection, a sidecar on pipeline pods | 200m, 80Mi | 1 CPU, 256Mi | | A pipeline's Dragonfly cache | 25m, 32Mi | 1 CPU, 1536Mi | | Scheduled-input controller | 10m, 128Mi | 200m CPU, 128Mi | | Alert service, one per organization | 3m, 20Mi | 1 CPU, 512Mi | A few connectors request more or less than their type's default. For how many pipeline pods to expect, see [Cluster Sizing](/docs/guides/cluster-sizing). To have the operator fit each pod's requests to its own usage, see [Right-size requests with VerticalPodAutoscalers](#right-size-requests-with-verticalpodautoscalers). #### Right-size requests with VerticalPodAutoscalers The operator can create a VerticalPodAutoscaler (VPA) for each of these Deployments: - Each pipeline node: inputs, transforms, enrichments and outputs. Scheduled inputs and the scheduled-input controller run as Jobs and get none. - Each organization's alert service. - The Dragonfly cache of each pipeline that has one. A VPA changes requests only, so limits stay as in the table above. It resizes a running pod in place where it can, and recreates the pod otherwise (update mode `InPlaceOrRecreate`). - **Pipeline nodes and alert services**: a request follows the pod's usage, below or above its default, up to the container's limit and to what the largest node matching the pods' node selector can hold, less room for system pods. The schema detection sidecar keeps its default requests. - **Dragonfly**: a request stays between its default and its limit. The cluster needs: - The [Vertical Pod Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler) 1.5.0 or later, where `InPlaceOrRecreate` is on by default: its CRDs, the [Metrics Server](https://github.com/kubernetes-sigs/metrics-server) it reads usage from, and all three of its components. The recommender sizes requests from usage, the updater resizes running pods, and the admission controller sets the requests on each new pod. - In-place pod resize (`InPlacePodVerticalScaling`), on by default from Kubernetes 1.33. Turn it on under `operator`. The operator also needs to manage VPAs in the release namespace and to read nodes, which it lists to cap each request: ```yaml operator: env: MONAD_MANAGED_VPA: value: "true" rbac: role: extraRules: - apiGroups: ["autoscaling.k8s.io"] resources: ["verticalpodautoscalers"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] clusterrole: extraRules: - apiGroups: [""] resources: ["nodes"] verbs: ["get", "list", "watch"] ``` `role.extraRules` adds to the operator's Role. `clusterrole` gives the operator a ClusterRole and ClusterRoleBinding, so the account that runs `helm install` also needs permission to create both and to read nodes itself. If an Argo CD AppProject restricts cluster-scoped resources, allow both kinds. :::warning[Install the VPA first] Without the VPA's CRDs or this RBAC, the operator's reconciles fail: pipeline status stops updating, and pipelines that use a Dragonfly cache stop picking up changes. ::: Setting `MONAD_MANAGED_VPA` back to `"false"` leaves the VPAs in place, and they keep resizing pods. Delete them yourself, such as with `kubectl delete vpa --all -n monad` when nothing else in the namespace uses a VPA. ### Storage The volume sizes and StorageClasses for NATS, Postgres and VictoriaMetrics are listed in the [storage table](/docs/guides/self-managed/before-you-install#storage). Set them before the first install.