AWS Access for Connectors
Read this page if your pipelines use AWS connectors, such as S3, CloudTrail, GuardDuty, Security Hub, Inspector or the S3 output, and you want them to assume an IAM role rather than use an access key. Skip it otherwise. AWS connectors authenticate in one of two ways, and only one of them needs anything from your Helm install:
- Static credentials. You give the connector an access key and secret key, stored as Monad secrets. Nothing on this page applies.
- IAM role assumption. The connector pod assumes a role in your account. The pod needs an AWS identity of its own, which a self-managed install has to provide. That is what this page covers.
The connector side, which fields to fill in and the trust policy templates, is in AWS Connectors. This page covers the cluster side.
How role assumption works
A connector configured with a Role ARN points at the target role: the role that holds the permissions for the data you want. How Monad reaches it depends on one environment variable, MONAD_SECURE_ACCESS_DELEGATE_ROLE:
MONAD_SECURE_ACCESS_DELEGATE_ROLE | Behavior |
|---|---|
| Empty or unset (the default) | Single hop. The pod's own identity assumes the connector's Role ARN directly. |
| An IAM role ARN | Two hops. The pod assumes that delegate role first, then assumes the connector's Role ARN with the delegate's credentials. |
Self-managed installs normally use the single hop, which is the default: leave the variable unset and the pods use whatever AWS identity you give them. The two-hop path exists for Monad Cloud, where a Monad-owned delegate role sits between the workload and your account.
Either way, Monad passes your organization ID as the ExternalId when it assumes the target role, so that role's trust policy must expect it. Your organization ID is the UUID in the console's address bar while you view your organization (/organizations/<organization-id>). It is also shown in the details of any pipeline node. In a sub-org, use the sub-org's ID; see Which organization ID to use.
Give the pods an AWS identity
Monad does not create an AWS identity for you. The connectors and the api read credentials through the AWS SDK's default credential chain, so any mechanism that provides credentials that way works. On EKS the usual route is IAM roles for service accounts (IRSA): annotate the Kubernetes service accounts with a role ARN and let the pods federate into it. On a cluster outside EKS the annotation does nothing by itself; something has to inject the token and the AWS environment variables, such as a copy of the Amazon EKS Pod Identity Webhook that you run yourself.
Three service accounts make AWS calls, and different things create them:
| Service account | Why it needs an AWS identity | Created by |
|---|---|---|
api | Connection tests run in the api process | The Helm chart; annotate it with api.serviceAccount.annotations |
inputs, outputs | Every AWS connector pod runs as one of these, whether it is a long-running Deployment or a scheduled Job | The pipeline operator, at runtime; see the warning below |
Create the connector service accounts yourself
The operator creates its service accounts (inputs, outputs, transforms, enrichments and alerts) only if they do not already exist. It creates them with no annotations and never updates them. The chart has no value for them, because the operator creates the connector pods, not the chart. Create the ones you need yourself, annotated, and the operator leaves them alone.
Create the connector service accounts in the Monad namespace:
Code
Then annotate the api service account in your values file:
Code
Restart running pods after adding annotations
IRSA credentials are injected when a pod is created, so annotating a service account does nothing for pods that are already running. If you add these annotations to a live install, delete the affected connector pods afterward so they are recreated with the credentials in place.
EKS Pod Identity works as well, through associations you create for the monad namespace and the api, inputs and outputs service accounts. The associated role then takes the place of the web-identity role below, with the trust policy EKS Pod Identity requires. If your cluster runs the Crossplane AWS provider, the chart can create those associations as PodIdentityAssociation resources: set api.podIdentityAssociations to a list of entries, each with name, clusterName, region, roleArn and serviceAccount. Set it under api, not at the top level, where every component would render the same resources.
IAM setup in your account
For a single-hop install you create two kinds of role:
- A web-identity role: the one in the annotations above. Its trust policy federates your cluster's OIDC provider, conditioned on the service account subjects (
system:serviceaccount:monad:api,system:serviceaccount:monad:inputsandsystem:serviceaccount:monad:outputs) and thests.amazonaws.comaudience. If your account has no IAM OIDC provider for the cluster yet, create that first. AWS Connectors has the full trust policy template. - The connector target roles: one per connector, or whatever granularity you want. Each trusts the web-identity role and requires your Monad organization ID as the
ExternalId. These hold the data permissions, and their ARNs go in the connector's Role ARN field in the console.
A target role's trust policy looks like this:
Code
The web-identity role also needs permission to assume those target roles. This is required when a target role is in a different account from the web-identity role, and harmless when they share one:
Code
Use a delegate role
If Monad support asks you to route through a delegate role, set it on both the api and the operator. The api uses it for connection tests. The operator copies it into the input and output pods and the scheduled Jobs it creates, so these two settings cover every connector:
Code
If your values file already has an api block, such as the service account annotation above, add env under it rather than adding a second api key: a second copy of a top-level key silently replaces the first.
Verify
Check that the annotation reached the connector service accounts, and that a running connector pod picked up the projected credentials:
Code
If a connector fails with an STS AccessDenied, the error names the hop that failed: failed to assume Role A is the delegate hop and failed to assume Role B is the target role. A failure on the target role usually means its trust policy is missing the web-identity role as principal, or the ExternalId condition does not match your organization ID.