Copy page

AWS

EKS Audit Logs

Ingests Kubernetes API server audit logs from Amazon EKS clusters via CloudWatch Logs. These audit logs provide detailed records of all API requests made to the Kubernetes control plane, essential for compliance, security monitoring, and troubleshooting.

Sync Type: Incremental

Overview

Amazon EKS control plane audit logging captures all API requests made to your Kubernetes API server. This connector polls CloudWatch Logs to retrieve these audit events, maintaining an incremental cursor to ensure only new logs are ingested on subsequent syncs.

How It Works

1. AWS EKS publishes Kubernetes API server audit logs to CloudWatch Logs under the log group /aws/eks/<cluster-name>/cluster
2. Monad polls this log group at regular intervals using the CloudWatch Logs FilterLogEvents API
3. An incremental timestamp cursor tracks the last processed log, ensuring only new events are fetched on subsequent runs
4. Audit logging must be enabled on the EKS cluster. On the first run, Monad ingests everything retained in the log group (or from the specified backfill start time, if set)

Requirements

Before using this connector, ensure the following prerequisites are met:

1. Enable Control Plane Audit Logging on Your EKS Cluster

By default, control plane audit logging is disabled on EKS clusters. You must explicitly enable it for this connector to receive data.

Using AWS Console:

1. Navigate to Amazon EKS in the AWS Console
2. Select your cluster
3. Go to the Observability tab
4. Under Control plane logging, toggle the Audit log type to On
5. Click Save changes

Using AWS CLI:

Code
aws eks update-cluster-config \
  --name <cluster-name> \
  --region <region> \
  --logging '{"clusterLogging":[{"types":["audit"],"enabled":true}]}'

After enabling, AWS will create the CloudWatch log group /aws/eks/<cluster-name>/cluster and begin publishing audit logs within a few minutes.

2. Grant IAM Permissions

The role or credentials used must have permissions to read from the CloudWatch log group containing the audit logs.

Minimum Required Permissions:

Code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "logs:DescribeLogGroups",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "logs:FilterLogEvents",
      "Resource": "arn:aws:logs:<region>:<account-id>:log-group:/aws/eks/<cluster-name>/cluster:*"
    }
  ]
}

Replace:

• <region> with your AWS region (e.g., us-east-1)
• <account-id> with your 12-digit AWS account ID
• <cluster-name> with your EKS cluster name

For Role ARN Authentication:

1. Create or update an IAM role in your AWS account
2. Add the above permissions to the role
3. Update the trust relationship to allow Monad to assume it (see AWS Authentication Methods)

For Static Credentials:

1. Create or use an IAM user with programmatic access
2. Attach the above policy
3. Generate or use existing Access Key ID and Secret Access Key

Configuration

Settings

Setting	Type	Required	Default	Description
cluster_name	string	Yes	-	The name of the EKS cluster. Used to construct the CloudWatch log group /aws/eks/<cluster-name>/cluster.
region	string	Yes	us-east-1	The AWS region to use. Allowed values: us-east-1, us-east-2, us-west-1, us-west-2, af-south-1, ap-east-1, ap-south-1, ap-northeast-1, ap-northeast-2, ap-northeast-3, ap-southeast-1, ap-southeast-2, ca-central-1, ca-west-1, cn-north-1, cn-northwest-1, eu-central-1, eu-west-1, eu-west-2, eu-west-3, eu-north-1, eu-south-1, me-south-1, sa-east-1, us-gov-east-1, us-gov-west-1.
role_arn	string	No	-	The ARN of the IAM role to assume for accessing AWS resources.
use_synthetic_data	bool	No	false	Generate synthetic data for testing, instead of connecting to a real data source. Defaults to an hourly cron schedule for cron-based inputs.
backfill_start_time	date	No	-	Date to start fetching data from, in ISO 8601 format. If not specified, all retained history of the cluster's log group is fetched on the first sync. All syncs thereafter are incremental.
ingestion_lag_seconds	number	No	10	Absorbs CloudWatch ingestion lag so late-arriving audit events are not dropped. Minimum 5s (unset defaults to 10s), maximum 14 days. See the input's docs for how to measure your cluster's ingestion lag and pick a value. See Ingestion Lag Time.

Secrets

Secret	Type	Required	Description
access_key	secret	No	The Access Key from AWS to authenticate with AWS
secret_key	secret	No	The Secret Key from AWS to authenticate with AWS

Authentication Note: Choose either Role ARN (recommended for security) or static credentials (Access Key + Secret Key). See AWS Authentication Methods for detailed setup instructions.

Details

Incremental Syncing

This connector maintains state using an incremental timestamp cursor:

• First sync: If Backfill Start Time is not set, Monad ingests all retained history in the cluster's log group. Set a Backfill Start Time to limit how far back the first sync reads.
• Subsequent syncs: Only logs with timestamps after the last checkpoint are processed
• Checkpointing: The cursor is updated after each batch of logs is successfully processed
• Restart behavior: If a sync is interrupted, the next run resumes from the last checkpointed timestamp

Log Group and Stream Structure

EKS publishes audit logs to:

• Log Group: /aws/eks/<cluster-name>/cluster
• Log Streams: Multiple streams within the log group, each prefixed with a timestamp

Monad queries the entire log group using FilterLogEvents, which automatically searches across all streams.

Filtering and Limitations

• No filtering: Monad reads every log stream in the /aws/eks/<cluster-name>/cluster log group, not only the audit streams. If you have enabled other control-plane log types (API server, authenticator, controller manager, scheduler), EKS writes them to the same log group and Monad ingests them too. Enable only the Audit log type if you want audit events only, or filter on LogStreamName (audit streams start with kube-apiserver-audit) downstream.
• Log retention: By default, CloudWatch Logs retains logs indefinitely. You can set a retention policy in the AWS Console to manage costs.
• Rate limiting: CloudWatch Logs API has rate limits (typically 1,000 requests per second). For clusters with high API activity, monitor CloudWatch Logs metrics to ensure limits are not exceeded.

Audit Log Format

Each record is a CloudWatch Logs event wrapper, not a bare audit event. The Kubernetes audit event is a JSON string in the Message field:

Code
{
  "EventId": "38251636462291832457981738261432987115483520172837847040",
  "IngestionTime": 1715340003210,
  "LogStreamName": "kube-apiserver-audit-0a1b2c3d4e5f60718293a4b5c6d7e8f9",
  "Message": "{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Metadata\",\"auditID\":\"5f0e7c2a-9b1d-4e3f-8a6c-2d4b1e9f7a30\",\"stage\":\"ResponseComplete\",\"requestURI\":\"/api/v1/namespaces/default/pods\",\"verb\":\"list\",\"user\":{\"username\":\"system:serviceaccount:kube-system:default\"},\"sourceIPs\":[\"10.0.12.34\"],\"objectRef\":{\"resource\":\"pods\",\"namespace\":\"default\",\"apiVersion\":\"v1\"},\"responseStatus\":{\"code\":200}}",
  "Timestamp": 1715340001987
}

To work with the audit event's fields, parse Message as JSON downstream (for example, with a jq transform: .Message | fromjson). The audit event itself conforms to the Kubernetes audit event specification and includes:

• verb: API action (e.g., create, get, list, delete)
• user: User identity and authentication info
• sourceIPs: Source IP address(es) of the request
• objectRef: Reference to the Kubernetes object affected
• requestObject/responseObject: Request and response payloads (if enabled in audit policy)
• annotations: Additional metadata

For details on the audit log format, see the Kubernetes Audit Event Specification.

Ingestion Lag Time

Some audit events reach CloudWatch late — after the moment they actually occurred on the cluster. For EKS this is usually a few seconds, but delays stretch into minutes during control-plane scaling, API server throttling, or CloudWatch slowdowns. If Monad has already moved past that moment, those late events would be missed.

The Ingestion Lag (seconds) is a hold-back window. On every poll, Monad only emits events that are at least this many seconds old; events newer than now minus the lag are left for a later poll. This gives CloudWatch time to ingest late-arriving events before Monad moves past their timestamp, at the cost of adding this much latency to every event.

How to choose a value:

• Leave it unset to use the default: 10 seconds. Works for most clusters under normal load.
• Raise it if your cluster experiences frequent control-plane scaling, API server throttling, or has a history of CloudWatch delivery issues.
• Allowed range: 5 seconds to 1209600 seconds (14 days).

Measuring ingestion lag in your cluster

If you want a data-driven number instead of guessing, you can measure your cluster's actual audit-log delay directly in the AWS Console.

Steps (Console → CloudWatch → Logs → Logs Analytics):

1. Select the log group /aws/eks/<cluster-name>/cluster.
2. Set the time range to 7 days.
3. Paste and run:

Code
fields @ingestionTime - @timestamp AS lag_ms, @logStream
| filter @logStream like /kube-apiserver-audit/
| stats max(lag_ms) AS max_lag_ms,
        avg(lag_ms) AS avg_lag_ms,
        pct(lag_ms, 99) AS p99_lag_ms
        BY bin(1h)
| sort @timestamp desc

Read the results:

• Look at p99_lag_ms — this is the delay 99% of your audit events land inside. Divide by 1000 and round up to get a safe value in seconds.
• If max_lag_ms regularly spikes into the minutes (typically during control-plane scaling or throttling), use that spike size instead.
• Steady, low-traffic clusters usually stay well under the default; no change needed.

Rule of thumb: start with the default. Only raise it if the query above tells you delays are larger, or if you notice missing events after a known incident. Setting it to hours or days "just in case" adds needless work without improving reliability.

Rate Limits

The connector respects CloudWatch Logs API rate limits:

Limit	Value	Notes
API Requests per second	1,000	Shared across all API operations in your account
Concurrent requests	Limited by account quota	Monad uses a single worker thread per input
Log event payload	≤ 1 MB per batch	CloudWatch automatically batches smaller events

Source: AWS CloudWatch Logs API Reference

Monitoring: Use CloudWatch Metrics to monitor:

• IncomingLogEvents — Number of log events published
• IncomingBytes — Volume of log data published

Related Articles

• Amazon EKS Control Plane Audit Logging
• Kubernetes Audit Event Specification
• AWS CloudWatch Logs FilterLogEvents API

Sync frequency

By default this input polls approximately every 10 seconds, with each sync beginning after the previous one completes. A cron schedule configured on the pipeline overrides this cadence. See Input Sync Frequency for details.