Copy page

AWS

Cloudwatch Logs

Sync Type: Incremental

Overview

Collects log events from AWS CloudWatch Logs to monitor application logs, system logs, and AWS service logs across your environment for security analysis and operational insights.

Functionality

On initialization, Monad discovers all CloudWatch log groups in the specified region. For each log group, the connector retrieves log events using time-based filtering and maintains state to ensure incremental updates on subsequent runs. Only new log events since the last sync are collected, minimizing duplicates and API calls.

Requirements

• IAM Role Assumption / Static Credentials
• Example permission to attach to the role/user:

Code
{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "",
			"Effect": "Allow",
			"Action": [
        "logs:DescribeLogGroups",
        "logs:FilterLogEvents"
			],
			"Resource": "*"
		}
	]
}

Configuration

The following configuration defines the input parameters. Each field's specifications, such as type, requirements, and descriptions, are detailed below.

Settings

Setting	Type	Required	Description
Region	string	Yes	The AWS region where Cloudwatch is enabled.
Role ARN	string	Yes	The ARN of the IAM role to assume for accessing Cloudwatch.
Backfill Start Time	string	No	The date to start fetching data from. If not specified, no past records will be fetched.
LogGroup Name Prefix	string	No	Only fetch log groups whose name starts with this prefix.
Ingestion Lag (seconds)	number	No	Hold-back window: events are only emitted once their producer timestamp is at least this many seconds old, giving CloudWatch time to ingest late-arriving peers at the same timestamp before Monad locks the cursor past it. Adds this much latency to every event. Minimum 5s (unset defaults to 10s), maximum 14 days (1209600 seconds). See the Ingestion Lag Time section below for how to size this in your environment.

Secrets (Static Credentials Only)

Setting	Type	Required	Description
Access Key	string	Conditional	AWS Access Key ID
Secret Key	string	Conditional	AWS Secret Access Key

⚠️ Authentication: Choose either Role ARN (recommended) or static credentials. See AWS Authentication Guide for setup instructions.

Ingestion Lag Time

Some log events arrive at CloudWatch late — after the moment they were actually generated. Common reasons include agents that buffer before sending, retries after a network hiccup, and bulk replays from outages. Monad tracks progress by producer timestamp, so if it moves the cursor past a second before a late-arriving event at that second lands, that event is missed.

The Ingestion Lag (seconds) setting solves this by holding events back until they are old enough that CloudWatch has had time to receive any late-arriving peers at the same timestamp. Concretely: on every poll Monad only emits events whose producer timestamp is older than now - IngestionLagSeconds. Anything newer is left for a later poll.

Trade-off: every event is delayed by at least this many seconds before Monad emits it. Pick the smallest value that comfortably covers your environment's worst late arrivals.

How to choose a value:

• Leave it unset to use the default: 10 seconds. Works for most environments.
• Raise it if you know log delivery to CloudWatch is bursty, retried often, or backfilled after outages.
• Allowed range: 5 seconds to 1209600 seconds (14 days). Setting hours or days adds hours or days of latency to every event.

Measuring ingestion lag in your environment

If you want a data-driven number instead of guessing, you can measure your actual lag directly in the AWS Console.

Steps (Console → CloudWatch → Logs → Logs Analytics):

1. Select the log group you plan to ingest.
2. Set the time range to 7 days.
3. Paste and run:

Code
fields @ingestionTime - @timestamp AS lag_ms, @logStream
| stats max(lag_ms) AS max_lag_ms,
        avg(lag_ms) AS avg_lag_ms,
        pct(lag_ms, 99) AS p99_lag_ms
        BY bin(1h)
| sort @timestamp desc

Read the results:

• Look at p99_lag_ms — this is the delay 99% of your events land inside. Divide by 1000 and round up to get a safe value in seconds. This will be your emission latency floor.
• If max_lag_ms regularly spikes into the minutes, weigh the reliability cost of missing those spikes against the latency cost of covering them.
• Steady, low-traffic log groups usually stay well under the default; no change needed.

Rule of thumb: start with the default. Only raise it if the query above tells you delays are consistently larger, or if you notice missing events after a known incident. Setting it to hours or days "just in case" delays every event by that same amount for no gain in typical operation.

Tip: get fast log groups quickly without slowing them down for the slow ones

A single Ingestion Lag value applies to every log group the input scans, so one bursty log group forces the same delay on every other group in that input. If most of your log groups are timely but a handful (bulk backfills, cross-region agents, mobile-uploaded traces) run minutes late, split them into two inputs:

1. Fast input — set LogGroup Name Prefix to the fast, well-behaved log groups (e.g. /aws/eks/, /aws/lambda/) and keep Ingestion Lag low (default 10s). These events land in Monad within seconds.
2. Slow input — a second copy of this connector pointed at the laggy prefix (e.g. /aws/backfill/, /aws/vendor-x/) with a larger Ingestion Lag sized from that group's own p99_lag_ms.

Each input maintains its own cursor state, so the two can run at very different cadences without interfering. The result: real-time visibility for your normal traffic and reliable delivery for the outliers, without paying the slow group's latency on everything.

If you can't cleanly separate groups by prefix, run one input per problem log group with an explicit prefix that matches only it.

Related Articles

• AWS Cloudwatch Logs User Guide
• Cloudwatch Logs API Reference

Sample Record

Code
{
  "EventId": "19290253317859809123455432178906645308301437257699635872",
  "IngestionTime": 1761939524706,
  "LogStreamName": "EC2-Log-Stream",
  "Message": "User login activity",
  "Timestamp": 1761939586709
}

Sync frequency

By default this input polls approximately every 10 seconds, with each sync beginning after the previous one completes. A cron schedule configured on the pipeline overrides this cadence. See Input Sync Frequency for details.