AWS Cloudwatch Logs
Sync Type: Incremental
Overview
Collects log events from AWS CloudWatch Logs to monitor application logs, system logs, and AWS service logs across your environment for security analysis and operational insights.
Functionality
At the start of every sync, Monad discovers all CloudWatch log groups in the specified region (matching the Log Group Name Prefix, if set), so log groups created later are picked up automatically. For each log group, the connector retrieves log events using time-based filtering and maintains state to ensure incremental updates on subsequent runs. Only new log events since the last sync are collected, minimizing duplicates and API calls.
If Backfill Start Time is not set, the first sync of each log group fetches all of its retained history, including log groups discovered later. For log groups with long retention this can be a large one-time ingest; set a Backfill Start Time to limit how far back the input reads.
Requirements
- IAM Role Assumption / Static Credentials
- Example permission to attach to the role/user:
Code
Configuration
The following configuration defines the input parameters. Each field's specifications, such as type, requirements, and descriptions, are detailed below.
Settings
| Setting | Type | Required | Default | Description |
|---|---|---|---|---|
role_arn | string | No | - | The ARN of the IAM role to assume for accessing AWS resources. |
region | string | Yes | us-east-1 | The AWS region to use. Allowed values: us-east-1, us-east-2, us-west-1, us-west-2, af-south-1, ap-east-1, ap-south-1, ap-northeast-1, ap-northeast-2, ap-northeast-3, ap-southeast-1, ap-southeast-2, ca-central-1, ca-west-1, cn-north-1, cn-northwest-1, eu-central-1, eu-west-1, eu-west-2, eu-west-3, eu-north-1, eu-south-1, me-south-1, sa-east-1, us-gov-east-1, us-gov-west-1. |
use_synthetic_data | bool | No | false | Generate synthetic data for testing, instead of connecting to a real data source. Defaults to an hourly cron schedule for cron-based inputs. |
log_group_name_prefix | string | No | - | Log group name prefix to fetch Cloudwatch Logs from. |
backfill_start_time | date | No | - | Date to start fetching data from, in ISO 8601 format. If not specified, all retained history of each matching log group is fetched on the first sync, including log groups discovered later. All syncs thereafter are incremental. |
ingestion_lag_seconds | number | No | 10 | Hold-back window that delays emission of events until they are at least this many seconds old, giving CloudWatch time to ingest late-arriving peers at the same timestamp. Adds this much latency to every event but prevents data loss on late arrivals. Minimum 5s (unset defaults to 10s), maximum 14 days. See the input's docs to size this for your environment. See Ingestion Lag Time. |
Secrets (Static Credentials Only)
| Secret | Type | Required | Description |
|---|---|---|---|
access_key | secret | No | The Access Key from AWS to authenticate with AWS |
secret_key | secret | No | The Secret Key from AWS to authenticate with AWS |
⚠️ Authentication: Choose either Role ARN (recommended) or static credentials. See AWS Authentication Guide for setup instructions.
Ingestion Lag Time
Some log events arrive at CloudWatch late — after the moment they were actually generated. Common reasons include agents that buffer before sending, retries after a network hiccup, and bulk replays from outages. Monad tracks progress by producer timestamp, so if it moves the cursor past a second before a late-arriving event at that second lands, that event is missed.
The Ingestion Lag (seconds) setting solves this by holding events back until they are old enough that CloudWatch has had time to receive any late-arriving peers at the same timestamp. Concretely: on every poll Monad only emits events whose producer timestamp is older than now - IngestionLagSeconds. Anything newer is left for a later poll.
Trade-off: every event is delayed by at least this many seconds before Monad emits it. Pick the smallest value that comfortably covers your environment's worst late arrivals.
How to choose a value:
- Leave it unset to use the default:
10seconds. Works for most environments. - Raise it if you know log delivery to CloudWatch is bursty, retried often, or backfilled after outages.
- Allowed range:
5seconds to1209600seconds (14 days). Setting hours or days adds hours or days of latency to every event.
Measuring ingestion lag in your environment
If you want a data-driven number instead of guessing, you can measure your actual lag directly in the AWS Console.
Steps (Console → CloudWatch → Logs → Logs Analytics):
- Select the log group you plan to ingest.
- Set the time range to 7 days.
- Paste and run:
Code
Read the results:
- Look at
p99_lag_ms— this is the delay 99% of your events land inside. Divide by1000and round up to get a safe value in seconds. This will be your emission latency floor. - If
max_lag_msregularly spikes into the minutes, weigh the reliability cost of missing those spikes against the latency cost of covering them. - Steady, low-traffic log groups usually stay well under the default; no change needed.
Rule of thumb: start with the default. Only raise it if the query above tells you delays are consistently larger, or if you notice missing events after a known incident. Setting it to hours or days "just in case" delays every event by that same amount for no gain in typical operation.
Tip: get fast log groups quickly without slowing them down for the slow ones
A single Ingestion Lag value applies to every log group the input scans, so one bursty log group forces the same delay on every other group in that input. If most of your log groups are timely but a handful (bulk backfills, cross-region agents, mobile-uploaded traces) run minutes late, split them into two inputs:
- Fast input — set LogGroup Name Prefix to the fast, well-behaved log groups (e.g.
/aws/eks/,/aws/lambda/) and keep Ingestion Lag low (default10s). These events land in Monad within seconds. - Slow input — a second copy of this connector pointed at the laggy prefix (e.g.
/aws/backfill/,/aws/vendor-x/) with a larger Ingestion Lag sized from that group's ownp99_lag_ms.
Each input maintains its own cursor state, so the two can run at very different cadences without interfering. The result: real-time visibility for your normal traffic and reliable delivery for the outliers, without paying the slow group's latency on everything.
If you can't cleanly separate groups by prefix, run one input per problem log group with an explicit prefix that matches only it.
Related Articles
Sample Record
Code
Sync frequency
By default this input polls approximately every 10 seconds, with each sync beginning after the previous one completes. A cron schedule configured on the pipeline overrides this cadence. See Input Sync Frequency for details.