AWS SQS S3 CloudTrail
Ingests AWS CloudTrail log files from S3 via Amazon Simple Queue Service (SQS).
Sync Type: Streaming
Details
The Amazon SQS S3 CloudTrail connector is a specialized variant of the SQS S3 connector for ingesting AWS CloudTrail log files delivered to S3. It handles the CloudTrail-specific file structure (a top-level Records array containing many events per file) so events are delivered correctly through the data pipeline.
Use this connector when ingesting native AWS CloudTrail S3 deliveries. Use the standard SQS S3 connector for all other S3-based data sources.
How It Works
- S3 Event Generation: When CloudTrail delivers a log file to the S3 bucket, S3 generates an object-created event notification
- SQS Message Receipt: The notification is sent to the configured SQS queue as a JSON message
- Event Processing: The connector polls the SQS queue, receives S3 event messages, and parses them
- File Download: For each S3 event, the connector downloads the referenced object from S3
- Data Processing: Downloaded files are decompressed (gzip) and parsed as JSON
- Record Extraction: Individual records are extracted and sent to the data pipeline
- Cleanup: Successfully processed SQS messages are deleted from the queue
One queue per pipeline
Do not point two pipelines at the same SQS queue, whether they share this input or each have their own input with the same queue URL. The pipelines compete for the queue's messages: each message is delivered to only one of them and then deleted, so each pipeline silently receives only part of the data. To send this data to several destinations, add more outputs to a single pipeline, or give each pipeline its own queue. See Using One Input in Multiple Pipelines for both patterns.
Prerequisites
- An AWS account with existing S3 buckets and SQS queue, or permissions to create them
- AWS CloudTrail trail configured to deliver log files to the S3 bucket
- An IAM role that the platform can assume, with appropriate permissions to access both SQS and S3, or using static AWS credentials
- S3 bucket configured to send event notifications to the SQS queue
- If the CloudTrail log files are encrypted with SSE-KMS, the role also needs
kms:Decrypton that key. If the queue uses SSE-KMS, the role also needskms:Decrypton the queue's key, and the key policy must allow S3 (s3.amazonaws.com) to usekms:GenerateDataKeyandkms:Decryptso S3 can publish to the queue. - Network connectivity between the platform and AWS
Setup Instructions
Step 1: Create the IAM role Monad will assume
Monad authenticates to your AWS account by assuming an IAM role that you create. The role needs two policies: a trust policy that lets Monad assume it, and a permissions policy that grants read access to the queue and the bucket. If your Monad deployment runs on a dedicated cluster outside AWS, use the OIDC trust policy in Authentication Methods instead of the one below. To use static access keys instead of a role, see the same section.
- Find your Monad organization ID.
It is the UUID in the Monad UI address bar while you are viewing your organization (
/organizations/<organization-id>), and it is also shown on the detail panel of any pipeline node. - Sign in to the AWS Management Console at https://console.aws.amazon.com/iam and click Roles, then Create role.
- Under Trusted entity type, select Custom trust policy, delete the policy JSON, and paste the following.
Replace
{your-monad-org-id}with your Monad organization ID.
Code
Monad always presents your organization ID as the external ID when it assumes the role, so this condition means the role can only be assumed on behalf of your organization.
Keep the external ID condition
Monad does not check whether your trust policy requires the external ID, so a role without the sts:ExternalId condition still works.
Without that condition, anyone with a Monad organization who knows your role ARN could configure Monad to assume your role.
Always include the condition when you use Monad Cloud; it is only unnecessary for self-hosted deployments where Monad runs in your own environment.
- Attach a permissions policy scoped to your queue and bucket. Replace the placeholder values in braces with your values. If you have not created the queue or bucket yet, you can create the role now with a placeholder policy and come back to this step after Step 3.
Code
- Name the role (e.g.,
monad-sqs-s3-input) and click Create role. - Note the role ARN. You will enter it as Role ARN when configuring the connector.
Step 2: Create S3 Bucket (if needed)
Create an S3 bucket where CloudTrail will deliver log files (skip if you already have one):
- Sign in to the AWS Management Console at https://console.aws.amazon.com/s3.
- Click the AWS region drop-down list next to the account name in the upper right and select the desired region.
- Under General Purpose Buckets, click the Create Bucket button.
- Under General Configuration, enter a name for the bucket (e.g.,
my-cloudtrail-logs-bucket). - Under Object Ownership, ensure ACLs Disabled is selected.
- Leave the remaining default selections unchanged and click the Create Bucket button.
- Note the bucket name and ARN for later use.
Step 3: Create SQS Queue
Set up an SQS queue to receive S3 bucket event notifications:
- Sign in to the AWS Management Console at https://console.aws.amazon.com/sqs.
- Under Get Started, click the Create Queue button.
- Under Details, enter a name for the queue (e.g.,
cloudtrail-events-queue). - Under Configuration:
- For Visibility Timeout: Leave the default; Monad sets its own visibility timeout on every receive, so the queue setting is overridden
- For Message Retention Period: Enter
7days - Set Receive message wait time to
20seconds for long polling
- Under Access Policy > Choose Method, select the Advanced radio button.
- Delete the entire policy JSON and copy/paste the following policy:
Code
- Replace the placeholders:
{AWS_REGION}with the AWS region (e.g.,us-east-1){AWS_ACCOUNT_ID}with the 12-digit AWS account ID{QUEUE_NAME}with the queue name{BUCKET_NAME}with the S3 bucket name
- Click the Create Queue button.
- Note the queue URL and ARN for later use.
Step 4: Configure S3 Event Notifications
Configure the S3 bucket to send event notifications to the SQS queue:
- Sign in to the AWS Management Console at https://console.aws.amazon.com/s3.
- Under General Purpose Buckets, click on the source bucket.
- Click the Properties tab at the top.
- Locate the Event Notifications section and click Create Event Notification.
- Configure the event notification:
- Event Name: Enter a descriptive name (e.g.,
cloudtrail-notification) - Event Types: Select the checkbox for All object create events
- Prefix/Suffix: Optionally filter by object key prefix or suffix. CloudTrail delivers to
AWSLogs/<account-id>/CloudTrail/<region>/— set the prefix toAWSLogs/(or narrower) if the bucket is shared with non-CloudTrail uploads. - Destination: Select the radio button for SQS queue
- SQS Queue: Choose the queue from the dropdown or enter the queue ARN
- Event Name: Enter a descriptive name (e.g.,
- Click the Save Changes button.
Step 5: Verify Event Notifications
AWS sends a test notification to the queue after creation. To verify:
- Navigate to the SQS queue in the AWS Console.
- Under Details, locate Messages available, which should have a value of 1.
- This confirms that S3 can successfully send notifications to the queue.
Note: The test message is an s3:TestEvent payload, not a real S3 event. The connector ignores it once polling starts.
Step 6: Test the Configuration
Before configuring the connector in the platform, verify that the AWS setup is working:
- Wait for the next CloudTrail delivery (typically within 5–15 minutes), or push a CloudTrail-shaped log file to the bucket manually for faster feedback. A manually uploaded file must be gzip-compressed like a real CloudTrail delivery; an uncompressed file fails to process.
- Check the SQS queue to confirm a message was received:
- Navigate to the SQS queue in the AWS Console
- Under Details, verify Messages available has increased
- Use the IAM Policy Simulator (optional) to test permissions:
- Go to https://policysim.aws.amazon.com/home/index.jsp
- Select the IAM role
- Test the required S3 and SQS actions listed in the permissions section
Configuration
Settings
This connector is purpose-built for native AWS CloudTrail S3 deliveries, so the file format (json), compression (gzip), and record location are fixed internally — they're not exposed as user settings.
| Setting | Type | Required | Default | Description |
|---|---|---|---|---|
queue_url | string | Yes | - | The URL of the SQS queue to poll for messages. |
role_arn | string | No | - | The ARN of the IAM role to assume for accessing AWS resources. Required when not using static credentials. See Authentication Methods. |
region | string | Yes | us-east-1 | The AWS region to use. Allowed values: us-east-1, us-east-2, us-west-1, us-west-2, af-south-1, ap-east-1, ap-south-1, ap-northeast-1, ap-northeast-2, ap-northeast-3, ap-southeast-1, ap-southeast-2, ca-central-1, ca-west-1, cn-north-1, cn-northwest-1, eu-central-1, eu-west-1, eu-west-2, eu-west-3, eu-north-1, eu-south-1, me-south-1, sa-east-1, us-gov-east-1, us-gov-west-1. |
use_fips | bool | No | false | Route STS, SQS, and S3 traffic to FIPS 140-2 endpoints. Required for AWS GovCloud. Only available in US regions (us-east-1/2, us-west-1/2, ca-central-1, ca-west-1, us-gov-east-1, us-gov-west-1). |
uses_sns | bool | No | false | Uses AWS SNS in the middle of S3 and SQS for fan-out usecases. |
with_metadata | bool | No | false | Whether to include S3 object metadata in the output. |
chunking_mode | string | No | by_size | How CloudTrail's top-level Records array is split before emission. By size keeps the historical 1MB-per-message behavior. Per record emits one Records[] element per message as a bare event (envelope dropped). Allowed values: by_size (By Size (1MB)), per_record (Per Record). |
exclude_digest_files | bool | No | false | When enabled, skips CloudTrail digest files (files under /CloudTrail-Digest/ paths). These files contain hash signatures rather than events. |
Secrets
| Secret | Type | Required | Description |
|---|---|---|---|
access_key | secret | No | The Access Key from AWS to authenticate with AWS |
secret_key | secret | No | The Secret Key from AWS to authenticate with AWS |
Files are automatically decompressed (gzip) and parsed as JSON; no user configuration is needed for either.
Important Notes
-
Event Filtering: Only
ObjectCreated:*events are processed.s3:TestEventmessages are skipped and deleted. -
Non-JSON messages: Messages whose body is not valid JSON are deleted without processing.
-
JSON messages that are not S3 event notifications: A JSON message without a
Recordsarray fails to decode and is not deleted; it is redelivered (about every 3 minutes) until it expires or moves to a dead-letter queue. Configure a dead-letter queue on your queue so such messages do not retry indefinitely. -
Order Preservation: The order of events within the
Recordsarray is preserved -
File Processing: Each S3 object is downloaded, decompressed (gzip), and parsed as JSON
-
Record Extraction: Individual records are extracted from files and sent downstream
-
Delivery is at-least-once: A message is deleted only after all records from its objects have been emitted. If processing fails, the message stays in the queue and is retried. An object that takes longer than about 2 minutes to process is abandoned and retried, and records already emitted from it can repeat.
-
Visibility timeout: Monad sets its own visibility timeout on every receive, overriding the queue's configured value.
-
Digest files: When Exclude Digest Files is enabled, notifications for digest files are deleted from the queue without processing.
-
Shared queues: If several pipelines or inputs read the same queue, the messages are split between them; each message goes to only one consumer. To send the same bucket's log files to more than one pipeline, give each pipeline its own queue, either with separate S3 event notifications per prefix or with SNS fan-out to one queue per pipeline (enable Uses SNS).
-
Output Shape: Depends on
Chunking Mode.by_size(default): each emitted message is the CloudTrail envelope withRecordschunked into ~1MB batches. All other top-level fields (_monad_metadata, etc.) are preserved on every chunk.per_record: each emitted message is a single bare CloudTrail event (the envelope is dropped). IfWith Metadatais enabled,_monad_metadatais added to each event individually rather than to the envelope.
With
by_size, downstream consumers that want one event per message can still flatten with ajqtransform:.Records[]. -
Metadata Preservation: If
With Metadatais enabled, the_monad_metadatafield (containing bucket information) is added at the top level of each emitted message
Related Articles
- AWS CloudTrail Documentation
- Creating a Trail (delivers logs to S3)
- CloudTrail Log File Examples
- Amazon S3 Event Notifications
- Amazon SQS Documentation
- S3 Event Message Structure
- AWS IAM Role Chaining and Temporary Credentials
Sync frequency
This input consumes from the source queue or subscription, ingesting records at whatever rate the source delivers them rather than on a fixed poll. See Input Sync Frequency for details.
Community transforms
The Monad Transforms Repository has ready-made transforms for this input. Import one into your organization, then add it to a pipeline after this input.
| Format | Transform |
|---|---|
| OCSF v1.9.0 | AWS CloudTrail (SQS/S3) to OCSF v1.9.0 |
| ECS v9.5.0 | AWS CloudTrail (SQS/S3) to ECS v9.5.0 |
| CEF | AWS CloudTrail (SQS/S3) to CEF |
CEF transforms emit an intermediate record; add the CEF Serializer transform after one to produce the CEF:0 string.