AWS SQS S3
Enables seamless ingestion of S3 objects via Amazon Simple Queue Service (SQS) notifications.
Sync Type: Streaming
Details
The Amazon SQS S3 connector allows you to automatically process files uploaded to one or more S3 buckets by listening for S3 event notifications through a single SQS queue. Whenever files are uploaded to your S3 buckets, S3 publishes event notifications to an SQS queue, which this connector monitors. The connector then automatically downloads and processes the files.
The connector polls an Amazon SQS queue for S3 event notifications, downloads the referenced S3 objects, processes them according to your configured format and compression settings, and extracts records for further processing in your data pipeline.
How It Works
- S3 Event Generation: When objects are created in your S3 bucket, S3 generates event notifications (only
ObjectCreated:*events are processed; other event types in a message are skipped) - SQS Message Receipt: These events are sent to your configured SQS queue as JSON messages
- Event Processing: The connector polls the SQS queue, receives S3 event messages, and parses them
- File Download: For each S3 event, the connector downloads the referenced object from S3
- Data Processing: Downloaded files are decompressed (if needed) and parsed according to your format settings
- Record Extraction: Individual records are extracted and sent to your data pipeline
- Cleanup: Successfully processed SQS messages are deleted from the queue
One queue per pipeline
Do not point two pipelines at the same SQS queue, whether they share this input or each have their own input with the same queue URL. The pipelines compete for the queue's messages: each message is delivered to only one of them and then deleted, so each pipeline silently receives only part of the data. To send this data to several destinations, add more outputs to a single pipeline, or give each pipeline its own queue. See Using One Input in Multiple Pipelines for both patterns.
Prerequisites
- An AWS account with existing S3 buckets and SQS queue, or permissions to create them
- An IAM role that Monad can assume, with permissions to read the SQS queue and the S3 bucket, or static AWS credentials with the same permissions
- S3 bucket configured to send event notifications to your SQS queue
- If the bucket uses SSE-KMS, the role also needs
kms:Decrypton that key. If the queue uses SSE-KMS, the role also needskms:Decrypton the queue's key, and the key policy must allow S3 (s3.amazonaws.com) to usekms:GenerateDataKeyandkms:Decryptso S3 can publish to the queue. - Network connectivity between Monad and AWS
Setup Instructions
Step 1: Create the IAM role Monad will assume
Monad authenticates to your AWS account by assuming an IAM role that you create. The role needs two policies: a trust policy that lets Monad assume it, and a permissions policy that grants read access to the queue and the bucket. If your Monad deployment runs on a dedicated cluster outside AWS, use the OIDC trust policy in Authentication Methods instead of the one below. To use static access keys instead of a role, see the same section.
- Find your Monad organization ID.
It is the UUID in the Monad UI address bar while you are viewing your organization (
/organizations/<organization-id>), and it is also shown on the detail panel of any pipeline node. - Sign in to the AWS Management Console at https://console.aws.amazon.com/iam and click Roles, then Create role.
- Under Trusted entity type, select Custom trust policy, delete the policy JSON, and paste the following.
Replace
{your-monad-org-id}with your Monad organization ID.
Code
Monad always presents your organization ID as the external ID when it assumes the role, so this condition means the role can only be assumed on behalf of your organization.
Keep the external ID condition
Monad does not check whether your trust policy requires the external ID, so a role without the sts:ExternalId condition still works.
Without that condition, anyone with a Monad organization who knows your role ARN could configure Monad to assume your role.
Always include the condition when you use Monad Cloud; it is only unnecessary for self-hosted deployments where Monad runs in your own environment.
- Attach a permissions policy scoped to your queue and bucket.
Replace
{AWS_REGION},{AWS_ACCOUNT_ID},{QUEUE_NAME}, and{BUCKET_NAME}with your values. If you have not created the queue or bucket yet, you can create the role now with a placeholder policy and come back to this step after Step 3.
Code
- Name the role (e.g.,
monad-sqs-s3-input) and click Create role. - Note the role ARN. You will enter it as Role ARN when configuring the connector.
Step 2: Create S3 Bucket (if needed)
Create an S3 bucket to store your source data:
- Sign in to the AWS Management Console at https://console.aws.amazon.com/s3.
- Click the AWS region drop-down list next to your name in the upper right and select the desired region.
- Under General Purpose Buckets, click the Create Bucket button.
- Under General Configuration, enter a name for your bucket (e.g.,
my-data-source-bucket). - Under Object Ownership, ensure ACLs Disabled is selected.
- Leave the remaining default selections unchanged and click the Create Bucket button.
- Note the bucket name and ARN for later use.
Step 3: Create SQS Queue
Set up an SQS queue to receive S3 bucket event notifications:
- Sign in to the AWS Management Console at https://console.aws.amazon.com/sqs.
- Under Get Started, click the Create Queue button.
- Under Details, enter a name for your queue (e.g.,
s3-events-queue). - Under Configuration:
- For Visibility Timeout: Leave the default; Monad sets its own visibility timeout on every receive, so the queue setting is overridden
- For Message Retention Period: Enter
7days - Set Receive message wait time to
20seconds for long polling
- Under Access Policy > Choose Method, select the Advanced radio button.
- Delete the entire policy JSON and copy/paste the following policy:
Code
- Replace the placeholders:
{AWS_REGION}with your AWS region (e.g.,us-east-1){AWS_ACCOUNT_ID}with your 12-digit AWS account ID{QUEUE_NAME}with your queue name{BUCKET_NAME}with your S3 bucket name
- Click the Create Queue button.
- Note the queue URL and ARN for later use.
Step 4: Configure S3 Event Notifications
Configure your S3 bucket to send event notifications to the SQS queue:
- Sign in to the AWS Management Console at https://console.aws.amazon.com/s3.
- Under General Purpose Buckets, click on your source bucket.
- Click the Properties tab at the top.
- Locate the Event Notifications section and click Create Event Notification.
- Configure the event notification:
- Event Name: Enter a descriptive name (e.g.,
object-created-notification) - Event Types: Select the checkbox for All object create events
- Prefix/Suffix: Optionally filter by object key prefix or suffix
- Destination: Select the radio button for SQS queue
- SQS Queue: Choose your queue from the dropdown or enter the queue ARN
- Event Name: Enter a descriptive name (e.g.,
- Click the Save Changes button.
Step 5: Verify Event Notifications
AWS sends a test notification to the queue after creation. To verify:
- Navigate to your SQS queue in the AWS Console.
- Under Details, locate Messages available, which should have a value of 1.
- This confirms that S3 can successfully send notifications to your queue.
Step 6: Test the Configuration
Before configuring the connector in Monad, verify that the AWS setup is working:
- Upload a test file to your S3 bucket to trigger an event notification.
- Check the SQS queue to confirm a message was received:
- Navigate to your SQS queue in the AWS Console
- Under Details, verify Messages available has increased
- Use the IAM Policy Simulator (optional) to test permissions:
- Go to https://policysim.aws.amazon.com/home/index.jsp
- Select your IAM role
- Test the S3 and SQS actions granted by the permissions policy in Step 1
Note: The s3:TestEvent message that S3 sends when you create the notification is detected and skipped automatically, so you do not need to purge it.
Configuration
Settings
| Setting | Type | Required | Default | Description |
|---|---|---|---|---|
queue_url | string | Yes | - | The URL of the SQS queue to poll for messages. |
role_arn | string | No | - | The ARN of the IAM role to assume for accessing AWS resources. Required when not using static credentials. See Authentication Methods. |
region | string | Yes | us-east-1 | The AWS region to use. Allowed values: us-east-1, us-east-2, us-west-1, us-west-2, af-south-1, ap-east-1, ap-south-1, ap-northeast-1, ap-northeast-2, ap-northeast-3, ap-southeast-1, ap-southeast-2, ca-central-1, ca-west-1, cn-north-1, cn-northwest-1, eu-central-1, eu-west-1, eu-west-2, eu-west-3, eu-north-1, eu-south-1, me-south-1, sa-east-1, us-gov-east-1, us-gov-west-1. |
use_fips | bool | No | false | Route STS, SQS, and S3 traffic to FIPS 140-2 endpoints. Required for AWS GovCloud. Only available in US regions (us-east-1/2, us-west-1/2, ca-central-1, ca-west-1, us-gov-east-1, us-gov-west-1). |
uses_sns | bool | No | false | Uses AWS SNS in the middle of S3 and SQS for fan-out usecases. |
with_metadata | bool | No | false | Whether to include S3 object metadata in the output. |
compression | string | Yes | - | Compression format of the S3 objects. Allowed values: gzip, none, auto. |
format | string | Yes | json | File format of the S3 objects. Allowed values: csv, delimited, json, jsonl, wsv. |
record_location | string | No | - | Location of the record in the object. Applies only for JSON objects. Leave empty for the entire record. See Record Location for syntax and examples. |
key_filter | one of | Yes | - | Optional filter applied to the S3 object key (case-insensitive). Choose "None" to process every key, or "Filter" to include/exclude keys matching a substring, prefix, or suffix. See Key Filter. |
Secrets
| Secret | Type | Required | Description |
|---|---|---|---|
access_key | secret | No | The Access Key from AWS to authenticate with AWS |
secret_key | secret | No | The Secret Key from AWS to authenticate with AWS |
Files are automatically decompressed before processing based on the compression setting in your connector configuration.
For JSON objects, leaving Record Location empty treats the whole document as the record source: a top-level JSON array is split into one record per array element, and a single object becomes one record.
With Metadata adds only the source bucket name to each record, as _monad_metadata.bucket.
When Uses SNS is enabled, the SNS subscription to the queue must deliver the standard SNS envelope (raw message delivery turned off). If the setting does not match how messages arrive (SNS envelope without Uses SNS, or raw delivery with Uses SNS), Monad cannot decode the messages, and they stay in the queue and are redelivered until they expire or move to a dead-letter queue.
Key Filter
Filters S3 object keys before the connector fetches them. Useful when the SQS queue receives notifications for objects this connector should never process (for example, manifests or a noisy prefix). Matching is case-insensitive on both the key and the configured value.
Key Filter does not split a queue between pipelines
Keys skipped by the Key Filter are deleted from the queue along with their message. If several pipelines read the same queue, each message goes to only one of them, so a key that one pipeline filters out is never seen by the pipeline that wanted it. To route different prefixes of one bucket to different pipelines, give each pipeline its own queue (see Important Notes).
Key Filter is a discriminated union on the type field — the empty case is {"type": "none"} rather than an empty {mode, operator, value} object, so the per-field rules only apply when filtering is actually requested:
| Field | Type | Required | Description |
|---|---|---|---|
type | string | Yes | none passes every key through; filter enables the rules in filter below |
filter | object | When type is filter | The match parameters described below. Omit when type is none. |
When type is filter, the nested filter object takes:
| Field | Type | Required | Description |
|---|---|---|---|
mode | string | Yes | include processes only matching keys; exclude skips matching keys |
operator | string | Yes | One of contains, ends_with, starts_with |
value | string | Yes | The substring (or suffix/prefix) to match against the S3 object key |
Examples:
- No filtering —
{ "type": "none" } - Process only gzipped JSON files —
{ "type": "filter", "filter": { "mode": "include", "operator": "ends_with", "value": ".json.gz" } } - Skip a noisy prefix —
{ "type": "filter", "filter": { "mode": "exclude", "operator": "starts_with", "value": "AWSLogs/123456789012/internal/" } }
When the field is omitted entirely (or set to {"type": "none"}), every S3 object referenced by the queue is processed.
Example S3 Event Message
When S3 sends an event notification to SQS, the message body contains JSON like this:
Code
The connector parses these events and downloads the referenced S3 objects for processing.
Important Notes
- Event Filtering: Only
ObjectCreated:*events are processed.s3:TestEventmessages are skipped and deleted. - Non-JSON messages: Messages whose body is not valid JSON are deleted without processing.
- JSON messages that are not S3 event notifications: A JSON message without a
Recordsarray fails to decode and is not deleted; it is redelivered (about every 3 minutes) until it expires or moves to a dead-letter queue. Configure a dead-letter queue on your queue so such messages do not retry indefinitely. - File Processing: Each S3 object is downloaded, decompressed, and parsed according to your format settings
- Record Extraction: Individual records are extracted from files and sent downstream
- Delivery is at-least-once: A message is deleted only after all records from its objects have been emitted. If processing fails, the message stays in the queue and is retried. An object that takes longer than about 2 minutes to process is abandoned and retried, and records already emitted from it can repeat.
- Visibility timeout: Monad sets its own visibility timeout on every receive, overriding the queue's configured value.
- Shared queues: If several pipelines or inputs read the same queue, the messages are split between them; each message goes to only one consumer. To send the same bucket's objects to more than one pipeline, give each pipeline its own queue, either with separate S3 event notifications per prefix or with SNS fan-out to one queue per pipeline (enable Uses SNS).
Related Articles
- Amazon S3 Event Notifications
- Amazon SQS Documentation
- S3 Event Message Structure
- AWS IAM Role Chaining and Temporary Credentials
Sync frequency
This input consumes from the source queue or subscription, ingesting records at whatever rate the source delivers them rather than on a fixed poll. See Input Sync Frequency for details.