Copy page

AWS

SQS

Enables seamless ingestion of messages from Amazon Simple Queue Service (SQS) for data processing and integration.

Sync Type: Streaming

Details

The Amazon SQS connector allows you to collect messages from any Amazon SQS queue for further processing in your data pipeline. This connector is versatile and can process messages from any source that publishes to SQS, making it ideal for integrating multiple AWS services or custom applications with your platform.

The connector long-polls an Amazon SQS queue (20-second wait), drains it until it is empty, and then checks again about every 10 seconds. Each message is deleted from the queue after it has been emitted. Polling, wait time and visibility timeout are managed by Monad and are not configurable.

Each record is the raw message body, exactly as it was sent. Message attributes and other SQS metadata are not included, and SNS envelopes are not unwrapped (a queue subscribed to an SNS topic without raw message delivery produces the SNS JSON envelope as the record).

Prerequisites

• An AWS account with an existing SQS queue or permissions to create one
• An IAM role that your platform can assume, or static AWS credentials (access key and secret key), with appropriate permissions to access the SQS queue (see Authentication Methods)
• If the queue uses SSE-KMS, the role or user also needs kms:Decrypt on the queue's KMS key
• Network connectivity between your platform and AWS

Setup Instructions

Step 1: Create or Configure an SQS Queue (if needed)

1. Sign in to the AWS Management Console.
2. Navigate to the Amazon SQS service.
3. If creating a new queue:
   • Click Create queue.
   • Select Standard or FIFO queue type. Monad processes several messages concurrently and delivers at-least-once, so a FIFO queue does not give strict ordering or exactly-once delivery through this input.
   • Enter a name for your queue.
   • Configure basic settings:
     • Configure message retention, maximum message size and other settings as needed. Monad sets its own visibility timeout on every receive, so the queue's default visibility timeout is overridden.
   • Click Create queue.
4. Note the queue URL for later use in the connector configuration.

Step 2: Create IAM Role for Access

• Follow the IAM Role Setup Guide in the main AWS documentation.
• Make sure to attach a policy with the following permissions to your user/role:

  Code
  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "sqs:ReceiveMessage",
          "sqs:DeleteMessage",
          "sqs:GetQueueAttributes"
        ],
        "Resource": "[SQS-QUEUE-ARN]"
      }
    ]
  }

Configuration

The following configuration defines the input parameters for the SQS connector:

Settings

Setting	Type	Required	Default	Description
queue_url	string	Yes	-	The URL of the SQS queue to poll for messages.
role_arn	string	No	-	The ARN of the IAM role to assume for accessing AWS resources.
region	string	Yes	us-east-1	The AWS region to use. Allowed values: us-east-1, us-east-2, us-west-1, us-west-2, af-south-1, ap-east-1, ap-south-1, ap-northeast-1, ap-northeast-2, ap-northeast-3, ap-southeast-1, ap-southeast-2, ca-central-1, ca-west-1, cn-north-1, cn-northwest-1, eu-central-1, eu-west-1, eu-west-2, eu-west-3, eu-north-1, eu-south-1, me-south-1, sa-east-1, us-gov-east-1, us-gov-west-1.
use_fips	bool	No	false	Route STS, SQS, and S3 traffic to FIPS 140-2 endpoints. Required for AWS GovCloud. Only available in US regions (us-east-1/2, us-west-1/2, ca-central-1, ca-west-1, us-gov-east-1, us-gov-west-1).

Secrets

Secret	Type	Required	Description
access_key	secret	No	The Access Key from AWS to authenticate with AWS
secret_key	secret	No	The Secret Key from AWS to authenticate with AWS

Important Notes

• Message Deletion: Messages are automatically deleted from the queue after successful processing. If processing fails, messages become visible again after the visibility timeout period.

• Delivery is at-least-once: A message is deleted only after it has been emitted. If the input is interrupted between emitting a message and deleting it, the message is delivered again and the record can repeat.

• FIFO Queues: Monad processes up to four messages at a time concurrently, so records from a FIFO queue are not guaranteed to be emitted in message-group order, and FIFO does not give exactly-once delivery through this input.

• Shared queues: If several pipelines or inputs read the same queue, the messages are split between them; each message goes to only one consumer. To deliver the same messages to more than one pipeline, use SNS fan-out to one queue per pipeline.

• Visibility timeout: Monad sets its own visibility timeout on every receive, overriding the queue's configured value.

IAM Permissions Required

The IAM role specified in Role ARN must have the following permissions:

Code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes"
      ],
      "Resource": "arn:aws:sqs:[REGION]:[ACCOUNT-ID]:[QUEUE-NAME]"
    }
  ]
}

Troubleshooting

No Messages Being Received

1. Queue Verification:

   • Verify the SQS queue exists and contains messages
   • Use the AWS Console or CLI to view the queue's approximate number of messages
   • Send a test message manually to verify queue functionality

2. Permission Issues:

   • Confirm the IAM role has the required permissions (sqs:ReceiveMessage, sqs:DeleteMessage, sqs:GetQueueAttributes)
   • Check that the trust relationship is correctly configured to allow the platform to assume the role
   • Verify the role ARN is correctly specified in the connector configuration

3. Region Configuration:

   • Ensure the Region specified in the connector configuration matches the region where the SQS queue is deployed

Connectivity Issues

1. Network Connectivity:

   • Ensure the platform has network connectivity to AWS SQS in the specified region
   • Check for any firewall rules or security groups that might be blocking the connection

2. Service Quotas:

   • Review AWS service quotas for SQS to ensure you're not hitting any limits
   • Monitor for API throttling errors in the connector logs

Best Practices

1. Message Size:

   • Keep message sizes small for optimal performance
   • For large payloads, consider storing the data in S3 and placing only the reference in the SQS message

2. Error Handling:

   • Implement a dead-letter queue (DLQ) for messages that fail processing repeatedly
   • Monitor the DLQ to identify and resolve processing issues

3. Monitoring:

   • Set up CloudWatch alarms on queue metrics like approximate number of messages and age of oldest message
   • Monitor the connector logs for any recurring issues

4. Cost Optimization:

   • Monad always uses long polling (20-second wait), which keeps empty receive calls to a minimum; polling frequency is not configurable

Related Articles

• Amazon SQS Documentation
• Best Practices for Amazon SQS
• SQS Visibility Timeout
• AWS IAM Role Chaining and Temporary Credentials

Sync frequency

This input consumes from the source queue or subscription, ingesting records at whatever rate the source delivers them rather than on a fixed poll. See Input Sync Frequency for details.