Skip to main content

SQS

Enables seamless ingestion of messages from Amazon Simple Queue Service (SQS) for data processing and integration.

Details​

The Amazon SQS connector allows you to collect messages from any Amazon SQS queue for further processing in your data pipeline. This connector is versatile and can process messages from any source that publishes to SQS, making it ideal for integrating multiple AWS services or custom applications with your platform.

The connector polls an Amazon SQS queue at regular intervals, retrieves messages, processes them, and then deletes them from the queue after successful processing. This ensures reliable and efficient data ingestion while maintaining message delivery guarantees.

Prerequisites​

  • An AWS account with an existing SQS queue or permissions to create one
  • An IAM role that your platform can assume, with appropriate permissions to access the SQS queue
  • Network connectivity between your platform and AWS

Setup Instructions​

Step 1: Create or Configure an SQS Queue (if needed)​

  1. Sign in to the AWS Management Console.
  2. Navigate to the Amazon SQS service.
  3. If creating a new queue:
    • Click Create queue.
    • Select Standard queue type (or FIFO if you need exact ordering and exactly-once processing).
    • Enter a name for your queue.
    • Configure basic settings:
      • Configure message retention, maximum message size, default visibility timeout and other settings as needed.
    • Click Create queue.
  4. Note the queue URL for later use in the connector configuration.

Step 2: Create IAM Role for Access​

  • Follow the IAM Role Setup Guide in the main AWS documentation.
  • Make sure to attach a policy with the following permissions to your user/role: 
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "sqs:ReceiveMessage",
            "sqs:DeleteMessage",
            "sqs:GetQueueAttributes"
          ],
          "Resource": "[SQS-QUEUE-ARN]"
        }
      ]
    }

Configuration​

The following configuration defines the input parameters for the SQS connector:

Settings​

SettingTypeRequiredDescription
Queue URLstringYesThe URL of the SQS queue to poll for messages.
Role ARNstringYesThe ARN of the IAM role to assume for accessing the SQS queue.
RegionstringYesThe AWS region where the SQS queue is located.

Important Notes​

  • Message Deletion: Messages are automatically deleted from the queue after successful processing. If processing fails, messages become visible again after the visibility timeout period.

  • FIFO Queues: When using FIFO queues, messages with the same message group ID are processed in order, but this may affect throughput if one message in a group is taking longer to process.

IAM Permissions Required​

The IAM role specified in Role ARN must have the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes"
      ],
      "Resource": "arn:aws:sqs:[REGION]:[ACCOUNT-ID]:[QUEUE-NAME]"
    }
  ]
}

Troubleshooting​

No Messages Being Received​

  1. Queue Verification:

    • Verify the SQS queue exists and contains messages
    • Use the AWS Console or CLI to view the queue's approximate number of messages
    • Send a test message manually to verify queue functionality

  2. Permission Issues:

    • Confirm the IAM role has the required permissions (sqs:ReceiveMessage, sqs:DeleteMessage, sqs:GetQueueAttributes)
    • Check that the trust relationship is correctly configured to allow the platform to assume the role
    • Verify the role ARN is correctly specified in the connector configuration

  3. Region Configuration:

    • Ensure the Region specified in the connector configuration matches the region where the SQS queue is deployed

Connectivity Issues​

  1. Network Connectivity:

    • Ensure the platform has network connectivity to AWS SQS in the specified region
    • Check for any firewall rules or security groups that might be blocking the connection

  2. Service Quotas:

    • Review AWS service quotas for SQS to ensure you're not hitting any limits
    • Monitor for API throttling errors in the connector logs

Best Practices​

  1. Message Size:

    • Keep message sizes small for optimal performance
    • For large payloads, consider storing the data in S3 and placing only the reference in the SQS message

  2. Error Handling:

    • Implement a dead-letter queue (DLQ) for messages that fail processing repeatedly
    • Monitor the DLQ to identify and resolve processing issues

  3. Monitoring:

    • Set up CloudWatch alarms on queue metrics like approximate number of messages and age of oldest message
    • Monitor the connector logs for any recurring issues

  4. Cost Optimization:

    • Adjust polling frequency based on your throughput needs
    • Use long polling to reduce empty API calls