Copy page

AWS

SQS S3 GuardDuty

Ingests AWS GuardDuty findings exported to S3 via Amazon Simple Queue Service (SQS).

Details

The Amazon SQS S3 GuardDuty connector is a specialized variant of the SQS S3 connector for ingesting AWS GuardDuty findings delivered to S3 through the GuardDuty findings export feature. GuardDuty exports findings as gzip-compressed JSON Lines files (one finding per line), and this connector emits each line as an individual record.

Use this connector when ingesting native GuardDuty S3 findings exports. Use the standard SQS S3 connector for all other S3-based data sources, and the SQS S3 CloudTrail connector for CloudTrail logs.

How It Works

1. Findings Export: GuardDuty exports new findings to the configured S3 bucket as gzip-compressed JSON Lines objects (encrypted with a KMS key).
2. S3 Event Generation: When GuardDuty delivers a findings file to the S3 bucket, S3 generates an object-created event notification.
3. SQS Message Receipt: The notification is sent to the configured SQS queue as a JSON message.
4. Event Processing: The connector polls the SQS queue, receives S3 event messages, and parses them.
5. File Download: For each S3 event, the connector downloads the referenced object from S3 (decrypting it via the KMS key).
6. Record Extraction: The gzip-compressed JSON Lines file is decompressed and each line is emitted as an individual finding record.
7. Cleanup: Successfully processed SQS messages are deleted from the queue.

Prerequisites

• An AWS account with GuardDuty enabled in the target region
• A configured GuardDuty findings export to an S3 bucket, encrypted with a KMS key
• An existing S3 bucket and SQS queue, or permissions to create them
• An IAM role that the platform can assume, with permissions to access SQS and S3 and to decrypt with the KMS key, or using static AWS credentials
• S3 bucket configured to send event notifications to the SQS queue
• Network connectivity between the platform and AWS

Setup Instructions

Step 1: Create IAM Policy

Create an IAM policy that grants the connector access to S3, SQS, and the KMS key used to encrypt the findings:

1. IAM Role Assumption / Static Credentials
2. Example permission to attach to the role/user:

Code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket",
        "sqs:ChangeMessageVisibility",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes",
        "sqs:GetQueueUrl",
        "sqs:ReceiveMessage"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:{AWS_REGION}:{AWS_ACCOUNT_ID}:key/{KMS_KEY_ID}"
    }
  ]
}

Note: GuardDuty findings exports are always KMS-encrypted, so the kms:Decrypt permission is required — without it s3:GetObject fails with AccessDenied. The KMS key policy must also permit this principal to use the key.

Step 2: Configure GuardDuty Findings Export

Configure GuardDuty to export findings to an S3 bucket:

1. Sign in to the AWS Management Console and open the GuardDuty console.
2. Select the region matching your data.
3. Navigate to Settings > Findings export options.
4. Under S3 bucket, choose an existing bucket or create a new one, and specify a KMS key for encryption (GuardDuty requires KMS encryption for S3 exports).
5. Attach the recommended S3 bucket policy (allows guardduty.amazonaws.com to s3:PutObject) and KMS key policy statement (allows guardduty.amazonaws.com to kms:GenerateDataKey).
6. Optionally set the Frequency of updated findings (defaults to 6 hours; lower it to 15 minutes for faster delivery during testing).

New active findings are exported within about 5 minutes. To generate test data, use Generate sample findings in the GuardDuty console (or aws guardduty create-sample-findings).

Step 3: Create SQS Queue

Set up a standard SQS queue to receive S3 bucket event notifications (FIFO queues are not supported by S3 event notifications):

1. Sign in to the AWS Management Console at https://console.aws.amazon.com/sqs.
2. Under Get Started, click the Create Queue button.
3. For Type, keep Standard.
4. Under Details, enter a name for the queue (e.g., guardduty-findings-queue).
5. Under Configuration:
   • For Visibility Timeout: Enter 600 seconds (10 minutes)
   • For Message Retention Period: Enter 7 days
   • Set Receive message wait time to 20 seconds for long polling
6. Under Access Policy > Choose Method, select the Advanced radio button.
7. Delete the entire policy JSON and copy/paste the following policy:

Code
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "example-statement-ID",
            "Effect": "Allow",
            "Principal": {
                "Service": "s3.amazonaws.com"
            },
            "Action": [
                "SQS:SendMessage"
            ],
            "Resource": "arn:aws:sqs:{AWS_REGION}:{AWS_ACCOUNT_ID}:{QUEUE_NAME}",
            "Condition": {
                "ArnLike": {
                    "aws:SourceArn": "arn:aws:s3:*:*:{BUCKET_NAME}"
                },
                "StringEquals": {
                    "aws:SourceAccount": "{AWS_ACCOUNT_ID}"
                }
            }
        }
    ]
}

8. Replace the placeholders:
   • {AWS_REGION} with the AWS region (e.g., us-east-1)
   • {AWS_ACCOUNT_ID} with the 12-digit AWS account ID
   • {QUEUE_NAME} with the queue name
   • {BUCKET_NAME} with the S3 bucket name
9. Click the Create Queue button.
10. Note the queue URL and ARN for later use.

Step 4: Configure S3 Event Notifications

Configure the S3 bucket to send event notifications to the SQS queue:

1. Sign in to the AWS Management Console at https://console.aws.amazon.com/s3.
2. Under General Purpose Buckets, click on the GuardDuty findings bucket.
3. Click the Properties tab at the top.
4. Locate the Event Notifications section and click Create Event Notification.
5. Configure the event notification:
   • Event Name: Enter a descriptive name (e.g., guardduty-notification)
   • Event Types: Select the checkbox for All object create events
   • Prefix: Scope the notification to GuardDuty findings only. GuardDuty delivers to AWSLogs/<account-id>/GuardDuty/<region>/, so set the prefix to AWSLogs/<account-id>/GuardDuty/ (substituting your 12-digit account ID). This matters when the bucket or SQS queue is shared: a broader prefix such as AWSLogs/ would also notify on other AWS logs delivered there (for example CloudTrail or VPC Flow Logs), and the connector would then try to ingest non-GuardDuty files. Append <region>/ to the prefix to narrow it to a single region.
   • Destination: Select the radio button for SQS queue
   • SQS Queue: Choose the queue from the dropdown or enter the queue ARN
6. Click the Save Changes button.

Step 5: Verify Event Notifications

AWS sends a test notification to the queue after creation. To verify:

1. Navigate to the SQS queue in the AWS Console.
2. Under Details, locate Messages available, which should have a value of 1.
3. This confirms that S3 can successfully send notifications to the queue.

Note: The test message is an s3:TestEvent payload, not a real S3 event. Purge the queue of test messages before starting the connector to avoid decode errors on the control message.

Step 6: Test the Configuration

Before configuring the connector in the platform, verify that the AWS setup is working:

1. Generate sample findings in the GuardDuty console (or wait for a real findings export, typically within 5–15 minutes).
2. Check the SQS queue to confirm a message was received:
   • Navigate to the SQS queue in the AWS Console
   • Under Details, verify Messages available has increased
3. Use the IAM Policy Simulator (optional) to test permissions:
   • Go to https://policysim.aws.amazon.com/home/index.jsp
   • Select the IAM role
   • Test the required S3, SQS, and KMS actions listed in the permissions section

Settings

This connector is purpose-built for native GuardDuty S3 findings exports, so the file format (jsonl), compression (gzip), and record location are fixed internally — they're not exposed as user settings.

Setting	Type	Required	Description
Queue URL	string	Yes	The URL of the SQS queue that receives S3 event notifications (e.g., https://sqs.us-east-1.amazonaws.com/123456789012/my-queue). Use the queue URL, not the ARN.
Role ARN	string	No	The ARN of the IAM role to assume for accessing SQS, S3, and KMS (e.g., arn:aws:iam::123456789012:role/s3-sqs-connector-role). Required when not using static credentials. See Authentication Methods.
Region	string	Yes	The AWS region where the SQS queue and S3 bucket are located (e.g., us-east-1, us-west-2)
Use FIPS Endpoints	boolean	No	When true, routes STS, SQS, and S3 traffic to AWS FIPS 140-2 endpoints. Required for AWS GovCloud (us-gov-east-1, us-gov-west-1). Only supported in US regions (us-east-1, us-east-2, us-west-1, us-west-2, ca-central-1, ca-west-1, us-gov-east-1, us-gov-west-1); enabling it in any other region is rejected during validation. Default: false.
With Metadata	boolean	No	If enabled, adds a _monad_metadata field with bucket information to each emitted message (default: false)
Uses SNS	boolean	No	If enabled, expects SQS messages to be SNS envelopes wrapping S3 event payloads. Enable this only if the bucket sends notifications through SNS before reaching SQS (default: false)

Files are automatically decompressed (gzip) and parsed as JSON Lines; each line is emitted as a separate finding record. No user configuration is needed for either.

Important Notes

• File Format: GuardDuty findings are delivered as gzip-compressed JSON Lines (Content-Encoding: gzip, Content-Type: application/json) — one finding per line. Each line becomes one emitted record.
• KMS Encryption: GuardDuty S3 exports are always KMS-encrypted. The connector's principal must have kms:Decrypt on the export key (in both the IAM policy and the KMS key policy), or object downloads fail with AccessDenied.
• Event Filtering: Only ObjectCreated: S3 events are processed; other events are ignored.
• Error Handling: If file processing fails, the SQS message remains in the queue and will be retried.
• Message Deletion: SQS messages are deleted after successful file processing.
• Metadata Preservation: If With Metadata is enabled, the _monad_metadata field (containing bucket information) is added at the top level of each emitted record.

Related Articles

• Amazon GuardDuty Documentation
• Exporting GuardDuty findings to Amazon S3 buckets
• GuardDuty finding format
• Amazon S3 Event Notifications
• S3 Event Message Structure
• AWS IAM Role Chaining and Temporary Credentials

Sync frequency

This input consumes from the source queue or subscription, ingesting records at whatever rate the source delivers them rather than on a fixed poll. See Input Sync Frequency for details.