Copy page

AWS

SQS S3 GuardDuty

Ingests AWS GuardDuty findings exported to S3 via Amazon Simple Queue Service (SQS).

Sync Type: Streaming

Details

The Amazon SQS S3 GuardDuty connector is a specialized variant of the SQS S3 connector for ingesting AWS GuardDuty findings delivered to S3 through the GuardDuty findings export feature. GuardDuty exports findings as gzip-compressed JSON Lines files (one finding per line), and this connector emits each line as an individual record.

Use this connector when ingesting native GuardDuty S3 findings exports. Use the standard SQS S3 connector for all other S3-based data sources, and the SQS S3 CloudTrail connector for CloudTrail logs.

How It Works

1. Findings Export: GuardDuty exports new findings to the configured S3 bucket as gzip-compressed JSON Lines objects (encrypted with a KMS key).
2. S3 Event Generation: When GuardDuty delivers a findings file to the S3 bucket, S3 generates an object-created event notification.
3. SQS Message Receipt: The notification is sent to the configured SQS queue as a JSON message.
4. Event Processing: The connector polls the SQS queue, receives S3 event messages, and parses them.
5. File Download: For each S3 event, the connector downloads the referenced object from S3 (decrypting it via the KMS key).
6. Record Extraction: The gzip-compressed JSON Lines file is decompressed and each line is emitted as an individual finding record.
7. Cleanup: Successfully processed SQS messages are deleted from the queue.

Prerequisites

• An AWS account with GuardDuty enabled in the target region
• A configured GuardDuty findings export to an S3 bucket, encrypted with a KMS key
• An existing S3 bucket and SQS queue, or permissions to create them
• An IAM role that the platform can assume, with permissions to access SQS and S3 and to decrypt with the KMS key, or using static AWS credentials
• S3 bucket configured to send event notifications to the SQS queue
• If the queue uses SSE-KMS, the role also needs kms:Decrypt on the queue's key, and the key policy must allow S3 (s3.amazonaws.com) to use kms:GenerateDataKey and kms:Decrypt so S3 can publish to the queue.
• Network connectivity between the platform and AWS

Setup Instructions

Step 1: Create the IAM role Monad will assume

Monad authenticates to your AWS account by assuming an IAM role that you create. The role needs two policies: a trust policy that lets Monad assume it, and a permissions policy that grants read access to the queue and the bucket and permission to decrypt the findings with their KMS key. If your Monad deployment runs on a dedicated cluster outside AWS, use the OIDC trust policy in Authentication Methods instead of the one below. To use static access keys instead of a role, see the same section.

1. Find your Monad organization ID. It is the UUID in the Monad UI address bar while you are viewing your organization (/organizations/<organization-id>), and it is also shown on the detail panel of any pipeline node.
2. Sign in to the AWS Management Console at https://console.aws.amazon.com/iam and click Roles, then Create role.
3. Under Trusted entity type, select Custom trust policy, delete the policy JSON, and paste the following. Replace {your-monad-org-id} with your Monad organization ID.

Code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowMonadAssumeRole",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::339712996529:role/monad-app"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "{your-monad-org-id}"
        }
      }
    }
  ]
}

Monad always presents your organization ID as the external ID when it assumes the role, so this condition means the role can only be assumed on behalf of your organization.

4. Attach a permissions policy scoped to your queue and bucket, plus decrypt access to the findings KMS key. Replace the placeholder values in braces with your values. If you have not created the queue or bucket yet, you can create the role now with a placeholder policy and come back to this step after Step 3.

Code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadAndDeleteQueueMessages",
      "Effect": "Allow",
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes"
      ],
      "Resource": "arn:aws:sqs:{AWS_REGION}:{AWS_ACCOUNT_ID}:{QUEUE_NAME}"
    },
    {
      "Sid": "ReadBucketObjects",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": "arn:aws:s3:::{BUCKET_NAME}/*"
    },
    {
      "Sid": "DecryptFindings",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:{AWS_REGION}:{AWS_ACCOUNT_ID}:key/{KMS_KEY_ID}"
    }
  ]
}

Note: GuardDuty findings exports are always KMS-encrypted, so the kms:Decrypt permission is required — without it s3:GetObject fails with AccessDenied. The KMS key policy must also permit this role to use the key.

5. Name the role (e.g., monad-sqs-s3-input) and click Create role.
6. Note the role ARN. You will enter it as Role ARN when configuring the connector.

Step 2: Configure GuardDuty Findings Export

Configure GuardDuty to export findings to an S3 bucket:

1. Sign in to the AWS Management Console and open the GuardDuty console.
2. Select the region matching your data.
3. Navigate to Settings > Findings export options.
4. Under S3 bucket, choose an existing bucket or create a new one, and specify a KMS key for encryption (GuardDuty requires KMS encryption for S3 exports).
5. Attach the recommended S3 bucket policy (allows guardduty.amazonaws.com to s3:PutObject) and KMS key policy statement (allows guardduty.amazonaws.com to kms:GenerateDataKey).
6. Optionally set the Frequency of updated findings (defaults to 6 hours; lower it to 15 minutes for faster delivery during testing).

New active findings are exported within about 5 minutes. To generate test data, use Generate sample findings in the GuardDuty console (or aws guardduty create-sample-findings).

Step 3: Create SQS Queue

Set up a standard SQS queue to receive S3 bucket event notifications (FIFO queues are not supported by S3 event notifications):

1. Sign in to the AWS Management Console at https://console.aws.amazon.com/sqs.
2. Under Get Started, click the Create Queue button.
3. For Type, keep Standard.
4. Under Details, enter a name for the queue (e.g., guardduty-findings-queue).
5. Under Configuration:
   • For Visibility Timeout: Leave the default; Monad sets its own visibility timeout on every receive, so the queue setting is overridden
   • For Message Retention Period: Enter 7 days
   • Set Receive message wait time to 20 seconds for long polling
6. Under Access Policy > Choose Method, select the Advanced radio button.
7. Delete the entire policy JSON and copy/paste the following policy:

Code
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "example-statement-ID",
            "Effect": "Allow",
            "Principal": {
                "Service": "s3.amazonaws.com"
            },
            "Action": [
                "SQS:SendMessage"
            ],
            "Resource": "arn:aws:sqs:{AWS_REGION}:{AWS_ACCOUNT_ID}:{QUEUE_NAME}",
            "Condition": {
                "ArnLike": {
                    "aws:SourceArn": "arn:aws:s3:*:*:{BUCKET_NAME}"
                },
                "StringEquals": {
                    "aws:SourceAccount": "{AWS_ACCOUNT_ID}"
                }
            }
        }
    ]
}

8. Replace the placeholders:
   • {AWS_REGION} with the AWS region (e.g., us-east-1)
   • {AWS_ACCOUNT_ID} with the 12-digit AWS account ID
   • {QUEUE_NAME} with the queue name
   • {BUCKET_NAME} with the S3 bucket name
9. Click the Create Queue button.
10. Note the queue URL and ARN for later use.

Step 4: Configure S3 Event Notifications

Configure the S3 bucket to send event notifications to the SQS queue:

1. Sign in to the AWS Management Console at https://console.aws.amazon.com/s3.
2. Under General Purpose Buckets, click on the GuardDuty findings bucket.
3. Click the Properties tab at the top.
4. Locate the Event Notifications section and click Create Event Notification.
5. Configure the event notification:
   • Event Name: Enter a descriptive name (e.g., guardduty-notification)
   • Event Types: Select the checkbox for All object create events
   • Prefix: Scope the notification to GuardDuty findings only. GuardDuty delivers to AWSLogs/<account-id>/GuardDuty/<region>/, so set the prefix to AWSLogs/<account-id>/GuardDuty/ (substituting your 12-digit account ID). This matters when the bucket or SQS queue is shared: a broader prefix such as AWSLogs/ would also notify on other AWS logs delivered there (for example CloudTrail or VPC Flow Logs), and the connector would then try to ingest non-GuardDuty files. Append <region>/ to the prefix to narrow it to a single region.
   • Destination: Select the radio button for SQS queue
   • SQS Queue: Choose the queue from the dropdown or enter the queue ARN
6. Click the Save Changes button.

Step 5: Verify Event Notifications

AWS sends a test notification to the queue after creation. To verify:

1. Navigate to the SQS queue in the AWS Console.
2. Under Details, locate Messages available, which should have a value of 1.
3. This confirms that S3 can successfully send notifications to the queue.

Note: The test message is an s3:TestEvent payload, not a real S3 event. The connector detects and skips it automatically, so you do not need to purge it.

Step 6: Test the Configuration

Before configuring the connector in the platform, verify that the AWS setup is working:

1. Generate sample findings in the GuardDuty console (or wait for a real findings export, typically within 5–15 minutes).
2. Check the SQS queue to confirm a message was received:
   • Navigate to the SQS queue in the AWS Console
   • Under Details, verify Messages available has increased
3. Use the IAM Policy Simulator (optional) to test permissions:
   • Go to https://policysim.aws.amazon.com/home/index.jsp
   • Select the IAM role
   • Test the required S3, SQS, and KMS actions listed in the permissions section

Configuration

Settings

This connector is purpose-built for native GuardDuty S3 findings exports, so the file format (jsonl), compression (gzip), and record location are fixed internally — they're not exposed as user settings.

Setting	Type	Required	Default	Description
queue_url	string	Yes	-	The URL of the SQS queue to poll for messages.
role_arn	string	No	-	The ARN of the IAM role to assume for accessing AWS resources. Required when not using static credentials. See Authentication Methods.
region	string	Yes	us-east-1	The AWS region to use. Allowed values: us-east-1, us-east-2, us-west-1, us-west-2, af-south-1, ap-east-1, ap-south-1, ap-northeast-1, ap-northeast-2, ap-northeast-3, ap-southeast-1, ap-southeast-2, ca-central-1, ca-west-1, cn-north-1, cn-northwest-1, eu-central-1, eu-west-1, eu-west-2, eu-west-3, eu-north-1, eu-south-1, me-south-1, sa-east-1, us-gov-east-1, us-gov-west-1.
use_fips	bool	No	false	Route STS, SQS, and S3 traffic to FIPS 140-2 endpoints. Required for AWS GovCloud. Only available in US regions (us-east-1/2, us-west-1/2, ca-central-1, ca-west-1, us-gov-east-1, us-gov-west-1).
uses_sns	bool	No	false	Uses AWS SNS in the middle of S3 and SQS for fan-out usecases.
with_metadata	bool	No	false	Whether to include S3 object metadata in the output.

Secrets

Secret	Type	Required	Description
access_key	secret	No	The Access Key from AWS to authenticate with AWS
secret_key	secret	No	The Secret Key from AWS to authenticate with AWS

Files are automatically decompressed (gzip) and parsed as JSON Lines; each line is emitted as a separate finding record. No user configuration is needed for either.

Important Notes

• File Format: GuardDuty findings are delivered as gzip-compressed JSON Lines (Content-Encoding: gzip, Content-Type: application/json) — one finding per line. Each line becomes one emitted record.
• KMS Encryption: GuardDuty S3 exports are always KMS-encrypted. The connector's principal must have kms:Decrypt on the export key (in both the IAM policy and the KMS key policy), or object downloads fail with AccessDenied.
• Event Filtering: Only ObjectCreated: S3 events are processed; other events are ignored. s3:TestEvent messages are skipped and deleted.
• Non-JSON messages: Messages whose body is not valid JSON are deleted without processing.
• JSON messages that are not S3 event notifications: A JSON message without a Records array fails to decode and is not deleted; it is redelivered (about every 3 minutes) until it expires or moves to a dead-letter queue.
• Delivery is at-least-once: A message is deleted only after all records from its objects have been emitted. If processing fails, the message stays in the queue and is retried. An object that takes longer than about 2 minutes to process is abandoned and retried, and records already emitted from it can repeat.
• Visibility timeout: Monad sets its own visibility timeout on every receive, overriding the queue's configured value.
• Shared queues: If several pipelines or inputs read the same queue, the messages are split between them; each message goes to only one consumer. To send the same findings to more than one pipeline, give each pipeline its own queue, either with separate S3 event notifications per prefix or with SNS fan-out to one queue per pipeline (enable Uses SNS).
• Metadata Preservation: If With Metadata is enabled, the _monad_metadata field (containing bucket information) is added at the top level of each emitted record.

Related Articles

• Amazon GuardDuty Documentation
• Exporting GuardDuty findings to Amazon S3 buckets
• GuardDuty finding format
• Amazon S3 Event Notifications
• S3 Event Message Structure
• AWS IAM Role Chaining and Temporary Credentials

Sync frequency

This input consumes from the source queue or subscription, ingesting records at whatever rate the source delivers them rather than on a fixed poll. See Input Sync Frequency for details.