# SQS S3 GuardDuty

Ingests AWS GuardDuty findings exported to S3 via Amazon Simple Queue Service (SQS).

## Details

The Amazon SQS S3 GuardDuty connector is a specialized variant of the SQS S3 connector for ingesting AWS GuardDuty findings delivered to S3 through the GuardDuty **findings export** feature. GuardDuty exports findings as gzip-compressed JSON Lines files (one finding per line), and this connector emits each line as an individual record.

Use this connector when ingesting native GuardDuty S3 findings exports. Use the standard [SQS S3](./aws-sqs-s3.mdx) connector for all other S3-based data sources, and the [SQS S3 CloudTrail](./aws-sqs-s3-cloudtrail.mdx) connector for CloudTrail logs.

## How It Works

1. **Findings Export**: GuardDuty exports new findings to the configured S3 bucket as gzip-compressed JSON Lines objects (encrypted with a KMS key).
2. **S3 Event Generation**: When GuardDuty delivers a findings file to the S3 bucket, S3 generates an object-created event notification.
3. **SQS Message Receipt**: The notification is sent to the configured SQS queue as a JSON message.
4. **Event Processing**: The connector polls the SQS queue, receives S3 event messages, and parses them.
5. **File Download**: For each S3 event, the connector downloads the referenced object from S3 (decrypting it via the KMS key).
6. **Record Extraction**: The gzip-compressed JSON Lines file is decompressed and each line is emitted as an individual finding record.
7. **Cleanup**: Successfully processed SQS messages are deleted from the queue.

## Prerequisites

- An AWS account with GuardDuty enabled in the target region
- A configured GuardDuty **findings export** to an S3 bucket, encrypted with a KMS key
- An existing S3 bucket and SQS queue, or permissions to create them
- An IAM role that the platform can assume, with permissions to access SQS and S3 **and to decrypt with the KMS key**, or using static AWS credentials
- S3 bucket configured to send event notifications to the SQS queue
- Network connectivity between the platform and AWS

## Setup Instructions

### Step 1: Create IAM Policy

Create an IAM policy that grants the connector access to S3, SQS, and the KMS key used to encrypt the findings:

1. [IAM Role Assumption / Static Credentials](./index.mdx#authentication-methods)
2. Example permission to attach to the role/user:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket",
        "sqs:ChangeMessageVisibility",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes",
        "sqs:GetQueueUrl",
        "sqs:ReceiveMessage"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:{AWS_REGION}:{AWS_ACCOUNT_ID}:key/{KMS_KEY_ID}"
    }
  ]
}
```

**Note:** GuardDuty findings exports are always KMS-encrypted, so the `kms:Decrypt` permission is required — without it `s3:GetObject` fails with `AccessDenied`. The KMS **key policy** must also permit this principal to use the key.

### Step 2: Configure GuardDuty Findings Export

Configure GuardDuty to export findings to an S3 bucket:

1. Sign in to the AWS Management Console and open the GuardDuty console.
2. Select the region matching your data.
3. Navigate to **Settings** > **Findings export options**.
4. Under **S3 bucket**, choose an existing bucket or create a new one, and specify a **KMS key** for encryption (GuardDuty requires KMS encryption for S3 exports).
5. Attach the recommended **S3 bucket policy** (allows `guardduty.amazonaws.com` to `s3:PutObject`) and **KMS key policy** statement (allows `guardduty.amazonaws.com` to `kms:GenerateDataKey`).
6. Optionally set the **Frequency of updated findings** (defaults to 6 hours; lower it to 15 minutes for faster delivery during testing).

New active findings are exported within about 5 minutes. To generate test data, use **Generate sample findings** in the GuardDuty console (or `aws guardduty create-sample-findings`).

### Step 3: Create SQS Queue

Set up a **standard** SQS queue to receive S3 bucket event notifications (FIFO queues are not supported by S3 event notifications):

1. Sign in to the AWS Management Console at https://console.aws.amazon.com/sqs.
2. Under Get Started, click the **Create Queue** button.
3. For **Type**, keep **Standard**.
4. Under Details, enter a name for the queue (e.g., `guardduty-findings-queue`).
5. Under Configuration:
   - For **Visibility Timeout**: Enter `600` seconds (10 minutes)
   - For **Message Retention Period**: Enter `7` days
   - Set **Receive message wait time** to `20` seconds for long polling
6. Under Access Policy > Choose Method, select the **Advanced** radio button.
7. Delete the entire policy JSON and copy/paste the following policy:

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "example-statement-ID",
            "Effect": "Allow",
            "Principal": {
                "Service": "s3.amazonaws.com"
            },
            "Action": [
                "SQS:SendMessage"
            ],
            "Resource": "arn:aws:sqs:{AWS_REGION}:{AWS_ACCOUNT_ID}:{QUEUE_NAME}",
            "Condition": {
                "ArnLike": {
                    "aws:SourceArn": "arn:aws:s3:*:*:{BUCKET_NAME}"
                },
                "StringEquals": {
                    "aws:SourceAccount": "{AWS_ACCOUNT_ID}"
                }
            }
        }
    ]
}
```

8. Replace the placeholders:
   - `{AWS_REGION}` with the AWS region (e.g., `us-east-1`)
   - `{AWS_ACCOUNT_ID}` with the 12-digit AWS account ID
   - `{QUEUE_NAME}` with the queue name
   - `{BUCKET_NAME}` with the S3 bucket name
9. Click the **Create Queue** button.
10. Note the queue URL and ARN for later use.

### Step 4: Configure S3 Event Notifications

Configure the S3 bucket to send event notifications to the SQS queue:

1. Sign in to the AWS Management Console at https://console.aws.amazon.com/s3.
2. Under General Purpose Buckets, click on the GuardDuty findings bucket.
3. Click the **Properties** tab at the top.
4. Locate the **Event Notifications** section and click **Create Event Notification**.
5. Configure the event notification:
   - **Event Name**: Enter a descriptive name (e.g., `guardduty-notification`)
   - **Event Types**: Select the checkbox for **All object create events**
   - **Prefix**: Scope the notification to GuardDuty findings only. GuardDuty delivers to `AWSLogs/<account-id>/GuardDuty/<region>/`, so set the prefix to `AWSLogs/<account-id>/GuardDuty/` (substituting your 12-digit account ID). This matters when the bucket or SQS queue is shared: a broader prefix such as `AWSLogs/` would also notify on other AWS logs delivered there (for example CloudTrail or VPC Flow Logs), and the connector would then try to ingest non-GuardDuty files. Append `<region>/` to the prefix to narrow it to a single region.
   - **Destination**: Select the radio button for **SQS queue**
   - **SQS Queue**: Choose the queue from the dropdown or enter the queue ARN
6. Click the **Save Changes** button.

### Step 5: Verify Event Notifications

AWS sends a test notification to the queue after creation. To verify:

1. Navigate to the SQS queue in the AWS Console.
2. Under Details, locate **Messages available**, which should have a value of 1.
3. This confirms that S3 can successfully send notifications to the queue.

**Note**: The test message is an `s3:TestEvent` payload, not a real S3 event. Purge the queue of test messages before starting the connector to avoid decode errors on the control message.

### Step 6: Test the Configuration

Before configuring the connector in the platform, verify that the AWS setup is working:

1. **Generate sample findings** in the GuardDuty console (or wait for a real findings export, typically within 5–15 minutes).
2. **Check the SQS queue** to confirm a message was received:
   - Navigate to the SQS queue in the AWS Console
   - Under Details, verify **Messages available** has increased
3. **Use the IAM Policy Simulator** (optional) to test permissions:
   - Go to https://policysim.aws.amazon.com/home/index.jsp
   - Select the IAM role
   - Test the required S3, SQS, and KMS actions listed in the permissions section

### Settings

This connector is purpose-built for native GuardDuty S3 findings exports, so the file format (`jsonl`), compression (`gzip`), and record location are fixed internally — they're not exposed as user settings.

| Setting | Type | Required | Description |
| ------- | ---- | -------- | ----------- |
| Queue URL | string | Yes | The URL of the SQS queue that receives S3 event notifications (e.g., `https://sqs.us-east-1.amazonaws.com/123456789012/my-queue`). Use the queue URL, not the ARN. |
| Role ARN | string | No | The ARN of the IAM role to assume for accessing SQS, S3, and KMS (e.g., `arn:aws:iam::123456789012:role/s3-sqs-connector-role`). Required when not using static credentials. See [Authentication Methods](./index.mdx#authentication-methods). |
| Region | string | Yes | The AWS region where the SQS queue and S3 bucket are located (e.g., `us-east-1`, `us-west-2`) |
| Use FIPS Endpoints | boolean | No | When `true`, routes STS, SQS, and S3 traffic to AWS FIPS 140-2 endpoints. Required for AWS GovCloud (`us-gov-east-1`, `us-gov-west-1`). Only supported in US regions (`us-east-1`, `us-east-2`, `us-west-1`, `us-west-2`, `ca-central-1`, `ca-west-1`, `us-gov-east-1`, `us-gov-west-1`); enabling it in any other region is rejected during validation. Default: `false`. |
| With Metadata | boolean | No | If enabled, adds a `_monad_metadata` field with bucket information to each emitted message (default: `false`) |
| Uses SNS | boolean | No | If enabled, expects SQS messages to be SNS envelopes wrapping S3 event payloads. Enable this only if the bucket sends notifications through SNS before reaching SQS (default: `false`) |

Files are automatically decompressed (gzip) and parsed as JSON Lines; each line is emitted as a separate finding record. No user configuration is needed for either.

## Important Notes

- **File Format**: GuardDuty findings are delivered as gzip-compressed JSON Lines (`Content-Encoding: gzip`, `Content-Type: application/json`) — one finding per line. Each line becomes one emitted record.
- **KMS Encryption**: GuardDuty S3 exports are always KMS-encrypted. The connector's principal must have `kms:Decrypt` on the export key (in both the IAM policy and the KMS key policy), or object downloads fail with `AccessDenied`.
- **Event Filtering**: Only `ObjectCreated:` S3 events are processed; other events are ignored.
- **Error Handling**: If file processing fails, the SQS message remains in the queue and will be retried.
- **Message Deletion**: SQS messages are deleted after successful file processing.
- **Metadata Preservation**: If `With Metadata` is enabled, the `_monad_metadata` field (containing bucket information) is added at the top level of each emitted record.

## Related Articles

- [Amazon GuardDuty Documentation](https://docs.aws.amazon.com/guardduty/)
- [Exporting GuardDuty findings to Amazon S3 buckets](https://docs.aws.amazon.com/guardduty/latest/ug/guardduty_exportfindings.html)
- [GuardDuty finding format](https://docs.aws.amazon.com/guardduty/latest/ug/guardduty_findings.html)
- [Amazon S3 Event Notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html)
- [S3 Event Message Structure](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-content-structure.html)
- [AWS IAM Role Chaining and Temporary Credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_terms-and-concepts.html)

## Sync frequency

This input consumes from the source queue or subscription, ingesting records at whatever rate the source delivers them rather than on a fixed poll. See [Input Sync Frequency](../../guides/sync-frequency) for details.