Copy page

AWS

CloudTrail Events

Collects account activity and API usage across your AWS infrastructure for full visibility into control plane events.

Details

On initialization for each sync, Monad generates every account_id + region combination found in your logs bucket. This ensures that any new account_id + region combinations added are picked up automatically on the next run.

By default, Monad begins ingesting CloudTrail logs written from the time the input first runs onward — no historical logs are fetched. To ingest history, set a Backfill Start Time: the first run then performs a one-time sync of all logs from that date up to now, and subsequent runs are incremental. A newly discovered account_id + region combination is likewise picked up from the time it is first seen (or from the Backfill Start Time, if set). For subsequent runs, each AWS account_id + region combination maintains its own state to make sure only incremental data is processed. On restart in case of any form of failure, each partition resumes from its last checkpointed timestamp. A checkpoint occurs at every page within a prefix. So while processing a prefix, if a failure occurs, the processor will restart from the last completed page's checkpoint. You will not lose out on any records, however you may re-process some data in the S3 objects on the page where the failure occured in case of any catastrophic failures.

Each partition (account_id + region) is processed independently and asynchronously, ensuring quick data ingestion and isolating errors to a single partition without affecting others.

Sync Frequency

By default, Monad polls the bucket on a continuous cycle: a new sync begins roughly every 10 seconds, with each cycle starting 10 seconds after the previous one completes. Each sync is incremental, so only objects newer than the last checkpoint are fetched. If a cron schedule is configured on the pipeline, that schedule overrides the default and Monad syncs on the cron cadence instead.

Performance Considerations and Limitations

Rate Limiting Risk: Monad performs a full date prefix scan on every sync run for each account_id + region combination. During each scan, previously processed records are identified and dropped based on deduplication logic. This approach has important performance implications:

• For buckets with large volumes of data on a given date, the repeated full-prefix scanning can consume significant API quota
• Since regions are processed concurrently, accounts with simultaneous data bursts across multiple regions will amplify API usage, potentially triggering S3 rate limits
• The overhead increases proportionally with data volume, as the same objects are rescanned and filtered on every run throughout the day
• Processing speed degrades as data accumulates within a date prefix, since each sync must scan and filter through increasingly larger datasets before identifying new records
• To protect against tripping S3's own rate limits, Monad throttles its bucket-listing (pagination) requests to a maximum of 1,000 requests per second per partition. Because up to 4 partitions are processed concurrently, aggregate listing volume across a sync can be higher

Recommendation: For production deployments with high-volume CloudTrail data or multiple active partitions, we strongly recommend using an S3-to-SQS event notification architecture instead of prefix-based polling. This eliminates redundant scanning and provides near-real-time ingestion without rate limit concerns.

Note

• Prefixes must be hive compliant with simple date formatting. Any other structure can cause unexpected behavior in the input.

• Each log's last updated time should align with its logical prefix date. For example, objects in the 2025/08/10 prefix should have a last updated time of 2025-08-10 (in ISO8601 format). Misalignment can cause unexpected behavior in the input.

• To avoid such tight boundaries, we recommend publishing S3 data to an SQS queue to avoid such failures.

Prerequisites

• IAM Role Assumption / Static Credentials
• Example permission to attach to the role/user:

Code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::{bucket_name}"
    },
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::{bucket_name}/*"
    }
  ]
}

setup documentation

Related Articles

• https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-log-file-examples.html
• https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference-record-contents.html

Configuration

The following configuration defines the input parameters. Each field's specifications, such as type, requirements, and descriptions, are detailed below.

Settings

Setting	Type	Required	Description
Region	string	No	The region of the S3 bucket. If left blank, the region will be auto-detected.
Bucket	string	Yes	The name of the S3 bucket.
Prefix	string	No	Prefix of the S3 object keys to read. If your CloudTrail logs are at the standard location (starting with "AWSLogs/") at the root of the bucket, leave this field blank. Only specify a prefix if your CloudTrail logs are stored in a non-default location within your bucket.
Role ARN	string	Conditional	The ARN of the role to assume to access the bucket. Required unless static credentials (Access Key + Secret Key) are provided.
Backfill Start Time	string	No	The date to start fetching data from. If not specified, Monad performs a full sync of all available CloudTrail logs on the first run; all subsequent runs are incremental.

Secrets (Static Credentials Only)

Setting	Type	Required	Description
Access Key	string	Conditional	AWS Access Key ID
Secret Key	string	Conditional	AWS Secret Access Key

⚠️ Authentication: Choose either Role ARN (recommended) or static credentials. See AWS Authentication Guide for setup instructions.

API

To send a POST request to create this Cloudtrail Connector:

Code
curl -X 'POST' \
  '{base_url}/api/v2/{org_id}/inputs' \
  -H 'accept: application/json' \
  -H 'x-api-key: {api_key}' \
  -H 'Content-Type: application/json' \
  -d '{
  "config": {
    "secrets": {},
    "settings": {
      "region": "region",
      "bucket": "bucket",
      "prefix": "prefix",
      "role_arn": "role_arn"
    }
  },
  "description": "input_description",
  "name": "input_name",
  "promise_id": "",
  "type": "cloudtrail"
}'

OCSF Conversion

The following JQ transformation converts AWS CloudTrail data to OCSF Version 1.1 compliant format. This conversion maps essential CloudTrail activity data to standardized OCSF fields, facilitating better integration and analysis across security tools.

The transformation is provided as a starting point and can be modified to accommodate specific needs:

Code
{
  category_uid: 6, # Application Activity
  class_uid: 6003, # API Activity
  type_uid: 600300, # API Activity: Unknown
  activity_id: 0, # Unknown
  severity_id: 0, # Unknown

  time: (.eventTime | fromdateiso8601),
  time_dt: .eventTime,

  actor: {
    user: {
      name: .userIdentity.userName,
      type: .userIdentity.type,
      account: {
        name: .userIdentity.accountId,
        type: "AWS",
        uid: .userIdentity.principalId
      }
    }
  },

  api: {
    operation: .eventName,
    service: {
      name: .eventSource,
      uid: .eventSource
    },
    request: {
      uid: .requestID,
      params: .requestParameters
    },
    response: {
      result: .responseElements,
      error: (.errorCode // ""),
      error_message: (.errorMessage // "")
    }
  },

  cloud: {
    provider: "AWS",
    region: .awsRegion,
    account: {
      name: .recipientAccountId,
      uid: .recipientAccountId
    }
  },

  src_endpoint: {
    ip: .sourceIPAddress,
    type: "IPv4",
    type_id: 0
  },

  status_id: (if .errorCode != null then 1 else 0 end),
  status: (if .errorCode != null then "Failure" else "Success" end),

  resources: (
    if .resources then
      [.resources[] | {
        type: (.resourceType // "Resource"),
        uid: .resourceName,
        name: .resourceName,
        details: {
          type: .resourceType,
          arn: .ARN
        }
      }]
    else []
    end
  ),

  metadata: {
    version: "1.1.0",
    product: {
      name: "CloudTrail",
      vendor_name: "AWS",
      feature: {
        name: .eventCategory
      }
    }
  }
}

OCSF Mapping Details

The transformation follows OCSF Version 1.1 specifications with the following key mappings:

Core Fields

• category_uid: Fixed value of 6 (API Activity)
• class_uid: Fixed value of 6003 (API Activity)
• type_uid: Set to 600300 API Activity: Unknown
• activity_id: Set to 0 as CloudTrail events don't map directly to OCSF activity IDs
• severity_id: Set to 0 as CloudTrail doesn't provide native severity levels

Time and Actor Information

• time: Converts CloudTrail's ISO8601 timestamp to Unix epoch using fromdateiso8601
• time_dt: Preserves original ISO8601 timestamp from CloudTrail
• actor.user: Maps IAM identity information including:
  • name: Maps from userIdentity.userName
  • type: Maps from userIdentity.type
  • account: Contains AWS account details from userIdentity

API and Cloud Context

• api: Captures API operation details:
  • operation: Maps from eventName
  • service: Contains service name and ID from eventSource
  • request/response: Preserves full request and response details
• cloud: Contains AWS-specific context:
  • provider: Fixed as "AWS"
  • region: Maps from awsRegion
  • account: Contains recipient account details

Resource Handling

• resources: Uses a generic handler to capture all resource types:
  • Processes the CloudTrail resources array
  • Preserves resource type, name, and ARN
  • Handles null/missing resources gracefully
  • Maintains consistent structure across all AWS resource types

Metadata

• metadata: Contains OCSF and product information:
  • version: Fixed as "1.1.0" per OCSF specification
  • product.name: Fixed as "CloudTrail"
  • product.vendor_name: Fixed as "AWS"
  • product.feature.name: Mapped from CloudTrail's eventCategory field (for example, "Management", "Data", or "Insight")

Important Notes on OCSF Field Mapping

Several OCSF fields require special consideration when mapping from CloudTrail:

1. Event Classification (category_uid, class_uid)

   • All CloudTrail events are classified as API Activity
   • This is consistent across Management, Lambda Data, and S3 Data events
   • No need for complex classification logic

2. Feature Name Determination

   • Mapped directly from CloudTrail's eventCategory field, which is typically "Management", "Data", or "Insight"

3. Status Handling

   • status_id: 0 for success, 1 for failure
   • status: Maps directly from presence/absence of errorCode
   • Includes error messages when available

4. Resource Handling

   • Uses generic resource handling to support all AWS resource types
   • Maintains consistent structure regardless of resource type
   • Gracefully handles missing or null resource information

5. Optional Enhancements Consider these potential improvements based on your needs:

   • Add severity mapping based on event characteristics
   • Implement activity type classification based on event patterns
   • Add custom resource type normalization
   • Include additional context fields from CloudTrail

This transformation provides a foundation for CloudTrail to OCSF conversion. While it handles the core requirements, you can extend and customize it to address your specific security monitoring and compliance needs while maintaining OCSF compliance.

NOTE: this transformation does NOT work with Cloudtrail Logs Insights.

Sample Record

Code
{
  "eventVersion": "1.08",
  "userIdentity": {
    "type": "IAMUser",
    "principalId": "2481b50d-82ca-156d-0d68-21eee8e9a547",
    "arn": "arn:aws:iam::1898:user/Jane Johnson",
    "accountId": "2238",
    "accessKeyId": "17110a2d-24d2-c33a-621b-1841302ddca3",
    "userName": "Sarah Jones",
    "sessionContext": {
      "sessionIssuer": {},
      "webIdFederationData": {},
      "attributes": {
        "creationDate": "2025-08-11T23:46:57.369Z",
        "mfaAuthenticated": "false"
      }
    }
  },
  "eventTime": "2025-08-11T23:46:57Z",
  "eventSource": "ec2.amazonaws.com",
  "eventName": "StartInstances",
  "awsRegion": "us-west-1",
  "sourceIPAddress": "34.111.221.159",
  "userAgent": "aws-cli/2.13.5 Python/3.11.4 Linux/4.14.255-314-253.539.amzn2.x86_64 exec-env/CloudShell exe/x86_64.amzn.2 prompt/off command/ec2.start-instances",
  "requestParameters": {
    "instancesSet": {
      "items": [
        {
          "instanceId": "i-eb1d02b3-27ad-bf79-a180-582a876a9258"
        },
        {
          "instanceId": "i-0c08cb37-ba43-8b78-8f42-70d0a8fa230a"
        }
      ]
    }
  },
  "responseElements": {
    "requestId": "44eb719c-9db7-c45a-e537-e5ff0a20b668",
    "instancesSet": {
      "items": [
        {
          "instanceId": "i-13230666-26bd-136b-1db3-e597d1c2e9ff",
          "currentState": {
            "code": 0,
            "name": "pending"
          },
          "previousState": {
            "code": 80,
            "name": "stopped"
          }
        },
        {
          "instanceId": "i-0ff941c5-b2f7-6c35-0bc4-3b6e145ab8ca",
          "currentState": {
            "code": 0,
            "name": "pending"
          },
          "previousState": {
            "code": 80,
            "name": "stopped"
          }
        }
      ]
    }
  },
  "requestID": "f2951263-f0e7-6e76-86c2-cc7a20ec82d5",
  "eventID": "6a4b8177-c342-7676-f096-267c3e241bb7",
  "readOnly": false,
  "eventType": "AwsApiCall",
  "managementEvent": true,
  "recipientAccountId": "3739",
  "eventCategory": "Management",
  "tlsDetails": {
    "tlsVersion": "TLSv1.2",
    "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
    "clientProvidedHostHeader": "ec2.us-east-1.amazonaws.com"
  },
  "sessionCredentialFromConsole": "false"
}