CloudTrail Events

Collects account activity and API usage across your AWS infrastructure for full visibility into control plane events.

Details

On initialization for each sync, Monad generates every account_id + region combination found in your logs bucket. This ensures that any new regions or accounts added are picked up automatically on the next run.

During the first run, Monad performs a full sync of all CloudTrail logs. For subsequent runs, each AWS account_id + region combination maintains its own state to make sure only incremental data is processed. On restart in case of any form of failure, each partition resumes from its last checkpointed timestamp. A checkpoint occurs at every page within a prefix. So while processing a prefix, if a failure occurs, the processor will restart from the last completed page's checkpoint. You will not lose out on any records, however you may re-process some data in the S3 objects on the page where the failure occured in case of any catastrophic failures.

Each partition (account_id + region) is processed independently and asynchronously, ensuring quick data ingestion and isolating errors to a single partition without affecting others.

Performance Considerations and Limitations

Rate Limiting Risk: Monad performs a full date prefix scan on every sync run for each account-region combination. During each scan, previously processed records are identified and dropped based on deduplication logic. This approach has important performance implications:

• For buckets with large volumes of data on a given date, the repeated full-prefix scanning can consume significant API quota
• Since regions are processed concurrently, accounts with simultaneous data bursts across multiple regions will amplify API usage, potentially triggering S3 rate limits
• The overhead increases proportionally with data volume, as the same objects are rescanned and filtered on every run throughout the day
• Processing speed degrades as data accumulates within a date prefix, since each sync must scan and filter through increasingly larger datasets before identifying new records

Recommendation: For production deployments with high-volume CloudTrail data or multiple active accounts/regions, we strongly recommend using an S3-to-SQS event notification architecture instead of prefix-based polling. This eliminates redundant scanning and provides near-real-time ingestion without rate limit concerns.

Note

• Prefixes must be hive compliant with simple date formatting. Any other structure can cause unexpected behavior in the input.

• Each log's last updated time should align with its logical prefix date. For example, objects in the 2025/08/10 prefix should have a last updated time of 2025-08-10 (in ISO8601 format). Misalignment can cause unexpected behavior in the input.

• To avoid such tight boundaries, we recommend publishing S3 data to an SQS queue to avoid such failures.

Prerequisites

• IAM Role Assumption / Static Credentials
• Example permission to attach to the role/user:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::{bucket_name}"
    },
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::{bucket_name}/*"
    }
  ]
}

setup documentation

Related Articles

• https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-log-file-examples.html
• https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference-record-contents.html

Configuration

The following configuration defines the input parameters. Each field's specifications, such as type, requirements, and descriptions, are detailed below.

Settings

Setting	Type	Required	Description
Region	string	No	The region of the S3 bucket. If left blank, the region will be auto-detected.
Bucket	string	Yes	The name of the S3 bucket.
Prefix	string	No	Prefix of the S3 object keys to read. If your CloudTrail logs are at the standard location (starting with "AWSLogs/") at the root of the bucket, leave this field blank. Only specify a prefix if your CloudTrail logs are stored in a non-default location within your bucket.
Role ARN	string	Yes	The ARN of the role to assume to access the bucket.
Backfill Start Time	string	No	The date to start fetching data from. If not specified, no past records will be fetched.

Secrets (Static Credentials Only)

Setting	Type	Required	Description
Access Key	string	Conditional	AWS Access Key ID
Secret Key	string	Conditional	AWS Secret Access Key

⚠️ Authentication: Choose either Role ARN (recommended) or static credentials. See AWS Authentication Guide for setup instructions.

API

To send a POST request to create this Cloudtrail Connector:

curl -X 'POST' \
  '{base_url}/api/v2/{org_id}/inputs' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer {token}' \
  -H 'Content-Type: application/json' \
  -d '{
  "config": {
    "secrets": {},
    "settings": {
      "region": "region",
      "bucket": "bucket",
      "prefix": "prefix",
      "role_arn": "role_arn"
    }
  },
  "description": "input_description",
  "name": "input_name",
  "promise_id": "",
  "type": "cloudtrail"
}'

OCSF Conversion

The following JQ transformation converts AWS CloudTrail data to OCSF Version 1.1 compliant format. This conversion maps essential CloudTrail activity data to standardized OCSF fields, facilitating better integration and analysis across security tools.

The transformation is provided as a starting point and can be modified to accommodate specific needs:

{
  category_uid: 6, # Application Activity
  class_uid: 6003, # API Activity
  type_uid: 600300, # API Activity: Unknown
  activity_id: 03, # Update
  severity_id: 0, # Unkown

  time: (.eventTime | fromdateiso8601),
  time_dt: .eventTime,

  actor: {
    user: {
      name: .userIdentity.userName,
      type: .userIdentity.type,
      account: {
        name: .userIdentity.accountId,
        type: "AWS",
        uid: .userIdentity.principalId
      }
    }
  },

  api: {
    operation: .eventName,
    service: {
      name: .eventSource,
      uid: .eventSource
    },
    request: {
      uid: .requestID,
      params: .requestParameters
    },
    response: {
      result: .responseElements,
      error: (.errorCode // ""),
      error_message: (.errorMessage // "")
    }
  },

  cloud: {
    provider: "AWS",
    region: .awsRegion,
    account: {
      name: .recipientAccountId,
      uid: .recipientAccountId
    }
  },

  src_endpoint: {
    ip: .sourceIPAddress,
    type: "IPv4",
    type_id: 0
  },

  status_id: (if .errorCode != null then 1 else 0 end),
  status: (if .errorCode != null then "Failure" else "Success" end),

  resources: (
    if .resources then
      [.resources[] | {
        type: (.resourceType // "Resource"),
        uid: .resourceName,
        name: .resourceName,
        details: {
          type: .resourceType,
          arn: .ARN
        }
      }]
    else []
    end
  ),

  metadata: {
    version: "1.1.0",
    product: {
      name: "CloudTrail",
      vendor_name: "AWS"
    }
  }
}

OCSF Mapping Details

The transformation follows OCSF Version 1.1 specifications with the following key mappings:

Core Fields

• category_uid: Fixed value of 6 (API Activity)
• class_uid: Fixed value of 6003 (API Activity)
• type_uid: Set to 600303 API Activity: Update
• activity_id: Set to 0 as CloudTrail events don't map directly to OCSF activity IDs
• severity_id: Set to 0 as CloudTrail doesn't provide native severity levels

Time and Actor Information

• time: Converts CloudTrail's ISO8601 timestamp to Unix epoch using fromdateiso8601
• time_dt: Preserves original ISO8601 timestamp from CloudTrail
• actor.user: Maps IAM identity information including:
  • name: Maps from userIdentity.userName
  • type: Maps from userIdentity.type
  • account: Contains AWS account details from userIdentity

API and Cloud Context

• api: Captures API operation details:
  • operation: Maps from eventName
  • service: Contains service name and ID from eventSource
  • request/response: Preserves full request and response details
• cloud: Contains AWS-specific context:
  • provider: Fixed as "AWS"
  • region: Maps from awsRegion
  • account: Contains recipient account details

Resource Handling

• resources: Uses a generic handler to capture all resource types:
  • Processes the CloudTrail resources array
  • Preserves resource type, name, and ARN
  • Handles null/missing resources gracefully
  • Maintains consistent structure across all AWS resource types

Metadata

• metadata: Contains OCSF and product information:
  • version: Fixed as "1.1.0" per OCSF specification
  • product.name: Fixed as "CloudTrail"
  • product.vendor_name: Fixed as "AWS"
  • product.feature.name: Dynamically set to either "Data" or "Management"

Important Notes on OCSF Field Mapping

Several OCSF fields require special consideration when mapping from CloudTrail:

1. Event Classification (category_uid, class_uid)

   • All CloudTrail events are classified as API Activity
   • This is consistent across Management, Lambda Data, and S3 Data events
   • No need for complex classification logic

2. Feature Name Determination

   • Set to "Data" for:
     • Events where userIdentity.type is "AWSService"
     • Events involving Lambda or S3 resources (detected via ARN patterns)
   • Set to "Management" for all other events

3. Status Handling

   • status_id: 0 for success, 1 for failure
   • status: Maps directly from presence/absence of errorCode
   • Includes error messages when available

4. Resource Handling

   • Uses generic resource handling to support all AWS resource types
   • Maintains consistent structure regardless of resource type
   • Gracefully handles missing or null resource information

5. Optional Enhancements Consider these potential improvements based on your needs:

   • Add severity mapping based on event characteristics
   • Implement activity type classification based on event patterns
   • Add custom resource type normalization
   • Include additional context fields from CloudTrail

This transformation provides a foundation for CloudTrail to OCSF conversion. While it handles the core requirements, you can extend and customize it to address your specific security monitoring and compliance needs while maintaining OCSF compliance.

NOTE: this transformation does NOT work with Cloudtrail Logs Insights.

Sample Record

{
  "eventVersion": "1.08",
  "userIdentity": {
    "type": "IAMUser",
    "principalId": "2481b50d-82ca-156d-0d68-21eee8e9a547",
    "arn": "arn:aws:iam::1898:user/Jane Johnson",
    "accountId": "2238",
    "accessKeyId": "17110a2d-24d2-c33a-621b-1841302ddca3",
    "userName": "Sarah Jones",
    "sessionContext": {
      "sessionIssuer": {},
      "webIdFederationData": {},
      "attributes": {
        "creationDate": "2025-08-11T23:46:57.369Z",
        "mfaAuthenticated": "false"
      }
    }
  },
  "eventTime": "2025-08-11T23:46:57Z",
  "eventSource": "ec2.amazonaws.com",
  "eventName": "StartInstances",
  "awsRegion": "us-west-1",
  "sourceIPAddress": "34.111.221.159",
  "userAgent": "aws-cli/2.13.5 Python/3.11.4 Linux/4.14.255-314-253.539.amzn2.x86_64 exec-env/CloudShell exe/x86_64.amzn.2 prompt/off command/ec2.start-instances",
  "requestParameters": {
    "instancesSet": {
      "items": [
        {
          "instanceId": "i-eb1d02b3-27ad-bf79-a180-582a876a9258"
        },
        {
          "instanceId": "i-0c08cb37-ba43-8b78-8f42-70d0a8fa230a"
        }
      ]
    }
  },
  "responseElements": {
    "requestId": "44eb719c-9db7-c45a-e537-e5ff0a20b668",
    "instancesSet": {
      "items": [
        {
          "instanceId": "i-13230666-26bd-136b-1db3-e597d1c2e9ff",
          "currentState": {
            "code": 0,
            "name": "pending"
          },
          "previousState": {
            "code": 80,
            "name": "stopped"
          }
        },
        {
          "instanceId": "i-0ff941c5-b2f7-6c35-0bc4-3b6e145ab8ca",
          "currentState": {
            "code": 0,
            "name": "pending"
          },
          "previousState": {
            "code": 80,
            "name": "stopped"
          }
        }
      ]
    }
  },
  "requestID": "f2951263-f0e7-6e76-86c2-cc7a20ec82d5",
  "eventID": "6a4b8177-c342-7676-f096-267c3e241bb7",
  "readOnly": false,
  "eventType": "AwsApiCall",
  "managementEvent": true,
  "recipientAccountId": "3739",
  "eventCategory": "Management",
  "tlsDetails": {
    "tlsVersion": "TLSv1.2",
    "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
    "clientProvidedHostHeader": "ec2.us-east-1.amazonaws.com"
  },
  "sessionCredentialFromConsole": "false"
}