Falcon Data Replicator

Collects and ingests Falcon Data Replicator (FDR) records from CrowdStrike's AWS S3 bucket via SQS notifications.

Details

The CrowdStrike Falcon Data Replicator (FDR) collector is designed to automatically ingest security event data from CrowdStrike's Falcon platform. It works by:

1. Receiving notifications from an AWS SQS queue when new data is available in the configured S3 bucket
2. Processing the SQS messages to identify new FDR files
3. Downloading and processing the FDR files from S3
4. Ingesting the security event data into your pipeline

Features

• Automatic processing of FDR files via SQS notifications
• Gzip decompression of FDR files
• Configurable visibility timeout for SQS messages
• Configurable cron schedule for checking new messages
• Comprehensive logging and error handling
• Connection testing for both SQS and S3 services -->

Prerequisites

Before using this collector, you need:

1. A CrowdStrike Falcon account with FDR enabled
2. AWS credentials with access to:
   • The SQS queue configured for FDR notifications
   • The S3 bucket containing FDR files
3. The following information from CrowdStrike:
   • AWS region name
   • SQS queue URL
   • S3 bucket URL
   • AWS access key
   • AWS secret key

Related Articles

• CrowdStrike Falcon Data Replicator Documentation
• AWS SQS Documentation
• AWS S3 Documentation

Configuration

The collector requires the following configuration:

{
  "settings": {
    "aws_region_name": "us-west-2",
    "aws_queue_url": "https://sqs.region.amazonaws.com/account/queue-name",
    "aws_s3_url": "s3://bucket-name",
    "visibility_timeout": 300,
    "cron": "0 */1 * * *"
  },
  "secrets": {
    "aws_access_key": "your-aws-access-key",
    "aws_secret_key": "your-aws-secret-key"
  }
}

Settings

Setting	Type	Required	Description
aws_region_name	string	Yes	AWS region where the SQS queue and S3 bucket are located (e.g., us-east-1)
aws_queue_url	string	Yes	URL of the SQS queue configured for FDR notifications
aws_s3_url	string	Yes	URL of the S3 bucket containing FDR files
visibility_timeout	integer	Yes	Time in seconds before a message is returned to the SQS queue if not deleted
cron	string	Yes	Cron schedule for checking new messages (e.g., "0 */1 * * *" for every hour)

Secrets

Secret	Type	Required	Description
aws_access_key	string	Yes	AWS access key with permissions for SQS and S3
aws_secret_key	string	Yes	AWS secret key associated with the access key

API

To send a POST request to create this Crowdstrike Falcon Data Replicator (FDR) Connector.

NOTE: This curl command will create new secrets. If you wish to use and existing secret you've already created you can pass the id filed instead of name, value, and description.

curl -X 'POST' \
  '{base_url}/api/v2/{org_id}/inputs' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer {token}' \
  -H 'Content-Type: application/json' \
  -d '{
  "config": {
    "secrets": {
      "aws_access_key": {
       "name": "Crowdstrike FDR AWS Access Key",
       "description": "Crowdstrike FDR AWS Access Key",
       "value": "your-access-key"
     },
     "aws_secret_key": {
       "name": " Crowdstrike FDR AWS Secret Key",
       "description": "Crowdstrike FDR AWS Secret Key",
       "value": "your-secret-key"
     }
    },
    "settings": {
      "aws_region_name": "us-west-2",
      "aws_queue_url": "https://sqs.region.amazonaws.com/account/queue-name",
      "aws_s3_url": "s3://bucket-name",
      "visibility_timeout": 300,
      "cron": "0 */1 * * *"
    }
  },
  "description": "input_description",
  "name": "input_name",
  "promise_id": "",
  "type": "crowdstrike-falcon-data-replicator"
}'

Data Format

The collector processes FDR files in JSON format. Each line in the FDR file represents a separate security event. The events are forwarded to your pipeline as-is, maintaining the original JSON structure.

Data Retention

Important: CrowdStrike Falcon Data Replicator (FDR) retains data in the S3 bucket for only 7 days. After this period, CrowdStrike automatically removes the data from the S3 bucket. This means:

1. You must configure an appropriate collection schedule to ensure all data is collected within the 7-day window
2. Any interruption in your collection process that lasts longer than 7 days may result in permanent data loss
3. Historical data beyond 7 days is not available through the FDR service

It's recommended to:

• Set up your cron schedule to run at least daily
• Implement monitoring to detect collection failures
• Configure alerting for collection gaps that approach the 7-day limit

Error Handling

The collector implements comprehensive error handling:

1. SQS Connection Errors:

   • Logs detailed error messages
   • Continues processing other messages
   • Retries failed operations based on visibility timeout

2. S3 Operation Errors:

   • Logs file-specific errors
   • Continues processing other files
   • Maintains transaction integrity

3. File Processing Errors:

   • Logs line-specific errors
   • Continues processing other lines
   • Maintains data consistency

Monitoring

The collector provides the following monitoring capabilities:

1. Logging:

   • Debug logs for detailed operations
   • Info logs for major events
   • Error logs for failures
   • Structured logging with relevant fields

2. Metrics:

   • Number of messages processed
   • Number of files processed
   • Number of lines processed
   • Processing time per file

Troubleshooting

Common issues and solutions:

1. SQS Connection Issues:

   • Verify AWS credentials
   • Check SQS queue URL
   • Ensure proper IAM permissions

2. S3 Access Issues:

   • Verify bucket URL
   • Check bucket permissions
   • Validate AWS region

3. Missing Data Issues:

   • Check if data is older than 7 days (CrowdStrike retention limit)
   • Verify your collection schedule is running frequently enough
   • Ensure no collection gaps exceed the 7-day retention period

Sample Record

{
  "ConfigBuild": "9001.8.4731773.95",
  "ConfigIDBase": "65113064",
  "ConfigIDBuild": "66986",
  "ConfigIDPlatform": "7",
  "ConfigStateHash": "4961845598",
  "ConfigurationVersion": "4",
  "EffectiveTransmissionClass": "2",
  "Entitlements": "11",
  "EventOrigin": "9",
  "NetworkContainmentState": "1",
  "ProvisionState": "1",
  "SensorStateBitMap": "8",
  "aid": "2cdea456-0868-5050-91e8-d893836caa89",
  "aip": "2406:b400:d11:f32b:a97a:2ddd:5b97:ec4",
  "cid": "a1652781-1ddd-fce9-e869-3785865bf1de",
  "event_platform": "Linux",
  "event_simpleName": "SensorHeartbeat",
  "id": "1dfdc533-2092-5d73-1524-309c6756233e",
  "name": "SensorHeartbeatLinuxV4",
  "timestamp": "1754955992776"
}