Skip to main content

Firehose Stream

Sync Type: Streaming

Overview​

Receives data via Amazon Data Firehose HTTP endpoint delivery. This is a generic receiver that accepts any data sent through an Amazon Data Firehose stream configured with an HTTP endpoint destination. Unlike the CloudWatch Logs Stream input, this input is not limited to CloudWatch data — it can receive any vendor's data via Firehose.

The input automatically handles:

  • Base64 decoding of records
  • Optional gzip decompression
  • JSON data pass-through
  • Non-JSON data wrapping in an envelope with Firehose metadata

Requirements​

  • An AWS account with Amazon Data Firehose
  • A Monad organization API key with pipeline:data:write permission
  • AWS permissions to create Amazon Data Firehose delivery stream
  • Ability to configure Firehose to send data to an HTTP endpoint

Setup Instructions​

Step 1: Create the Monad Pipeline​

  1. In the Monad platform, create a new pipeline with the Firehose Stream input
  2. Note the Pipeline ID from the pipeline settings — you'll need this for the Firehose configuration

Step 2: Create a Monad API Key​

  1. In the Monad platform, navigate to Settings > API Keys
  2. Click Create new API key
  3. Give it a descriptive name (e.g., "AWS Firehose Receiver")
  4. Ensure the API key has the pipeline:data:write permission (included in Contributor and System Administrator roles)
  5. Copy and securely store the API key value — it won't be visible again

Step 3: Create an Amazon Data Firehose Delivery Stream​

  1. Navigate to the Amazon Data Firehose console in AWS
  2. Click Create Firehose stream
  3. Configure the stream:
    • Source: Select the source for your data (Direct PUT, Kinesis Data Stream, etc.)
    • Destination: Select HTTP Endpoint
  4. Configure the HTTP endpoint destination:
    • HTTP endpoint URL: https://app.monad.com/api/v2/http/send/{PIPELINE_ID}
    • Replace {PIPELINE_ID} with your Monad pipeline ID
    • Access key: Enter your Monad API key (this is sent as the X-Amz-Firehose-Access-Key header)
    • Content encoding: Select GZIP (recommended for reduced bandwidth)
  5. Configure buffering hints:
    • Buffer size: 1-5 MiB (recommended: 1 MiB for lower latency)
    • Buffer interval: 60-300 seconds (recommended: 60 seconds for near real-time)
  6. Configure backup settings as needed
  7. Create the delivery stream

Step 4: Start Sending Data​

Configure your data source (whether Kinesis stream, application, or other source) to put records into the Firehose delivery stream. Records will be automatically forwarded to your Monad pipeline.

Configuration​

This input requires no configuration in Monad beyond creating the pipeline. All configuration is done on the AWS side.

Settings​

SettingTypeRequiredDescription
None--No settings required

Secrets​

SecretTypeRequiredDescription
None--No secrets required

Authentication​

Authentication is handled via the Monad API key that you configure in the Firehose HTTP endpoint destination. The API key is sent by Firehose in the X-Amz-Firehose-Access-Key header.

API Key Requirements:

  • Must be an organization API key (not a personal API key)
  • Must have the pipeline:data:write permission

Data Format​

JSON Data​

If a record contains valid JSON, it is passed through unchanged:

{
  "event": "user_login",
  "user_id": "12345",
  "timestamp": 1706540800000
}

Non-JSON Data​

If a record contains non-JSON data (plain text, binary, etc.), it is automatically wrapped in an envelope with Firehose metadata:

{
  "data": "plain text log line",
  "firehoseRequestId": "firehose-req-xyz789",
  "firehoseTimestamp": 1706540799000
}

Fields​

FieldTypeDescription
datastringThe record data (for non-JSON records only)
firehoseRequestIdstringUnique identifier for the Firehose request
firehoseTimestampintegerFirehose delivery timestamp in Unix milliseconds

Rate Limits​

Amazon Data Firehose has the following service-level rate limits:

ScopeLimitNotes
Records per second5,000 records/secPer delivery stream
Data throughput5 MB/secPer delivery stream

You can request limit increases through AWS Support. The limits apply per Firehose delivery stream, not per HTTP endpoint destination.

Source: AWS Kinesis Data Firehose Quotas and Limits

Troubleshooting​

Common Issues​

Issue: Data is not appearing in Monad

Possible Causes:

  • Firehose delivery stream is not active
  • API key is invalid or lacks permissions
  • HTTP endpoint URL is incorrect
  • Network connectivity issues

Solution:

  • Check the Firehose stream monitoring metrics in the AWS console
  • Verify the API key has pipeline:data:write permission
  • Confirm the endpoint URL matches https://app.monad.com/api/v2/http/send/{PIPELINE_ID}
  • Check AWS VPC and security group configurations if using VPC endpoints

Issue: Firehose shows delivery failures

Possible Causes:

  • Incorrect HTTP endpoint URL
  • Invalid API key in the access key field
  • Firehose cannot reach the Monad endpoint

Solution:

  • Verify the endpoint URL is correct and accessible
  • Regenerate and update the API key in the Firehose configuration

Issue: Records are being wrapped unexpectedly

If your records are being wrapped in the envelope format (with data, firehoseRequestId, and firehoseTimestamp fields), it means the input detected non-JSON data. Ensure your data is valid JSON if you want it to pass through unchanged.

Sample Record​

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "timestamp": 1706540800000,
  "event_type": "log",
  "source": "application",
  "data": {
    "message": "Event processed",
    "level": "INFO",
    "correlation_id": "f47ac10b-58cc-4372-a567-0e02b2c3d479"
  }
}