# Firehose Stream

import SelfHostedIngestEndpoint from "../../../partials/ingest/self-hosted-http-endpoint.mdx";

**Sync Type:** Streaming

## Overview

Receives data via Amazon Data Firehose HTTP endpoint delivery. This is a generic receiver that accepts any data sent through an Amazon Data Firehose stream configured with an HTTP endpoint destination. Unlike the CloudWatch Logs Stream input, this input is not limited to CloudWatch data — it can receive any vendor's data via Firehose.

The input automatically handles:
- Base64 decoding of records
- Optional gzip decompression
- JSON data pass-through
- Non-JSON data wrapping in an envelope with Firehose metadata

## Requirements

- An AWS account with Amazon Data Firehose
- A Monad organization API key with `pipeline:data:write` permission
- AWS permissions to create Amazon Data Firehose delivery stream
- Ability to configure Firehose to send data to an HTTP endpoint

## Setup Instructions

### Step 1: Create the Monad Pipeline

1. In the Monad platform, create a new pipeline with the **Firehose Stream** input
2. Note the **Pipeline ID** from the pipeline settings — you'll need this for the Firehose configuration

### Step 2: Create a Monad API Key

1. In the Monad platform, navigate to **Settings > API Keys**
2. Click **Create new API key**
3. Give it a descriptive name (e.g., "AWS Firehose Receiver")
4. Ensure the API key has the `pipeline:data:write` permission (included in Contributor and System Administrator roles)
5. Copy and securely store the API key value — it won't be visible again

### Step 3: Create an Amazon Data Firehose Delivery Stream

1. Navigate to the **Amazon Data Firehose** console in AWS
2. Click **Create Firehose stream**
3. Configure the stream:
   - **Source:** Select the source for your data (Direct PUT, Kinesis Data Stream, etc.)
   - **Destination:** Select `HTTP Endpoint`
4. Configure the HTTP endpoint destination:
   - **HTTP endpoint URL:** `https://<pipeline-id>.data.monad.com`
   - Replace `<pipeline-id>` with your Monad pipeline ID
   - **Access key:** Enter your Monad API key (this is sent as the `X-Amz-Firehose-Access-Key` header)
   - **Content encoding:** Select `GZIP` (recommended for reduced bandwidth)
5. Configure buffering hints:
   - **Buffer size:** 1-5 MiB (recommended: 1 MiB for lower latency)
   - **Buffer interval:** 60-300 seconds (recommended: 60 seconds for near real-time)
6. Configure backup settings as needed
7. Create the delivery stream

<SelfHostedIngestEndpoint />

### Step 4: Start Sending Data

Configure your data source (whether Kinesis stream, application, or other source) to put records into the Firehose delivery stream. Records will be automatically forwarded to your Monad pipeline.

## Configuration

This input requires no configuration in Monad beyond creating the pipeline. All configuration is done on the AWS side.

### Settings

| Setting | Type | Required | Description |
|---------|------|----------|-------------|
| None | - | - | No settings required |

### Secrets

| Secret | Type | Required | Description |
|--------|------|----------|-------------|
| None | - | - | No secrets required |

## Authentication

Authentication is handled via the Monad API key that you configure in the Firehose HTTP endpoint destination. The API key is sent by Firehose in the `X-Amz-Firehose-Access-Key` header.

**API Key Requirements:**

- Must be an organization API key (not a personal API key)
- Must have the `pipeline:data:write` permission

## Data Format

### JSON Data

If a record contains valid JSON, it is passed through unchanged:

```json
{
  "event": "user_login",
  "user_id": "12345",
  "timestamp": 1706540800000
}
```

### Non-JSON Data

If a record contains non-JSON data (plain text, binary, etc.), it is automatically wrapped in an envelope with Firehose metadata:

```json
{
  "data": "plain text log line",
  "firehoseRequestId": "firehose-req-xyz789",
  "firehoseTimestamp": 1706540799000
}
```

### Fields

| Field | Type | Description |
|-------|------|-------------|
| `data` | string | The record data (for non-JSON records only) |
| `firehoseRequestId` | string | Unique identifier for the Firehose request |
| `firehoseTimestamp` | integer | Firehose delivery timestamp in Unix milliseconds |

## Rate Limits

Amazon Data Firehose has the following service-level rate limits:

| Scope | Limit | Notes |
|-------|-------|-------|
| Records per second | 5,000 records/sec | Per delivery stream |
| Data throughput | 5 MB/sec | Per delivery stream |

You can request limit increases through AWS Support. The limits apply per Firehose delivery stream, not per HTTP endpoint destination.

**Source:** [AWS Kinesis Data Firehose Quotas and Limits](https://docs.aws.amazon.com/kinesis/latest/dev/service-size-limits.html)

## Troubleshooting

### Common Issues

**Issue: Data is not appearing in Monad**

Possible Causes:
- Firehose delivery stream is not active
- API key is invalid or lacks permissions
- HTTP endpoint URL is incorrect
- Network connectivity issues

Solution:
- Check the Firehose stream monitoring metrics in the AWS console
- Verify the API key has `pipeline:data:write` permission
- Confirm the endpoint URL matches `https://<pipeline-id>.data.monad.com`
- Check AWS VPC and security group configurations if using VPC endpoints

**Issue: Firehose shows delivery failures**

Possible Causes:
- Incorrect HTTP endpoint URL
- Invalid API key in the access key field
- Firehose cannot reach the Monad endpoint

Solution:
- Verify the endpoint URL is correct and accessible
- Regenerate and update the API key in the Firehose configuration

**Issue: Records are being wrapped unexpectedly**

If your records are being wrapped in the envelope format (with `data`, `firehoseRequestId`, and `firehoseTimestamp` fields), it means the input detected non-JSON data. Ensure your data is valid JSON if you want it to pass through unchanged.

## Related Articles

- [Amazon Data Firehose HTTP Endpoint Destinations](https://docs.aws.amazon.com/kinesis/latest/dev/extended-s3-configure-dcs.html)
- [Amazon Data Firehose API Reference](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html)
- [Amazon Data Firehose Quotas and Limits](https://docs.aws.amazon.com/kinesis/latest/dev/service-size-limits.html)

## Sample Record

```json
{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "timestamp": 1706540800000,
  "event_type": "log",
  "source": "application",
  "data": {
    "message": "Event processed",
    "level": "INFO",
    "correlation_id": "f47ac10b-58cc-4372-a567-0e02b2c3d479"
  }
}
```

## Sync frequency

This is a push (receiver) input: Monad ingests records as the source sends them, so there is no polling interval. See [Input Sync Frequency](../../guides/sync-frequency) for details.