# CloudWatch Logs Stream

import SelfHostedIngestEndpoint from "../../../partials/ingest/self-hosted-http-endpoint.mdx";

**Sync Type:** Streaming

## Overview

Receives CloudWatch Logs in real-time via Amazon Data Firehose HTTP endpoint delivery. Instead of polling the CloudWatch Logs API, this input receives log events as they are pushed from AWS through a Firehose delivery stream configured with an HTTP endpoint destination.

## Requirements

- An AWS account with CloudWatch Logs
- A Monad organization API key with `pipeline:data:write` permission
- AWS permissions to create:
- Amazon Data Firehose delivery stream
- CloudWatch Logs subscription filter

## Setup Instructions

### Step 1: Create the Monad Pipeline

1. In the Monad platform, create a new pipeline with the **AWS CloudWatch Logs Stream** input
2. Note the **Pipeline ID** from the pipeline settings - you'll need this for the Firehose configuration

### Step 2: Create a Monad API Key

1. In the Monad platform, navigate to **Settings > API Keys**
2. Click **Create new API key**
3. Give it a descriptive name (e.g., "AWS Firehose CloudWatch Logs")
4. Ensure the API key has the `pipeline:data:write` permission (included in Contributor and System Administrator roles)
5. Copy and securely store the API key value - it won't be visible again

### Step 3: Create an Amazon Data Firehose Delivery Stream

1. Navigate to the **Amazon Data Firehose** console in AWS
2. Click **Create Firehose stream**
3. Configure the stream:
- **Source:** Select `Direct PUT`
- **Destination:** Select `HTTP Endpoint`
4. Configure the HTTP endpoint destination:
- **HTTP endpoint URL:** `https://<pipeline-id>.data.monad.com`
- Replace `<pipeline-id>` with your Monad pipeline ID
- **Access key:** Enter your Monad API key (this is sent as the `X-Amz-Firehose-Access-Key` header)
- **Content encoding:** Select `GZIP` (recommended for reduced bandwidth)
5. Configure buffering hints:
- **Buffer size:** 1-5 MiB (recommended: 1 MiB for lower latency)
- **Buffer interval:** 60-300 seconds (recommended: 60 seconds for near real-time)
6. Configure backup settings as needed
7. Create the delivery stream and note the Firehose ARN

<SelfHostedIngestEndpoint />

### Step 4: Create an IAM Role for CloudWatch Logs

CloudWatch Logs needs permission to write to your Firehose stream. Create an IAM role with the following configuration:

**Trust policy** (allows CloudWatch Logs to assume the role):

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.amazonaws.com"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringLike": {
                    "aws:SourceArn": "arn:aws:logs:*:YOUR_ACCOUNT_ID:*"
                }
            }
        }
    ]
}
```

Replace `YOUR_ACCOUNT_ID` with your AWS account ID.

**Permissions policy** (allows writing to Firehose):

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "firehose:PutRecord",
                "firehose:PutRecordBatch"
            ],
            "Resource": "arn:aws:firehose:*:YOUR_ACCOUNT_ID:deliverystream/*"
        }
    ]
}
```

Replace `YOUR_ACCOUNT_ID` with your AWS account ID. You can also scope this down to a specific delivery stream ARN for tighter security.

### Step 5: Create a CloudWatch Logs Subscription Filter

1. Navigate to the **CloudWatch Logs** console
2. Select the log group you want to stream
3. Click the **Subscription filters** tab
4. Click **Create** → **Create Kinesis Data Firehose subscription filter**
5. Configure the subscription filter:
- **Firehose stream:** Select the delivery stream created in Step 3
- **IAM role:** Select the role created in Step 4
- **Filter pattern:** Enter a filter pattern (leave empty to send all logs)
- **Filter name:** Give it a descriptive name
6. Create the subscription filter

## Configuration

This input requires no configuration in Monad beyond creating the pipeline. All configuration is done on the AWS side.

### Settings

| Setting | Type | Required | Description |
|---------|------|----------|-------------|
| None | - | - | No settings required |

### Secrets

| Secret | Type | Required | Description |
|--------|------|----------|-------------|
| None | - | - | No secrets required |

## Authentication

Authentication is handled via the Monad API key that you configure in the Firehose HTTP endpoint destination. The API key is sent by Firehose in the `X-Amz-Firehose-Access-Key` header.

**API Key Requirements:**

- Must be an organization API key (not a personal API key)
- Must have the `pipeline:data:write` permission

## Data Format

Each log event is enriched with metadata from both CloudWatch and Firehose before being delivered to the pipeline:

| Field | Type | Description |
|-------|------|-------------|
| `id` | string | Unique identifier for the log event |
| `timestamp` | integer | Log event timestamp in Unix milliseconds |
| `message` | object/string | The log message (preserved as JSON if valid, otherwise as string) |
| `owner` | string | AWS account ID that owns the log group |
| `logGroup` | string | Name of the CloudWatch log group |
| `logStream` | string | Name of the CloudWatch log stream |
| `subscriptionFilters` | array | Names of subscription filters that matched |
| `firehoseRequestId` | string | Firehose request ID for tracing |
| `firehoseTimestamp` | integer | Firehose delivery timestamp in Unix milliseconds |

## Troubleshooting

### Common Issues

**Issue: Logs are not appearing in Monad**

Possible Causes:
- Firehose delivery stream is not active
- API key is invalid or lacks permissions
- Subscription filter pattern is too restrictive

Solution:
- Check the Firehose stream monitoring metrics in AWS console
- Verify the API key has `pipeline:data:write` permission
- Review the subscription filter pattern

**Issue: Firehose shows delivery failures**

Possible Causes:
- Incorrect HTTP endpoint URL
- Invalid API key in the access key field
- Network connectivity issues

Solution:
- Verify the endpoint URL matches `https://<pipeline-id>.data.monad.com`
- Regenerate and update the API key
- Check AWS VPC and security group configurations

## Related Articles

- [Amazon Data Firehose HTTP Endpoint Destinations](https://docs.aws.amazon.com/firehose/latest/dev/http_deliverystream.html)
- [CloudWatch Logs Subscription Filters](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html)
- [Using Subscription Filters with Amazon Data Firehose](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html#FirehoseExample)

## Sample Record

```json
{
  "id": "38504832190534723847234234234234234234",
  "timestamp": 1706540800000,
  "message": {
    "level": "INFO",
    "action": "Request processed",
    "requestId": "abc123-def456-ghi789",
    "duration_ms": 127
  },
  "owner": "123456789012",
  "logGroup": "/aws/lambda/my-function",
  "logStream": "2024/01/29/[$LATEST]abc123def456",
  "subscriptionFilters": ["monad-subscription"],
  "firehoseRequestId": "firehose-req-xyz789",
  "firehoseTimestamp": 1706540799000
}
```

## Sync frequency

This is a push (receiver) input: Monad ingests records as the source sends them, so there is no polling interval. See [Input Sync Frequency](../../guides/sync-frequency) for details.