# HTTP Output

The HTTP Output allows you to send data from your pipeline to any HTTP endpoint. This versatile output can be configured to work with a wide variety of APIs and web services, making it easy to integrate your data with external systems.

## Configuration

The HTTP Output is configured using the following settings:

### Settings

| Setting | Type | Required | Default | Description |
|---------|------|----------|---------|-------------|
| endpoint | string | Yes | - | The full URL of the HTTP endpoint to send data to. Must include the scheme (http or https). |
| method | string | Yes | - | The HTTP method to use for requests (GET, POST, PUT, PATCH, or DELETE). |
| rate_limit | integer | Yes | - | Maximum number of requests per second to send to the endpoint. |
| payload_structure | string | Yes | - | Determines how the payload is structured. Options are 'single', 'array', or 'wrapped'. |
| wrapper_key | string | No | - | The key to use for wrapping the payload when payload_structure is set to 'wrapped'. |
| max_batch_data_size | integer | No | 1024 | The maximum size in KB for a single batch of data to be sent in one request. This does not affect the 'single' payload structure. |
| max_batch_record_count | integer | No | 1000 | The maximum number of records to include in a single batch. For 'single' payload structure, this is automatically set to 1. |
| tls_skip_verify | boolean | No | false | When set to true, skips TLS certificate verification for the endpoint. |
| headers | array of objects | No | - | Custom Non-secret headers to be included with each request. Each header object must contain 'header_key' and 'header_value' fields. |

### Secrets

| Setting | Type | Required | Description |
|---------|------|----------|-------------|
| auth_headers | object | No | Authentication headers to be included with each request. Used for sensitive information like API keys or tokens. |

## Payload Structures

The HTTP Output supports three payload structures:

1. **Single**: Each record is sent as a separate HTTP request.
2. **Array**: Multiple records are sent as an array in a single HTTP request.
3. **Wrapped**: Multiple records are sent as an array within a wrapper object, using the specified `wrapper_key`.

### Examples

#### Single Payload Structure
```json
{
  "id": 1,
  "name": "John Doe"
}
```

#### Array Payload Structure
```json
[
  {"id": 1, "name": "John Doe"},
  {"id": 2, "name": "Jane Smith"}
]
```

#### Wrapped Payload Structure
```json
{
  "data": [
    {"id": 1, "name": "John Doe"},
    {"id": 2, "name": "Jane Smith"}
  ]
}
```


## Best Practices
1. **Payload Structure**: Choose the `payload_structure` that best matches the target API's expectations. Using 'array' or 'wrapped' can significantly reduce the number of HTTP requests for high-volume data.

2. **Use HTTPS**: Always use HTTPS for secure data transmission, especially when dealing with sensitive information.

3. **TLS Verification**: By default, TLS certificate verification is enabled for HTTPS connections. Only set `tls_skip_verify` to true if you're working with self-signed certificates or in testing environments. Disabling TLS verification in production can pose security risks.

4. **Rate Limiting**: Set an appropriate `rate_limit` to avoid overwhelming the target API. Check the API's documentation for recommended limits.

5. **Batch Size**: Adjust `max_batch_data_size` and `max_batch_record_count` based on your API's requirements and the size of your records.

6. **Error Handling**: The output will automatically retry failed requests. Monitor your logs for persistent errors.

7. **Authentication**: Use the `auth_headers` setting to include API keys, tokens, or other sensitive authentication methods required by your API. Use `headers` for non-sensitive headers.

## User Agent

All requests sent by the HTTP Output include a custom Monad User Agent. This helps identify requests coming from your Monad pipeline to the receiving API.
`"Monad User Agent - Contact: security@monad.com"`

## Limitations

- Batch size is governed by max_batch_data_size and max_batch_record_count. However, to ensure a steady data flow, batches are automatically sent every 5 seconds, even if they haven't reached the maximum size.
- Redirects are not followed automatically. Ensure your `endpoint` URL is correct and does not redirect.
- All data is sent as JSON. Binary data is not supported.
- While custom headers can be set using the `headers` and `auth_headers` settings, not all headers will be honored in terms of data processing:
  - If you set the `Content-Type` header to a value other than `application/json` (e.g., `application/xml`), the data will still be sent as JSON.
  - Compression headers (e.g., `Content-Encoding: gzip`) will not result in the data being compressed. The output does not perform data compression.
- The output component prioritizes its internal data handling mechanisms over some custom header settings to ensure consistent behavior and performance.
- Success is determined by HTTP status codes in the range of 200-299. Any response with a status code outside this range is considered a failure:
  - A status code between 200 and 299 (inclusive) is treated as a success, and the batch is considered successfully sent.
  - Any status code outside this range (e.g., 3xx, 4xx, 5xx) is treated as a failure.
  - In case of a failure (status code outside 200-299), the entire batch of messages will be retried according to the Monad's retry policy.
- HTTP Request Timeout is defaulted to 30 seconds. Any request that takes longer than this will be treated as a failure.

## Troubleshooting

- If you're seeing rate limit errors, try decreasing the `rate_limit` setting.
- For "request entity too large" errors, reduce your `max_batch_data_size` or `max_batch_record_count`.
- If requests are timing out, check the API's status and your network connection.