HTTP Output

The HTTP Output allows you to send data from your pipeline to any HTTP endpoint. This versatile output can be configured to work with a wide variety of APIs and web services, making it easy to integrate your data with external systems.

Configuration

The HTTP Output is configured using the following settings:

Settings

Setting	Type	Required	Default	Description
endpoint	string	Yes	-	The full URL of the HTTP endpoint to send data to. Must include the scheme (http or https).
method	string	Yes	-	The HTTP method to use for requests (GET, POST, PUT, PATCH, or DELETE).
rate_limit	integer	Yes	-	Maximum number of requests per second to send to the endpoint.
payload_structure	string	Yes	-	Determines how the payload is structured. Options are 'single', 'array', or 'wrapped'.
wrapper_key	string	No	-	The key to use for wrapping the payload when payload_structure is set to 'wrapped'.
max_batch_data_size	integer	No	1024	The maximum size in KB for a single batch of data to be sent in one request. This does not affect the 'single' payload structure.
max_batch_record_count	integer	No	1000	The maximum number of records to include in a single batch. For 'single' payload structure, this is automatically set to 1.
tls_skip_verify	boolean	No	false	When set to true, skips TLS certificate verification for the endpoint.
headers	array of objects	No	-	Custom Non-secret headers to be included with each request. Each header object must contain 'header_key' and 'header_value' fields.

Secrets

Setting	Type	Required	Description
auth_headers	object	No	Authentication headers to be included with each request. Used for sensitive information like API keys or tokens.

Payload Structures

The HTTP Output supports three payload structures:

1. Single: Each record is sent as a separate HTTP request.
2. Array: Multiple records are sent as an array in a single HTTP request.
3. Wrapped: Multiple records are sent as an array within a wrapper object, using the specified wrapper_key.

Examples

Single Payload Structure

{
  "id": 1,
  "name": "John Doe"
}

Array Payload Structure

[
  {"id": 1, "name": "John Doe"},
  {"id": 2, "name": "Jane Smith"}
]

Wrapped Payload Structure

{
  "data": [
    {"id": 1, "name": "John Doe"},
    {"id": 2, "name": "Jane Smith"}
  ]
}

Best Practices

1. Payload Structure: Choose the payload_structure that best matches the target API's expectations. Using 'array' or 'wrapped' can significantly reduce the number of HTTP requests for high-volume data.

2. Use HTTPS: Always use HTTPS for secure data transmission, especially when dealing with sensitive information.

3. TLS Verification: By default, TLS certificate verification is enabled for HTTPS connections. Only set tls_skip_verify to true if you're working with self-signed certificates or in testing environments. Disabling TLS verification in production can pose security risks.

4. Rate Limiting: Set an appropriate rate_limit to avoid overwhelming the target API. Check the API's documentation for recommended limits.

5. Batch Size: Adjust max_batch_data_size and max_batch_record_count based on your API's requirements and the size of your records.

6. Error Handling: The output will automatically retry failed requests. Monitor your logs for persistent errors.

7. Authentication: Use the auth_headers setting to include API keys, tokens, or other sensitive authentication methods required by your API. Use headers for non-sensitive headers.

User Agent

All requests sent by the HTTP Output include a custom Monad User Agent. This helps identify requests coming from your Monad pipeline to the receiving API. "Monad User Agent - Contact: security@monad.com"

Limitations

• Batch size is governed by max_batch_data_size and max_batch_record_count. However, to ensure a steady data flow, batches are automatically sent every 5 seconds, even if they haven't reached the maximum size.
• Redirects are not followed automatically. Ensure your endpoint URL is correct and does not redirect.
• All data is sent as JSON. Binary data is not supported.
• While custom headers can be set using the headers and auth_headers settings, not all headers will be honored in terms of data processing:
  • If you set the Content-Type header to a value other than application/json (e.g., application/xml), the data will still be sent as JSON.
  • Compression headers (e.g., Content-Encoding: gzip) will not result in the data being compressed. The output does not perform data compression.
• The output component prioritizes its internal data handling mechanisms over some custom header settings to ensure consistent behavior and performance.
• Success is determined by HTTP status codes in the range of 200-299. Any response with a status code outside this range is considered a failure:
  • A status code between 200 and 299 (inclusive) is treated as a success, and the batch is considered successfully sent.
  • Any status code outside this range (e.g., 3xx, 4xx, 5xx) is treated as a failure.
  • In case of a failure (status code outside 200-299), the entire batch of messages will be retried according to the Monad's retry policy.
• HTTP Request Timeout is defaulted to 30 seconds. Any request that takes longer than this will be treated as a failure.

Troubleshooting

• If you're seeing rate limit errors, try decreasing the rate_limit setting.
• For "request entity too large" errors, reduce your max_batch_data_size or max_batch_record_count.
• If requests are timing out, check the API's status and your network connection.