Copy page

Outputs

HTTP Output

The HTTP Output allows you to send data from your pipeline to any HTTP endpoint. This versatile output can be configured to work with a wide variety of APIs and web services, making it easy to integrate your data with external systems.

Configuration

The HTTP Output is configured using the following settings:

Settings

Setting	Type	Required	Default	Description
endpoint	string	Yes	-	The full URL of the HTTP endpoint to send data to. Must include the scheme (http or https).
method	string	Yes	-	The HTTP method to use for requests (GET, POST, PUT, PATCH, or DELETE).
rate_limit	integer	Yes	-	Maximum number of requests per second to send to the endpoint.
payload_structure	string	Yes	-	Determines how the payload is structured. Options are 'single', 'array', or 'wrapped'.
wrapper_key	string	No	-	The key to use for wrapping the payload when payload_structure is set to 'wrapped'.
max_batch_data_size	integer	No	1024	The maximum size in KB for a single batch of data to be sent in one request. This does not affect the 'single' payload structure.
max_batch_record_count	integer	No	1000	The maximum number of records to include in a single batch. For 'single' payload structure, this is automatically set to 1.
tls_skip_verify	boolean	No	false	When set to true, skips TLS certificate verification for the endpoint.
headers	array of objects	No	-	Custom Non-secret headers to be included with each request. Each header object must contain 'header_key' and 'header_value' fields.

Secrets

Setting	Type	Required	Description
auth_headers	object	No	Authentication headers to be included with each request. Each key is the header name to send (for example Authorization or X-API-Key) and each value is a Monad secret whose value is sent as the header value exactly as stored. See Authentication headers.

Authentication headers

The output does not build the header value for you. Each entry in auth_headers is sent as <header name>: <secret value>, with the secret value inserted exactly as stored. The header name is the key you choose, and any authentication scheme the endpoint expects (such as Bearer or Basic) must be part of the secret value.

Endpoint expects	Header name (key)	Secret value
Bearer token	Authorization	Bearer <your token>
HTTP Basic	Authorization	Basic <base64 of username:password>
Vendor API key header	The header name the vendor specifies, for example X-API-Key	<your API key>

For example, storing only the raw token under an Authorization key sends Authorization: <your token> with no Bearer prefix, which most APIs reject with 401. If the same header name appears in both headers and auth_headers, the auth_headers value is sent.

Payload Structures

The HTTP Output supports three payload structures:

1. Single: Each record is sent as a separate HTTP request.
2. Array: Multiple records are sent as an array in a single HTTP request.
3. Wrapped: Multiple records are sent as an array within a wrapper object, using the specified wrapper_key.

Examples

Single Payload Structure

Code
{
  "id": 1,
  "name": "John Doe"
}

Array Payload Structure

Code
[
  {"id": 1, "name": "John Doe"},
  {"id": 2, "name": "Jane Smith"}
]

Wrapped Payload Structure

Code
{
  "data": [
    {"id": 1, "name": "John Doe"},
    {"id": 2, "name": "Jane Smith"}
  ]
}

Best Practices

1. Payload Structure: Choose the payload_structure that best matches the target API's expectations. Using 'array' or 'wrapped' can significantly reduce the number of HTTP requests for high-volume data.

2. Use HTTPS: Always use HTTPS for secure data transmission, especially when dealing with sensitive information.

3. TLS Verification: By default, TLS certificate verification is enabled for HTTPS connections. Only set tls_skip_verify to true if you're working with self-signed certificates or in testing environments. Disabling TLS verification in production can pose security risks.

4. Rate Limiting: Set an appropriate rate_limit to avoid overwhelming the target API. Check the API's documentation for recommended limits.

5. Batch Size: Adjust max_batch_data_size and max_batch_record_count based on your API's requirements and the size of your records.

6. Error Handling: The output will automatically retry failed requests. Monitor your logs for persistent errors.

7. Authentication: Use the auth_headers setting to include API keys, tokens, or other sensitive authentication methods required by your API. Use headers for non-sensitive headers. The secret value is sent exactly as stored, so include the scheme (for example Bearer ) in the secret when the API requires one. See Authentication headers.

User Agent

All requests sent by the HTTP Output include a custom Monad User Agent. This helps identify requests coming from your Monad pipeline to the receiving API. "Monad User Agent - Contact: security@monad.com"

Limitations

• Batch size is governed by max_batch_data_size and max_batch_record_count. However, to ensure a steady data flow, batches are automatically sent every 5 seconds, even if they haven't reached the maximum size.
• Redirects are not followed automatically. Ensure your endpoint URL is correct and does not redirect.
• All data is sent as JSON. Binary data is not supported.
• While custom headers can be set using the headers and auth_headers settings, not all headers will be honored in terms of data processing:
  • If you set the Content-Type header to a value other than application/json (e.g., application/xml), the data will still be sent as JSON.
  • Compression headers (e.g., Content-Encoding: gzip) will not result in the data being compressed. The output does not perform data compression.
• The output component prioritizes its internal data handling mechanisms over some custom header settings to ensure consistent behavior and performance.
• Success is determined by HTTP status codes in the range of 200-299. Any response with a status code outside this range is considered a failure:
  • A status code between 200 and 299 (inclusive) is treated as a success, and the batch is considered successfully sent.
  • Any status code outside this range (e.g., 3xx, 4xx, 5xx) is treated as a failure.
  • In case of a failure (status code outside 200-299), the entire batch of messages will be retried according to the Monad's retry policy.
• HTTP Request Timeout is defaulted to 30 seconds. Any request that takes longer than this will be treated as a failure.

Troubleshooting

• If you're seeing rate limit errors, try decreasing the rate_limit setting.
• For "request entity too large" errors, reduce your max_batch_data_size or max_batch_record_count.
• If requests are timing out, check the API's status and your network connection.