Copy page

Monad

HTTP

Accepts POSTed data directly to your pipeline's ingest endpoint.

Requirements

There are no Prerequisites to start using the HTTP input. Simply create an HTTP Input and attach it to a pipeline.

Details

The HTTP input is an extremely flexible method for ingesting data into the Monad platform. By creating an Http input, users can publish any data they wish to a pipeline without needing to have a specifically implemented input to support that data. This allows data to be sent via any programming language or even via curl as a starting point.

Sending data

Every pipeline has its own ingest host:

Code
https://<pipeline-id>.data.monad.com

POST your data to the root of that host. The hostname names the pipeline, so there is no path to build and no pipeline ID to repeat anywhere else in the request. Replace <pipeline-id> with the ID of your pipeline.

The ingest host is HTTPS only. There is no plain HTTP listener and no redirect from http://, so senders must use https://.

Request body

The request body accepts:

• A single JSON object
• Multiple JSON objects in JSONL (JSON Lines) format - line-delimited JSON where each line is a separate JSON object
• A data envelope - a single JSON object with a top-level data array, expanded into one record per element of that array

Each JSON object will be sent to the stream exactly as it is provided.

A top-level JSON array - for example [{"id": 1}, {"id": 2}] - is not an accepted body shape and returns error reading request body. Send those records as JSONL, or wrap them in a data envelope.

The data envelope

A request body that is a single JSON object containing a top-level data array is expanded into one record per element of that array:

Code
{
  "data": [
    { "event_type": "authentication", "user": "admin@example.com" },
    { "event_type": "file_access", "user": "user@example.com" }
  ]
}

The request above produces two records - the same result as sending both objects as JSONL.

When using the envelope, note that:

• Each element must be a JSON object. Elements that are not objects - strings, numbers, or nested arrays - are not valid records and will not reach your pipeline.
• The key is matched case-insensitively. data, Data, and DATA are all treated as the envelope key.
• Other top-level keys are discarded. In {"data": [...], "meta": 9}, only the array elements become records; meta is dropped.
• If data is not an array, the body is an ordinary record. For example, {"data": "hello"} is ingested as a single record with a data field.
• Any single object with a top-level data array is treated as an envelope, including one you intended as a record in its own right. The envelope is only detected when the body contains exactly one JSON object, so a JSONL body of two or more objects is never expanded this way.

Bodies may be compressed. Set Content-Encoding to gzip, br, deflate, or snappy.

Pipeline readiness

A newly created or redeployed pipeline takes a short time to start up before it can accept data. During this window the pipeline reports the Initializing status and ingest requests return 503 with {"error": "pipeline is not ready"}. This is expected, and no data is lost - the endpoint refuses traffic it cannot yet route rather than dropping it.

The wait is usually a few seconds, but can be tens of seconds. Handle it in one of two ways:

• Retry with backoff. The 503 response includes a Retry-After header telling you how many seconds to wait before retrying. Keep retrying until the request succeeds.
• Poll for readiness first. Call GET /v2/{organization_id}/pipelines/{pipeline_id}?include_status=true and send data once .status.status reports Running.

See Status Indicators for the full list of pipeline statuses.

Authentication

All requests to the HTTP input endpoint require authentication using an organization API key with the pipeline:data:write permission.

API Key Requirements

The API key must meet the following requirements:

• Must be an organization API key (not a personal API key)
• Must have the pipeline:data:write permission, which is included by default in the following roles:
  • Contributor
  • System Administrator
• Alternatively, the permission can be added to a custom role

Required Headers

You must include one of the following authentication headers:

• Authorization: ApiKey <API_KEY>
• x-api-key: <API_KEY>

The Content-Type header is not required, but if provided, it must be set to application/json.

Examples

cURL Example

Code
curl --request POST \
  --url https://<pipeline-id>.data.monad.com \
  --header 'authorization: ApiKey <api-key>' \
  --data '
    {
      "event_type": "authentication",
      "user": "admin@example.com",
      "status": "success",
      "timestamp": "2024-01-15T10:30:00Z"
    }{
      "event_type": "file_access",
      "user": "user@example.com",
      "file_path": "/sensitive/data.txt",
      "timestamp": "2024-01-15T10:31:00Z"
    }'

Python Example

Code
import requests
import json

url = 'https://<pipeline-id>.data.monad.com'
headers = {
    'accept': 'application/json',
    'Content-Type': 'application/json',
    'Authorization': 'ApiKey <API_KEY>',
}

# Line-delimited JSON records
data = '''{
  "event_type": "authentication",
  "user": "admin@example.com",
  "status": "success",
  "timestamp": "2024-01-15T10:30:00Z"
}{
  "event_type": "file_access",
  "user": "user@example.com",
  "file_path": "/sensitive/data.txt",
  "timestamp": "2024-01-15T10:31:00Z"
}'''

response = requests.post(url, headers=headers, data=data)
print(json.dumps(response.json(), indent=2))

Response Examples

Success Response (200)

Code
{
  "status": "success",
  "count": 1
}

Error Responses

Response	Description
authorization required	No authentication header was provided
invalid JWT	The provided JWT token is invalid
failed to get api key	The API key is no longer valid
access denied: this endpoint requires pipeline:data:write	The API key lacks the required pipeline:data:write permission
error reading request body	The request body is not valid JSON, or is not one of the accepted body shapes described in Sending data. A top-level JSON array returns this error
request body exceeds the maximum allowed size of 8387584 bytes	The request body is larger than the 8 MiB per-request limit. Split the payload across multiple requests
host does not name a pipeline	The first label of the ingest host is not a pipeline ID
An item of this type does not exist.	The provided pipeline ID does not exist
pipeline not found or not an HTTP input	The pipeline exists but is not configured as an HTTP input
pipeline is not ready	The pipeline exists but its input is still starting up (status Initializing), which is expected briefly after it is created or redeployed. The response includes a Retry-After header; retry after that delay, or poll the pipeline status until it reports Running. See Pipeline readiness

Sync frequency

This is a push (receiver) input: Monad ingests records as the source sends them, so there is no polling interval. See Input Sync Frequency for details.