Skip to main content

Cribl HTTP Source

Sends data to a Cribl HTTP Source using standard HTTP requests.

The Cribl Output allows you to send data from your pipeline to Cribl Stream. This output is designed to work seamlessly with Cribl Stream's HTTP Source, making it easy to integrate your data with Cribl's data observability and stream processing platform. Refer to the Cribl HTTP Source documentation for guidance on how to set up your HTTP Source

Requirements​

  • Access to your Cribl workspace
  • Administrative permissions to configure data sources

Step-by-Step Instructions​

1. Locate Your Endpoint​

  1. Navigate to Workspace > Datasource
  2. Note the endpoint URL for HTTP - this is where Monad will send your data and will be required for setting up your output.

2. Configure HTTP Source​

  1. Go to Quick Connect
  2. Click Add Source
  3. In the search bar, type "HTTP"
  4. Select the HTTP source option

3. Configure HTTP Settings​

  1. Set the Port Number

    • Use the port from your datasource endpoint (default: 10080)
    • This must match the port shown in Step 1

  2. Configure Security

    • Add an Authentication Token
    • This token will be required for Monad to successfully send data
    • Keep this token secure - you'll need it for the Monad configuration

  3. Set Event API Path

    • Default path is /cribl
    • You can customize this if needed
    • Note: Whatever path you set here will be what you put in for the API Path parameter on the monad cribl output.

Next Steps​

After completing this setup, you will have:

  • A functioning Cribl HTTP input
  • All necessary information for configuring your Monad Cribl output:
    • Endpoint URL
    • Port number
    • Authentication token
    • API path

Important Notes​

  • Ensure all port numbers match between your datasource and HTTP input configuration
  • Keep your authentication token secure
  • Document your API path if you customize it from the default

Payload Structure​

The Cribl Output sends data in the Newline Delimited JSON (NDJSON) format. Each record is sent as a separate JSON object on a new line within a single HTTP request.

Best Practices​

  1. Use of HTTPS: The Cribl Output automatically uses HTTPS for secure data transmission.

  2. Authentication: If your Cribl Stream HTTP Source requires authentication, make sure to set the auth_token in the Secrets configuration.

  3. Batch Size: The output automatically handles batching with a maximum of 1000 events per batch or 10mb of data, whichever comes first.

  4. Error Handling: The output will automatically retry failed requests. Monitor your logs for persistent errors.

User Agent​

All requests sent by the Cribl Output include a custom Monad User Agent. This helps identify requests coming from your Monad pipeline to Cribl Stream.

Limitations​

  • The maximum batch size is set to 1000 events or 10mb of data.
  • The HTTP timeout is set to 10 seconds. Any request that takes longer than this will be treated as a failure.

Troubleshooting​

  • If you're seeing connection errors, verify that the ingress_address, port, and path are correctly set and that your Cribl Stream instance is accessible.
  • For authentication errors, check that the auth_token is correctly set and matches the token configured in your Cribl Stream HTTP Source.
  • If requests are timing out, check the status of your Cribl Stream instance

Configuration​

The Cribl Output is configured using the following settings:

Settings​

SettingTypeRequiredDescription
ingress_addressstringYesYour group's ingress address found in your group information panel.
portintegerYesThe port you've set your HTTP Source to listen on.
pathstringYesThe path you've set for your HTTP Source's HTTP Event API. You do not need to append _bulk to this path as Monad already does this for you.

Secrets​

SettingTypeRequiredDescription
auth_tokenstringNoThe authentication token value you've set for your HTTP Source.

This configuration sends data to a Cribl Stream instance at my-cribl-instance.cribl.cloud on port 10080, using the path /v1/events/_bulk. It includes an authentication token for secure communication.

Remember to replace YOUR_AUTH_TOKEN with your actual authentication token if required by your Cribl Stream HTTP Source.