Copy page

MLflow

GenAI Traces

Ingests MLflow GenAI/LLM tracing data via OpenTelemetry. Configure MLflow's OTel exporter to push to the Monad OTLP endpoint with OTEL_SERVICE_NAME=mlflow.

Sync Type: Incremental

Prerequisites

Before configuring this input, you need:

1. MLflow 2.13 or later — MLflow's built-in OpenTelemetry tracing support is required. Ensure your MLflow installation is up to date.

2. A Monad pipeline — Create a pipeline in Monad and note its Pipeline ID. You will configure MLflow's OTel exporter to push directly to that pipeline.

3. A Monad organization API key — The key must have the pipeline:data:write permission (included in the Contributor and System Administrator roles, or any custom role with that permission).

How it works

This input uses the OTLP push model — MLflow sends trace data directly to Monad over OTLP rather than Monad polling MLflow. There are no credentials to configure on the Monad input side. Authentication is handled entirely by the headers your MLflow OTel exporter sends to the Monad OTLP endpoint.

Each span emitted by MLflow is received and forwarded to your pipeline as-is in its JSON representation.

Authentication

All OTLP requests to Monad require two headers:

Header	Value
Authorization	ApiKey <your-org-api-key>
Monad-Pipeline-Id	<your-pipeline-id>

The API key must be an organization API key (not a personal API key) with the pipeline:data:write permission.

Configuration

This input has no Settings or Secrets to configure in Monad. All configuration is done on the MLflow/OTel exporter side.

Setup Walk-through

1. In Monad, create a new MLflow GenAI Traces input and attach it to a pipeline. Note the pipeline ID shown in the pipeline settings.

2. Generate an organization API key in Monad with the pipeline:data:write permission.

3. Set the following environment variables in the environment where MLflow is running:

Code
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://app.monad.com:4318/v1/traces
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=ApiKey <your-org-api-key>,Monad-Pipeline-Id=<your-pipeline-id>"
export OTEL_SERVICE_NAME=mlflow

If your OTel SDK uses a single endpoint variable instead of the traces-specific one, you can also use:

Code
export OTEL_EXPORTER_OTLP_ENDPOINT=https://app.monad.com:4318
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey <your-org-api-key>,Monad-Pipeline-Id=<your-pipeline-id>"
export OTEL_SERVICE_NAME=mlflow

4. Run your MLflow-instrumented application. Traces will be exported automatically to Monad via OTLP/HTTP.

5. Verify data is arriving by checking the pipeline run history in Monad.

Supported OTLP transports

Transport	Endpoint	Port
HTTPS/JSON	https://app.monad.com:4318	4318
HTTPS/Protobuf	https://app.monad.com:4318	4318
gRPC	https://app.monad.com:4317	4317

For gRPC, use monad-pipeline-id (lowercase) as the metadata key instead of Monad-Pipeline-Id.

Troubleshooting

Common Issues

1. No data arriving in Monad

   • Confirm OTEL_EXPORTER_OTLP_TRACES_ENDPOINT (or OTEL_EXPORTER_OTLP_ENDPOINT) points to the correct Monad endpoint.
   • Confirm OTEL_SERVICE_NAME=mlflow is set — MLflow uses this to identify itself to the OTel pipeline.
   • Check that the Authorization and Monad-Pipeline-Id headers are present and correctly formatted in OTEL_EXPORTER_OTLP_TRACES_HEADERS.

2. Authentication errors

   • authorization required — the Authorization header is missing or malformed.
   • access denied: this endpoint requires pipeline:data:write — the API key exists but lacks the required permission. Regenerate the key with a role that includes pipeline:data:write.
   • failed to get api key — the API key has been revoked or is no longer valid.

3. Pipeline not found

   • An item of this type does not exist. — the Monad-Pipeline-Id value does not match any pipeline in your organization.
   • pipeline not found or not an OTel input — the pipeline ID is valid but the pipeline is not configured as an MLflow GenAI Traces input.

4. MLflow version too old

   • OTel tracing support was added in MLflow 2.13. Upgrade MLflow if traces are not being emitted.

Related Articles

• Monad OpenTelemetry Input — the underlying OTLP endpoint; refer to this page for protocol details and response codes
• MLflow Tracing documentation
• MLflow OpenTelemetry integration
• OpenTelemetry OTLP Exporter SDK configuration