# GenAI Traces

Ingests MLflow GenAI/LLM tracing data via OpenTelemetry. Configure MLflow's OTel exporter to push to the Monad OTLP endpoint with `OTEL_SERVICE_NAME=mlflow`.

**Sync Type: Incremental**

## Prerequisites

Before configuring this input, you need:

1. **MLflow 2.13 or later** — MLflow's built-in OpenTelemetry tracing support is required. Ensure your MLflow installation is up to date.

2. **A Monad pipeline** — Create a pipeline in Monad and note its Pipeline ID. You will configure MLflow's OTel exporter to push directly to that pipeline.

3. **A Monad organization API key** — The key must have the `pipeline:data:write` permission (included in the Contributor and System Administrator roles, or any custom role with that permission).

## How it works

This input uses the **OTLP push model** — MLflow sends trace data directly to Monad over OTLP rather than Monad polling MLflow. There are no credentials to configure on the Monad input side. Authentication is handled entirely by the headers your MLflow OTel exporter sends to the Monad OTLP endpoint.

Each span emitted by MLflow is received and forwarded to your pipeline as-is in its JSON representation.

## Authentication

All OTLP requests to Monad require two headers:

| Header | Value |
|--------|-------|
| `Authorization` | `ApiKey <your-org-api-key>` |
| `Monad-Pipeline-Id` | `<your-pipeline-id>` |

The API key must be an **organization API key** (not a personal API key) with the `pipeline:data:write` permission.

## Configuration

This input has no Settings or Secrets to configure in Monad. All configuration is done on the MLflow/OTel exporter side.

## Setup Walk-through

1. In Monad, create a new **MLflow GenAI Traces** input and attach it to a pipeline. Note the pipeline ID shown in the pipeline settings.

2. Generate an organization API key in Monad with the `pipeline:data:write` permission.

3. Set the following environment variables in the environment where MLflow is running:

```bash
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://app.monad.com:4318/v1/traces
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=ApiKey <your-org-api-key>,Monad-Pipeline-Id=<your-pipeline-id>"
export OTEL_SERVICE_NAME=mlflow
```

   If your OTel SDK uses a single endpoint variable instead of the traces-specific one, you can also use:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT=https://app.monad.com:4318
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey <your-org-api-key>,Monad-Pipeline-Id=<your-pipeline-id>"
export OTEL_SERVICE_NAME=mlflow
```

4. Run your MLflow-instrumented application. Traces will be exported automatically to Monad via OTLP/HTTP.

5. Verify data is arriving by checking the pipeline run history in Monad.

### Supported OTLP transports

| Transport | Endpoint | Port |
|-----------|----------|------|
| HTTPS/JSON | `https://app.monad.com:4318` | 4318 |
| HTTPS/Protobuf | `https://app.monad.com:4318` | 4318 |
| gRPC | `https://app.monad.com:4317` | 4317 |

For gRPC, use `monad-pipeline-id` (lowercase) as the metadata key instead of `Monad-Pipeline-Id`.

## Troubleshooting

### Common Issues

1. **No data arriving in Monad**
   - Confirm `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` (or `OTEL_EXPORTER_OTLP_ENDPOINT`) points to the correct Monad endpoint.
   - Confirm `OTEL_SERVICE_NAME=mlflow` is set — MLflow uses this to identify itself to the OTel pipeline.
   - Check that the `Authorization` and `Monad-Pipeline-Id` headers are present and correctly formatted in `OTEL_EXPORTER_OTLP_TRACES_HEADERS`.

2. **Authentication errors**
   - `authorization required` — the `Authorization` header is missing or malformed.
   - `access denied: this endpoint requires pipeline:data:write` — the API key exists but lacks the required permission. Regenerate the key with a role that includes `pipeline:data:write`.
   - `failed to get api key` — the API key has been revoked or is no longer valid.

3. **Pipeline not found**
   - `An item of this type does not exist.` — the `Monad-Pipeline-Id` value does not match any pipeline in your organization.
   - `pipeline not found or not an OTel input` — the pipeline ID is valid but the pipeline is not configured as an MLflow GenAI Traces input.

4. **MLflow version too old**
   - OTel tracing support was added in MLflow 2.13. Upgrade MLflow if traces are not being emitted.

## Related Articles

- [Monad OpenTelemetry Input](/inputs/monad/monad-otel) — the underlying OTLP endpoint; refer to this page for protocol details and response codes
- [MLflow Tracing documentation](https://mlflow.org/docs/latest/llms/tracing/index.html)
- [MLflow OpenTelemetry integration](https://mlflow.org/docs/latest/llms/tracing/index.html#opentelemetry-integration)
- [OpenTelemetry OTLP Exporter SDK configuration](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/)