# OTel

Sends OTLP logs, metrics, and traces to any OTLP-compatible collector or backend.

## Details

The OTel output forwards data using the [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/specs/otlp/). It speaks either **OTLP/HTTP** (with protobuf request bodies) or **OTLP/gRPC**, and supports the three OTLP signals: **logs**, **metrics**, and **traces**.

There are two modes, picked at configuration time:

### Passthrough mode

Use when records are already OTLP-shaped JSON. This is the natural pairing for an [OTel Input](../inputs/monad/otel) feeding an OTel Output: data flows in over OTLP, is converted to JSON for the pipeline, then converted back to OTLP on the way out.

Signal is auto-detected per record by inspecting the top-level JSON keys:

| Top-level key | Signal |
|---|---|
| `resourceLogs` | logs |
| `resourceMetrics` | metrics |
| `resourceSpans` | traces |
| `scopeLogs` (no `resourceLogs`) | logs (bare ResourceLogs shape) |
| `scopeMetrics` (no `resourceMetrics`) | metrics (bare ResourceMetrics shape) |
| `scopeSpans` (no `resourceSpans`) | traces (bare ResourceSpans shape) |

Both the **wrapped** form (full `ExportLogsServiceRequest`, `{"resourceLogs": [...]}`) and the **bare** form (a single ResourceLogs object, `{"resource": {...}, "scopeLogs": [...]}`) are accepted. The bare form is what you typically see if an upstream transform iterated over the `resourceLogs` array and produced one record per entry.

Records that don't match any OTLP shape, or that fail protobuf-JSON decoding, are reported as per-record failures with a descriptive error.

### Wrap as Logs mode

Use when records are arbitrary JSON (not OTLP). Each input record is wrapped as an OTLP `LogRecord` and sent on the **logs** signal. The raw JSON becomes the log body as a string. Optionally, a timestamp field and a severity field can be extracted from the JSON.

This mode is the right pick when you want to send non-OTel log data to a destination that ingests OpenTelemetry.

### Batching across signals

The output groups records by signal within a batch. If a single batch contains a mix of logs, metrics, and traces, up to three separate outbound requests are made, one per signal:

- HTTP: `POST /v1/logs`, `POST /v1/metrics`, `POST /v1/traces`
- gRPC: `LogsService.Export`, `MetricsService.Export`, `TraceService.Export`

OTLP requires this split because each signal has its own endpoint path (HTTP) or RPC service (gRPC). Within each signal, all records' ResourceX entries are concatenated into one outbound request.

### Endpoint path appending (HTTP only)

For OTLP/HTTP, the standard convention is to append `/v1/{signal}` to a single base endpoint. The output does this automatically:

- `https://collector.example.com:4318` becomes `https://collector.example.com:4318/v1/logs`, `https://collector.example.com:4318/v1/metrics`, or `https://collector.example.com:4318/v1/traces` depending on signal.

If your collector uses non-standard paths (or different hostnames per signal), use the per-signal endpoint overrides to specify the full URL for each signal.

### Transport selection

HTTP/protobuf is the most compatible default. It works through standard HTTPS infrastructure (proxies, load balancers, firewalls).

gRPC requires HTTP/2 end-to-end. If any middlebox in the path doesn't preserve HTTP/2, gRPC will fail. Use gRPC only when your collector is gRPC-capable and you control the network path.

### Compression

The default is no compression. `gzip` is supported on both transports:

- For OTLP/HTTP, gzip is part of the spec and universally supported by spec-compliant receivers.
- For OTLP/gRPC, server support for `gzip` varies. Enable it only if your collector explicitly supports compressed gRPC requests.

## Use cases

- **Analytics and fanout on OTLP data**: Pair an OTel Input with an OTel Output to run transforms, enrichments, or routing on OTLP without leaving the protocol. Attach multiple Outputs to one pipeline to tee a single OTLP stream to several providers.
- **Send to a hosted OTel backend**: Forward to Honeycomb, Grafana Cloud, Datadog, New Relic, or any other backend that accepts OTLP. Most accept both HTTP/protobuf and gRPC.
- **Send to a self-hosted OTel Collector**: Forward to a Collector you run yourself for fanout, sampling, or further processing.
- **Convert arbitrary JSON to OTLP logs**: In `wrap_logs` mode, take any JSON record (firewall logs, application logs, audit events, etc.) and ship it as OpenTelemetry log data to an OTLP-receiving backend.

## Requirements

- The endpoint must be reachable from Monad over HTTPS. Plaintext HTTP is rejected.
- For gRPC, the endpoint must include an explicit port (e.g., `https://collector.example.com:4317`). The default OTLP port for gRPC is `4317`; for HTTP/protobuf it's `4318`, but either depends on your collector's configuration.
- Loopback, private, and link-local addresses are rejected for safety.

## Configuration

### Settings

| Setting | Type | Required | Description |
|---|---|---|---|
| Mode | oneOf | Yes | `passthrough` (records are already OTLP JSON) or `wrap_logs` (wrap arbitrary JSON as OTLP log records). |
| └── (Wrap as Logs) Service Name | string | No | Value for the `service.name` resource attribute. Defaults to `unknown_service`. |
| └── (Wrap as Logs) Timestamp Field | string | No | Dot-notation JSON path to the event timestamp (e.g., `@timestamp`, `metadata.ts`). Parsed via a multi-format parser. If missing or unparseable, the current time is used. |
| └── (Wrap as Logs) Severity Field | string | No | Dot-notation JSON path to the log severity (e.g., `level`). Common strings (`debug`, `info`, `warn`, `warning`, `error`, `fatal`, `critical`, etc.) are mapped to OTLP severity numbers. Unknown text is preserved verbatim in `severity_text` with `severity_number` left unspecified. |
| Transport | oneOf | Yes | `http_protobuf` (OTLP/HTTP with protobuf request bodies) or `grpc` (OTLP/gRPC). |
| └── (HTTP/protobuf) Logs Endpoint Override | string | No | Override URL for the logs signal. Used verbatim instead of appending `/v1/logs` to the base endpoint. |
| └── (HTTP/protobuf) Metrics Endpoint Override | string | No | Override URL for the metrics signal. Used verbatim instead of appending `/v1/metrics` to the base endpoint. |
| └── (HTTP/protobuf) Traces Endpoint Override | string | No | Override URL for the traces signal. Used verbatim instead of appending `/v1/traces` to the base endpoint. |
| Endpoint | string | Yes | HTTPS endpoint of the OTLP collector (e.g., `https://otel.example.com:4318`). For HTTP, `/v1/{signal}` is appended automatically per signal unless an override is set. For gRPC, an explicit port is required. |
| Compression | enum | No | `none` (default) or `gzip`. |
| Custom Headers | array of objects | No | Custom headers sent with each request. Applied as HTTP headers for HTTP transport, or as gRPC metadata for gRPC transport. Each entry has `header_key` and `header_value`. |
| Batch Record Count | integer | No | Max input records per outbound OTLP request. Default `1000`. In passthrough mode each input record may itself contain many OTLP records, so the actual downstream record count can be larger. |
| Batch Data Size | bytes | No | Max bytes per outbound OTLP request. Default `4 MiB` (`4194304`). Kept conservatively under typical gRPC `max_recv_msg_size` (4 MiB). |
| Maximum Flush Interval | seconds | No | Max seconds to hold records before flushing a batch. Default `5`. |

### Secrets

| Setting | Type | Required | Description |
|---|---|---|---|
| Authentication Headers | object | No | Secret authentication headers (e.g., `Authorization`). Applied as HTTP headers for HTTP transport, or as gRPC metadata for gRPC transport. Each key is the header name, each value is a reference to a Monad secret. |

### API Examples

The `mode` and `transport` fields are discriminated unions. Set `type` to select the variant, then provide the matching nested object. Secrets are referenced by their ID.

**Passthrough, HTTP/protobuf**

```json
{
  "settings": {
    "mode": {
      "type": "passthrough",
      "passthrough": {}
    },
    "transport": {
      "type": "http_protobuf",
      "http_protobuf": {}
    },
    "endpoint": "https://otel.example.com:4318",
    "compression": "gzip",
    "batch_config": {
      "batch_record_count": 1000,
      "batch_data_size": 4194304,
      "publish_rate": 5
    }
  },
  "secrets": {
    "auth_headers": {
      "Authorization": { "id": "<secret-id>" }
    }
  }
}
```

**Passthrough, gRPC, with per-signal endpoint overrides not applicable**

```json
{
  "settings": {
    "mode": {
      "type": "passthrough",
      "passthrough": {}
    },
    "transport": {
      "type": "grpc",
      "grpc": {}
    },
    "endpoint": "https://collector.example.com:4317",
    "compression": "none",
    "batch_config": {
      "batch_record_count": 1000,
      "batch_data_size": 4194304,
      "publish_rate": 5
    }
  }
}
```

**Wrap as Logs, HTTP/protobuf**

```json
{
  "settings": {
    "mode": {
      "type": "wrap_logs",
      "wrap_logs": {
        "service_name": "my-service",
        "timestamp_field": "@timestamp",
        "severity_field": "level"
      }
    },
    "transport": {
      "type": "http_protobuf",
      "http_protobuf": {}
    },
    "endpoint": "https://otel.example.com:4318",
    "compression": "none",
    "batch_config": {
      "batch_record_count": 1000,
      "batch_data_size": 4194304,
      "publish_rate": 5
    }
  }
}
```

**HTTP/protobuf with a per-signal endpoint override** (useful when a backend uses non-standard paths or different hosts per signal)

```json
{
  "settings": {
    "mode": {
      "type": "passthrough",
      "passthrough": {}
    },
    "transport": {
      "type": "http_protobuf",
      "http_protobuf": {
        "logs_endpoint": "https://logs.example.com/ingest/v1/logs"
      }
    },
    "endpoint": "https://otel.example.com:4318",
    "compression": "none",
    "batch_config": {
      "batch_record_count": 1000,
      "batch_data_size": 4194304,
      "publish_rate": 5
    }
  }
}
```

## Best Practices

1. **Default to HTTP/protobuf**: It works through any standard HTTPS infrastructure. Switch to gRPC only when your collector is gRPC-capable and you control the network path end-to-end.

2. **Verify compression support**: gzip is universally supported for OTLP/HTTP. For OTLP/gRPC, confirm your collector accepts compressed gRPC requests before enabling.

3. **Configure `service_name` in wrap_logs**: Most OTLP backends use the `service.name` resource attribute to group telemetry. The default `unknown_service` works, but a meaningful name makes the data easier to query.

4. **Set timestamp and severity fields in wrap_logs only if they exist in your data**: If left unset, the output uses the current time and leaves severity unspecified. Don't configure a path that doesn't exist; it'll log a warning per batch.

5. **Use overrides when collectors use non-standard paths**: A common pattern with multi-tenant SaaS backends is a per-signal hostname or path. Use the per-signal endpoint overrides to send each signal to the right destination.

## Limitations

- **TLS only**: Plaintext HTTP is rejected. The endpoint must use `https://`.
- **Private addresses blocked**: Loopback, private, and link-local addresses are rejected at configuration time.
- **No partial-success retry across signals**: If a batch contains mixed signals and one signal's send fails, only the failing signal's records are retried; the framework will not duplicate the succeeded signals. Per-record retry is fine within a single signal.
- **Bare LogRecord not accepted in passthrough**: A naked LogRecord (no `Resource`) doesn't carry enough information to be passed through; the smallest acceptable unit is a bare ResourceLogs (or ResourceMetrics / ResourceSpans). To send arbitrary JSON as logs, use `wrap_logs` mode.
- **Mixed input forms normalize on output**: OTLP/JSON allows enum fields to be either the enum name (`"SEVERITY_NUMBER_INFO"`) or the number (`9`). The output sends the enum as a varint on the protobuf wire. The receiver decides how to re-render; receivers using `protojson` default to the enum **name** form. This is the OTLP/JSON spec preference.
- **Retry policy is short**: The HTTP transport retries each request up to 3 times with 1s and 2s waits between attempts. Beyond that, the pipeline framework retries the whole batch.