# Schema Detection

:::info
Schema Detection requires the feature flag to be enabled. Please contact support@monad.com to have it turned on for your account.
:::

This feature is in beta. Please contact support@monad.com for access. Schema Detection lets Monad learn the shape of data flowing through a pipeline edge and alert you when that shape changes: new fields appear, fields change type, or existing structure is altered. It runs automatically in the background once enabled; no transforms or extra configuration are required.

## Simple mode

The everyday path: enable it, let it learn, and get alerted when your data changes shape.

### How it works

Schema Detection runs in two sequential modes on each monitored edge:

- **Learning mode**: When first enabled, the edge observes records passing through and builds a model of the schema: which fields exist, what types they carry, and how often they appear. It graduates to detecting mode automatically once it has seen enough data.
- **Detecting mode**: Monad continuously compares incoming records against the learned schema. When a record introduces a new field or changes the type of an existing field, a [Schema Drift Detection alert](/docs/alerts/schema-drift-detection) fires. For how to triage and respond, see [Handling drift alerts](#handling-drift-alerts) below.

![Learning mode](/img/schema-drift-detection/learning-mode.png)

![Detection mode](/img/schema-drift-detection/detecting-mode-edge.png)

### Enabling schema detection

Schema Detection is configured per edge in the pipeline editor.

1. Open your pipeline and click an edge to open the edge detail panel.
2. In the **Schema detection** section, toggle the switch on.

![Enabling](/img/schema-drift-detection/enabling.png)

3. Save the pipeline. The edge very soon enters learning mode.

> Schema Detection can be toggled independently on any edge in a pipeline. Enabling it on one edge does not affect others.

### Viewing the schema

Once data starts flowing, you can view the current schema and actions you can perform in the schema viewer by clicking on the edge.

---

## Advanced mode

Deeper detail on when learning graduates, how field types are represented, and how to control the process manually.

### Graduation thresholds

Learning mode graduates automatically to detecting mode once either:

- **1,000 records** have been observed and **48 hours** have elapsed since learning started, or
- **500,000 records** have been observed (volume ceiling), or
- **4 days** have elapsed regardless of record count (time ceiling)

You can also graduate manually from the schema viewer at any time (see [Graduating manually](#graduating-manually) below).

### Field types in the current schema

Each field in the current schema shows its observed types as badges:

| Badge | Meaning |
|---|---|
| `string` | String values observed |
| `number` | Numeric values observed |
| `boolean` | Boolean values observed |
| `null` | Null values observed |
| `array of objects` | Array of JSON objects |
| `array of strings` | Array of string values |

A field can carry multiple type badges if mixed types were observed in different records.

### Graduating manually

If you have collected enough data and want to start detecting changes instead of waiting for Monad to fully learn your schema:

1. Open the schema viewer for the edge.
2. Click **Actions → Start detecting**.
3. Confirm in the dialog.

The edge immediately switches to detecting mode. Any new fields or type changes from this point forward will fire a [Schema Drift Detection alert](/docs/alerts/schema-drift-detection).

### Resetting the schema

Resetting clears all learned state and restarts the learning period from zero.

1. Open the schema viewer for the edge.
2. Click **Actions → Reset schema**.
3. Confirm in the dialog, this action cannot be undone.

> Any active schema drift alerts for this edge stop firing after a reset.

### Limitations

- Schema Detection supports up to **500 fields** per edge. If the cap is exceeded, learning stops and a field cap event is recorded.

---

## Handling drift alerts

When an edge is in detecting mode, Monad fires a [Schema Drift Detection alert](/docs/alerts/schema-drift-detection) whenever a record deviates from the learned schema. There is no accept, reject, or acknowledge action on an individual alert, handling drift means deciding whether the change was intentional, then acting on the **edge** (rebaseline with Reset schema) or the **pipeline** (add a transform, update downstream mappings).

Monad reports two event types:

| Event type | Severity | Meaning |
|---|---|---|
| `new_field` | `low` | A field appeared that wasn't in the learned schema |
| `type_change` | `medium` | A field carried a type not observed during learning |

### Triage

Every alert comes down to one question: **did you expect this change?**

- **Intentional**: a planned upstream change. Adopt the new shape by rebaselining the edge (see [Resetting the schema](#resetting-the-schema)).
- **Unexpected**: investigate first (upstream misconfiguration, an unannounced vendor change, or a data-integrity issue), then remediate before rebaselining.

### Common scenarios

- **A new field appeared (`new_field`).** If you want it downstream, rebaseline with **Reset schema** and confirm downstream components handle it. If you don't want it, drop it with a [`drop_key`](/docs/transforms/drop_key) transform. If it's unexpected, investigate before rebaselining.
- **A field was renamed.** Renames have no dedicated event — Monad fires a `new_field` alert for the new name while the old field goes stale (check its **last seen** time in the schema viewer). Downstream mappings referencing the old name keep running but deliver empty values, so renames are the most likely drift type to cause silent data loss. Either update downstream components to the new name and rebaseline, or add a [`rename_key`](/docs/transforms/rename_key) transform mapping the new name back to the old one to preserve existing contracts.
- **A field changed type (`type_change`).** Higher severity because it can break typed destinations like databases and warehouses. Cast it back with a [`mutate_type`](/docs/transforms/mutate_type) transform, fix the source if the new type is an error, or rebaseline if the change is permanent and intended.

### Rebaseline vs. mask

You can mask almost any drift with a transform before the edge, but rebaselining is the healthier default: masking hides future drift, accumulates standing pipeline logic to maintain, and lets the learned schema drift from reality. Mask only when a downstream destination has a contract you can't change yet — a fixed table schema, a partner integration, a locked-down SIEM index. Treat it as temporary and rebaseline once the destination can accept the new shape.

### Route alerts to your team

Like all Monad alerts, drift events can be routed to Slack, PagerDuty, a SIEM, or any destination through a pipeline using the [Monad Alerts input](/docs/components/inputs/monad/monad-alerts), so a human reviews the change before it's absorbed. Routing doesn't resolve drift on its own — you still take one of the actions above each time an alert fires.

## Related

- [Monad Alerts input](/docs/components/inputs/monad/monad-alerts) — route drift alerts to any destination
- Transforms: [`rename_key`](/docs/transforms/rename_key), [`mutate_type`](/docs/transforms/mutate_type), [`drop_key`](/docs/transforms/drop_key), [`jq`](/docs/transforms/jq)