# Databricks Lakehouse

Stream data from your Monad pipeline into the Databricks Lakehouse -- either staging files for Databricks Autoloader or pushing records directly through the ZeroBus streaming protocol.

## Overview

The Databricks Lakehouse output supports two write modes:

- **Autoloader** -- Stages compressed JSONL files to a Unity Catalog Volume for Databricks Autoloader (`cloudFiles`) to ingest. You can configure the Autoloader pipeline in Databricks yourself (**Self Managed**), or set the Ingestion Pipeline to **Monad Managed** to have Monad create and schedule it for you.
- **ZeroBus** -- Sends records directly to the ZeroBus direct-write data-plane endpoint, bypassing file staging. Records land in the target Delta table with low end-to-end latency.

Both modes use OAuth M2M (service principal) authentication and validate access during connection testing.

## Requirements

1. **Databricks Workspace** with Unity Catalog enabled
2. **Catalog and Schema** must already exist in your workspace
3. **Target Delta table** must exist (ZeroBus mode) or an Autoloader pipeline configured (Autoloader mode -- Monad can create it for you, see [Managed Autoloader pipeline](#managed-autoloader-pipeline)). See [ZeroBus table requirements](#zerobus-table-requirements) for storage constraints
4. **Volume** for staging files (Autoloader mode only) -- Monad will create it if it doesn't exist
5. **Service principal** with OAuth M2M credentials and the [required permissions](#setting-up-permissions)

### Setting Up Permissions

#### Autoloader mode

Autoloader only needs volume access -- table writes happen inside your Autoloader job:

```sql
GRANT USE CATALOG ON CATALOG <your_catalog> TO `<principal>`;
GRANT USE SCHEMA ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
GRANT READ VOLUME, WRITE VOLUME ON VOLUME <your_catalog>.<your_schema>.<your_volume> TO `<principal>`;
GRANT CREATE VOLUME ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
```

#### Managed Autoloader pipeline (Monad Managed)

When the Ingestion Pipeline is set to **Monad Managed**, Monad also creates a notebook, a Lakeflow (Delta Live Tables) pipeline, and a scheduled job in your workspace, and the pipeline writes a bronze table into the target schema. This goes beyond the volume-only grants above -- the service principal additionally needs:

```sql
GRANT CREATE TABLE ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
```

Plus, at the workspace level:

- **Workspace access** so the service principal can create the notebook under `/Shared/monad/pipelines/`, the pipeline, and the schedule job
- **Serverless compute** enabled in the workspace -- the managed pipeline runs on serverless

See the Databricks docs on [service principals](https://docs.databricks.com/aws/en/admin/users-groups/service-principals) for granting workspace entitlements.

#### ZeroBus mode

ZeroBus writes directly to the target table, so it needs table-level privileges:

```sql
GRANT USE CATALOG ON CATALOG <your_catalog> TO `<principal>`;
GRANT USE SCHEMA ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
GRANT SELECT, MODIFY ON TABLE <your_catalog>.<your_schema>.<your_table> TO `<principal>`;
```

Where `<principal>` is your **service principal application ID**.

## Configuration

### Settings

| Setting         | Type   | Required | Default   | Description                                                                  |
| --------------- | ------ | -------- | --------- | ---------------------------------------------------------------------------- |
| Server Hostname | string | Yes      | -         | Databricks workspace hostname (e.g. `adb-1234567890.azuredatabricks.net`)    |
| Write Mode      | object | Yes      | autoloader| How data is loaded (see [Write Modes](#write-modes))                         |
| Catalog         | string | Yes      | -         | Unity Catalog name                                                           |
| Schema          | string | Yes      | -         | Target schema within the catalog                                             |
| Batch Config    | object | No       | See below | Batching configuration                                                       |

#### Write Modes

| Mode         | Description                                                                                          |
| ------------ | ---------------------------------------------------------------------------------------------------- |
| `autoloader` | Stages JSONL files to a Volume for Databricks Autoloader (`cloudFiles`) to ingest                    |
| `zerobus`    | Streams records directly to the target Delta table via the ZeroBus direct-write API                  |

**Autoloader** settings:

| Setting            | Type   | Required | Description                                                                                                              |
| ------------------ | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------ |
| Volume             | string | Yes      | Unity Catalog Volume used for staging JSONL files                                                                        |
| Ingestion Pipeline | oneof  | No       | Who manages the Auto Loader pipeline: **Self Managed** (default) or **Monad Managed** (see [Managed Autoloader pipeline](#managed-autoloader-pipeline)) |

**Monad Managed** pipeline settings:

| Setting            | Type   | Required | Description                                                                                                                                                                       |
| ------------------ | ------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Interval (minutes) | int    | No       | How often the pipeline runs, in minutes. Must divide evenly into an hour (5, 6, 10, 12, 15, 20, 30) or be a whole number of hours, up to 1440 (24 hours). Defaults to 60 (hourly)  |
| Table Name         | string | No       | The bronze table the pipeline writes into (letters, numbers, and underscores only). Defaults to `bronze_<volume>`                                                                  |

#### Managed Autoloader pipeline

When the Ingestion Pipeline is **Monad Managed**, Monad provisions the ingestion pipeline during output setup. Setup is idempotent -- the notebook is re-imported on each restart so configuration changes propagate, the existing pipeline is reused, and interval changes are applied to the existing schedule:

1. A Python notebook is created at `/Shared/monad/pipelines/monad_autoloader_<catalog>_<schema>_<volume>` that reads staged files from the volume via Auto Loader (`cloudFiles`) into the configured bronze table in the target schema, with schema inference and evolution enabled
2. A serverless Lakeflow (Delta Live Tables) pipeline named `monad_autoloader_<catalog>_<schema>_<volume>` is created for the notebook, targeting the configured catalog and schema
3. A scheduled job named `monad_autoloader_<catalog>_<schema>_<volume>_schedule` triggers a pipeline run at the configured interval

Changing the table name later points the pipeline at the new table; the previous table is left in place but is no longer updated. Keep the Ingestion Pipeline **Self Managed** if you prefer to manage your own Autoloader ingestion against the staging volume.

**ZeroBus** requires:

| Setting      | Type   | Required | Description                                                                                                  |
| ------------ | ------ | -------- | ------------------------------------------------------------------------------------------------------------ |
| Workspace ID | string | Yes      | Numeric Databricks workspace ID -- used to scope the ZeroBus OAuth token and form the data-plane endpoint    |
| Region       | string | Yes      | Workspace region (e.g. `us-west-2`) -- used to form the ZeroBus data-plane endpoint                          |
| Table Name   | string | Yes      | Target table name within the configured catalog and schema (alphanumeric, underscore, and hyphen only)       |

The ZeroBus data-plane endpoint is constructed as `https://<workspace_id>.zerobus.<region>.cloud.databricks.com`.

#### ZeroBus table requirements

ZeroBus does **not** support tables created against Unity Catalog's metastore-default (managed) storage. Attempting to write to such a table fails with error code **4024**.

The target table must be a **managed Delta table** whose **catalog or schema has an explicit managed location** backed by your own cloud storage (S3 / ADLS / GCS) via a Unity Catalog **external location** and **storage credential**.

#### Batch Configuration

Batch caps are intentionally tighter than the standard Databricks Delta Table output because ZeroBus enforces strict per-request size limits on the data plane. See [ZeroBus limits](https://docs.databricks.com/aws/en/ingestion/zerobus-limits) for the upstream constraints.

| Setting        | Default | Min   | Max    | Description                         |
| -------------- | ------- | ----- | ------ | ----------------------------------- |
| `record_count` | 50,000  | 5,000 | 50,000 | Maximum records per batch           |
| `data_size`    | 10 MB   | 5 MB  | 10 MB  | Maximum batch size                  |
| `publish_rate` | 300s    | 300s  | 600s   | Maximum time before sending a batch |

### Secrets

| Setting       | Type   | Required | Description                                                  |
| ------------- | ------ | -------- | ------------------------------------------------------------ |
| Client ID     | string | Yes      | OAuth M2M client ID for service principal authentication     |
| Client Secret | string | Yes      | OAuth M2M client secret for service principal authentication |

## Generate Client ID and Client Secret (OAuth Machine-to-Machine - Service Principal)

1. In the **Databricks Account Console**, go to **User management** > **Service principals**
2. Click **Add service principal** and create one
3. Select the service principal, go to **Secrets** > **Generate secret**
4. Copy the **Client ID** and **Client Secret**
5. Add the service principal to your workspace and grant it the [required permissions](#setting-up-permissions)

Use the client ID and client secret as the `client_id` and `client_secret` secrets.

## Where to Find Workspace Details

For the authoritative walkthrough -- including how to derive the ZeroBus endpoint from your workspace URL -- see Databricks's guide: [Get your workspace URL and ZeroBus ingest endpoint](https://docs.databricks.com/aws/en/ingestion/zerobus-ingest#get-your-workspace-url-and-zerobus-ingest-endpoint).

- **Server Hostname** -- The host portion of your Databricks workspace URL (e.g. `adb-1234567890.azuredatabricks.net`).
- **Workspace ID** (ZeroBus) -- The numeric ID in the workspace URL (e.g. `https://adb-<workspace_id>.<n>.azuredatabricks.net`) or under **Workspace settings**.
- **Region** (ZeroBus) -- The AWS region of the workspace (e.g. `us-west-2`).

## Troubleshooting

### Connection Issues

- **Server hostname**: Ensure the hostname is correct and reachable (e.g. `adb-1234567890.azuredatabricks.net`)
- **ZeroBus endpoint**: Verify the Workspace ID and Region -- a mismatch produces DNS or 404 errors at `<workspace_id>.zerobus.<region>.cloud.databricks.com`

### Authentication Errors

- **401 Unauthorized**: Check that your OAuth credentials are valid and not expired
- **Token request rejected (ZeroBus)**: The token endpoint validates the requested `authorization_details`. Missing `USE CATALOG`, `USE SCHEMA`, or `SELECT`/`MODIFY` on the target table will cause the token request to fail
- **OAuth M2M**: Ensure the service principal is added to the workspace

### Permission Errors

- **USE SCHEMA denied**: Grant `USE SCHEMA` on the target schema to your principal
- **Volume access denied** (Autoloader): Grant `READ VOLUME` and `WRITE VOLUME` on the volume
- **Table privileges denied** (ZeroBus): Grant `SELECT, MODIFY` on the target table

### Data Loading Issues

- **Autoloader not picking up files**: Verify your Autoloader pipeline reads from `/Volumes/<catalog>/<schema>/<volume>/`. With a Monad Managed pipeline, check the pipeline and schedule job named `monad_autoloader_<catalog>_<schema>_<volume>` in your workspace
- **Managed pipeline setup failures**: Errors during output setup usually indicate missing workspace access or `CREATE TABLE` grants for the service principal, or serverless compute not being enabled in the workspace
- **ZeroBus ingest failures**: The error message includes the HTTP status and Databricks response -- common causes are schema mismatches against the target Delta table or revoked table privileges
- **ZeroBus error 4024**: The target table is on Unity Catalog's default/managed storage, which ZeroBus rejects. Recreate the table inside a catalog or schema with an explicit managed location backed by your own S3 / ADLS / GCS storage credentials
- **Large batch failures**: If uploads fail with 413 errors, reduce `data_size` in batch configuration

## Limitations

- Catalog and schema must exist before configuring the output
- **Autoloader mode**: Monad stages files and, with a Monad Managed ingestion pipeline, manages the pipeline lifecycle. With a Self Managed pipeline you are responsible for configuring Autoloader ingestion in Databricks
- **ZeroBus mode**: The target table must already exist with a schema compatible with the incoming records; Monad does not create or evolve the table
- Table names in ZeroBus mode must match `^[A-Za-z0-9_-]+$`

### ZeroBus service limits

ZeroBus is a Databricks-managed direct-write API with strict per-request, per-stream, and per-workspace quotas. Monad's batch caps (5,000-50,000 records, 5-10 MB per batch) are sized to stay within these limits, but you should still review the upstream constraints before scaling up:

- **Maximum request size** -- the data plane rejects oversized payloads with a 413 / size-limit error
- **Per-stream and per-workspace throughput** -- ZeroBus throttles sustained writes beyond the documented QPS / MB-per-second limits
- **Schema compatibility** -- the target Delta table schema must already match the incoming records; ZeroBus does not auto-evolve schemas
- **Storage backing** -- target tables must live under a catalog or schema with an explicit managed location, **not** the metastore's default storage (see [ZeroBus table requirements](#zerobus-table-requirements))

For the authoritative list of quotas, supported regions, and table requirements, see the Databricks docs: [ZeroBus limits](https://docs.databricks.com/aws/en/ingestion/zerobus-limits).

## Best Practices

1. **Use default batch settings** -- they are optimized for bulk loading throughput
2. **Pre-create catalog, schema, and (ZeroBus) target table** -- Monad expects these to exist
3. **Use dedicated service principals** with only the required permissions
4. **Pick Autoloader** when you want Databricks to own ingestion scheduling and schema evolution
5. **Pick ZeroBus** when you want lower end-to-end latency and direct writes without staging files

## Related Articles

### Monad

- [Databricks output](./databricks) -- the standard Databricks output (Copy Into / Autoloader against a SQL warehouse)

### Databricks -- ZeroBus

- [ZeroBus ingest overview](https://docs.databricks.com/aws/en/ingestion/zerobus-ingest)
- [Get your workspace URL and ZeroBus ingest endpoint](https://docs.databricks.com/aws/en/ingestion/zerobus-ingest#get-your-workspace-url-and-zerobus-ingest-endpoint)
- [ZeroBus limits and quotas](https://docs.databricks.com/aws/en/ingestion/zerobus-limits)

### Databricks -- Autoloader & Volumes

- [Auto Loader (`cloudFiles`) overview](https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/)
- [Unity Catalog Volumes](https://docs.databricks.com/aws/en/volumes/)

### Databricks -- Unity Catalog & Storage

- [Managed tables](https://docs.databricks.com/aws/en/tables/managed)
- [Manage external locations and storage credentials](https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-external-locations-and-credentials)
- [Create catalogs](https://docs.databricks.com/aws/en/data-governance/unity-catalog/create-catalogs)

### Databricks -- Authentication

- [OAuth machine-to-machine (M2M) authentication](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m)
- [Service principals](https://docs.databricks.com/aws/en/admin/users-groups/service-principals)