Databricks Lakehouse
Stream data from your Monad pipeline into the Databricks Lakehouse -- either staging files for Databricks Autoloader or pushing records directly through the ZeroBus streaming protocol.
Overview
The Databricks Lakehouse output supports two write modes:
- Autoloader -- Stages compressed JSONL files to a Unity Catalog Volume for Databricks Autoloader (
cloudFiles) to ingest. You can configure the Autoloader pipeline in Databricks yourself (Self Managed), or set the Ingestion Pipeline to Monad Managed to have Monad create and schedule it for you. - ZeroBus -- Sends records directly to the ZeroBus direct-write data-plane endpoint, bypassing file staging. Records land in the target Delta table with low end-to-end latency.
Both modes use OAuth M2M (service principal) authentication and validate access during connection testing.
Requirements
- Databricks Workspace with Unity Catalog enabled
- Catalog and Schema must already exist in your workspace
- Target Delta table must exist (ZeroBus mode) or an Autoloader pipeline configured (Autoloader mode -- Monad can create it for you, see Managed Autoloader pipeline). See ZeroBus table requirements for storage constraints
- Volume for staging files (Autoloader mode only) -- Monad will create it if it doesn't exist
- Service principal with OAuth M2M credentials and the required permissions
Setting Up Permissions
Autoloader mode
Autoloader only needs volume access -- table writes happen inside your Autoloader job:
Code
Managed Autoloader pipeline (Monad Managed)
When the Ingestion Pipeline is set to Monad Managed, Monad also creates a notebook, a Lakeflow (Delta Live Tables) pipeline, and a scheduled job in your workspace, and the pipeline writes a bronze table into the target schema. This goes beyond the volume-only grants above -- the service principal additionally needs:
Code
Plus, at the workspace level:
- Workspace access so the service principal can create the notebook under
/Shared/monad/pipelines/, the pipeline, and the schedule job - Serverless compute enabled in the workspace -- the managed pipeline runs on serverless
See the Databricks docs on service principals for granting workspace entitlements.
ZeroBus mode
ZeroBus writes directly to the target table, so it needs table-level privileges:
Code
Where <principal> is your service principal application ID.
Configuration
Settings
| Setting | Type | Required | Default | Description |
|---|---|---|---|---|
| Server Hostname | string | Yes | - | Databricks workspace hostname (e.g. adb-1234567890.azuredatabricks.net) |
| Write Mode | object | Yes | autoloader | How data is loaded (see Write Modes) |
| Catalog | string | Yes | - | Unity Catalog name |
| Schema | string | Yes | - | Target schema within the catalog |
Write Modes
| Mode | Description |
|---|---|
autoloader | Stages JSONL files to a Volume for Databricks Autoloader (cloudFiles) to ingest |
zerobus | Streams records directly to the target Delta table via the ZeroBus direct-write API |
Autoloader settings:
| Setting | Type | Required | Description |
|---|---|---|---|
| Volume | string | Yes | Unity Catalog Volume used for staging JSONL files |
| Ingestion Pipeline | oneof | No | Who manages the Auto Loader pipeline: Self Managed (default) or Managed (see Managed Autoloader pipeline) |
| Batch Config | object | No | Batch flush thresholds -- see Autoloader batch configuration |
Monad Managed pipeline settings:
| Setting | Type | Required | Description |
|---|---|---|---|
| Interval (minutes) | int | No | How often the pipeline runs, in minutes. Must divide evenly into an hour (5, 6, 10, 12, 15, 20, 30) or be a whole number of hours, up to 1440 (24 hours). Defaults to 60 (hourly) |
| Table Name | string | No | The bronze table the pipeline writes into (letters, numbers, and underscores only). Defaults to bronze_<volume> |
| Schema Evolution Mode | string | No | How Auto Loader handles fields that aren't in the inferred schema -- see Column mapping and schema evolution. One of addNewColumns (default), rescue, failOnNewColumns, none |
Managed Autoloader pipeline
When the Ingestion Pipeline is Monad Managed, Monad provisions the ingestion pipeline during output setup. Setup is idempotent -- the notebook is re-imported on each restart so configuration changes propagate, the existing pipeline is reused, and interval changes are applied to the existing schedule:
- A Python notebook is created at
/Shared/monad/pipelines/monad_autoloader_<catalog>_<schema>_<volume>that reads staged files from the volume via Auto Loader (cloudFiles) into the configured bronze table in the target schema, with schema inference and evolution enabled. Each row also gets an_ingested_atcolumn (ingestion time) and a_source_filecolumn (the staged file it came from). - A serverless Lakeflow (Delta Live Tables) pipeline named
monad_autoloader_<catalog>_<schema>_<volume>is created for the notebook, targeting the configured catalog and schema - A scheduled job named
monad_autoloader_<catalog>_<schema>_<volume>_scheduletriggers a pipeline run at the configured interval
The schedule runs in UTC. Intervals are in minutes, so 60/120/180/etc fire every 1/2/3/etc hours, up to a maximum of 1440 (24 hours) -- so a long interval can mean a long wait before the first records appear in the bronze table.
Monad doesn't run the pipeline when it creates it. The first run comes from the schedule job, which Monad creates enabled.
Keep the Ingestion Pipeline Self Managed if you prefer to manage your own Autoloader ingestion against the staging volume.
Monad overwrites the notebook and re-enables the schedule
Every time the output starts, including after any change to the output's configuration in Monad, Monad re-imports the notebook from its own template and sets the schedule job back to enabled. Any edit you've made to the notebook is replaced, and a paused schedule job resumes. If you've edited the notebook or paused the job, re-check both after changing the output. Table properties already applied to the bronze table are kept.
Changing the table name or Volume
The notebook, pipeline, and schedule job are named after the catalog, schema, and Volume, not the table.
- Table Name -- Monad keeps the same pipeline and Volume and points the notebook at the new table. The new table ingests every file still staged in the Volume, so expect a full backfill with roughly the time and compute cost of a full refresh. The previous table is left in place but is no longer updated.
- Volume -- Monad creates a new notebook, pipeline, and schedule job named after the new Volume. Only data staged after the change is ingested, with no backfill. Monad doesn't delete the previous notebook, pipeline, schedule job, table, or Volume, and the previous schedule job keeps its current state. Pause or delete those objects in Databricks once you no longer need them.
Column mapping and schema evolution
Column mapping. Managed bronze tables are created with Delta column mapping in name mode. Source fields whose names contain characters Delta otherwise rejects -- spaces, ,;{}(), tabs, newlines -- are ingested under their original names instead of failing the write with DELTA_INVALID_CHARACTERS_IN_COLUMN_NAMES. Without it, a single new vendor field such as new_scope(s) stops ingestion for the whole table.
Column mapping upgrades the table protocol, and that cannot be undone
Column mapping requires Delta reader version 2 and writer version 5. The pipeline applies these table properties on its next run, which upgrades existing managed tables as well as new ones. The upgrade is not reversible, and clients older than those protocol versions can no longer read the table.
Schema evolution. The Schema Evolution Mode setting maps to Auto Loader's cloudFiles.schemaEvolutionMode:
| Mode | Behavior |
|---|---|
addNewColumns | Default. New fields are added to the table as columns. Each new field restarts the stream to update the schema |
rescue | The schema stays as inferred. New fields are preserved in the _rescued_data column as JSON |
failOnNewColumns | The run fails when a new field appears, until the schema is updated |
none | New fields are ignored and dropped |
Under rescue nothing is lost, but rescued fields are not queryable columns until you extract them out of _rescued_data. It suits sources whose schemas drift often, where a stream restart per change is more disruptive than deferring the new fields. The mode applies to the pipeline's next run; changing it does not rewrite records already ingested.
Under addNewColumns, when a run first meets a field the table doesn't have yet, Databricks stops that run with a schema-change error and starts a new run automatically.
The new run adds the column under the field's original name and completes, so a single failed run at that point is expected.
In every mode except none, the bronze table also has a _rescued_data column.
If a field arrives with a different type than its column, such as a string where an array is expected, or with a name that differs from an existing column only by case, Auto Loader keeps the original value as JSON in _rescued_data along with the source file path.
ZeroBus requires:
| Setting | Type | Required | Description |
|---|---|---|---|
| Workspace ID | string | Yes | Numeric Databricks workspace ID -- used to scope the ZeroBus OAuth token and form the data-plane endpoint |
| Region | string | Yes | Workspace region (e.g. us-west-2) -- used to form the ZeroBus data-plane endpoint |
| Table Name | string | Yes | Target table name within the configured catalog and schema (alphanumeric, underscore, and hyphen only) |
| Batch Config | object | No | Batch flush thresholds -- see ZeroBus batch configuration |
The ZeroBus data-plane endpoint is constructed as https://<workspace_id>.zerobus.<region>.cloud.databricks.com.
ZeroBus table requirements
ZeroBus does not support tables created against Unity Catalog's metastore-default (managed) storage. Attempting to write to such a table fails with error code 4024.
The target table must be a managed Delta table whose catalog or schema has an explicit managed location backed by your own cloud storage (S3 / ADLS / GCS) via a Unity Catalog external location and storage credential.
Autoloader batch configuration
Batching thresholds for the Autoloader write mode. Records are flushed to the staging volume as soon as any threshold is reached.
| Setting | Default | Min | Max | Description |
|---|---|---|---|---|
record_count | 50,000 | 5,000 | 50,000 | Maximum records per batch |
data_size | 10 MB | 5 MB | 10 MB | Maximum batch size |
publish_rate | 300s | 300s | 600s | Maximum flush interval, in seconds |
ZeroBus batch configuration
ZeroBus is a low-latency streaming API, so the flush interval bottoms out at 1 second for near-real-time delivery. Record-count and data-size caps still match Autoloader because ZeroBus enforces strict per-request size limits on the data plane -- see ZeroBus limits for the upstream constraints.
| Setting | Default | Min | Max | Description |
|---|---|---|---|---|
record_count | 50,000 | 5,000 | 50,000 | Maximum records per batch |
data_size | 10 MB | 5 MB | 10 MB | Maximum batch size |
publish_rate | 10s | 1s | 300s | Maximum flush interval, in seconds |
Secrets
| Setting | Type | Required | Description |
|---|---|---|---|
| Client ID | string | Yes | OAuth M2M client ID for service principal authentication |
| Client Secret | string | Yes | OAuth M2M client secret for service principal authentication |
Generate Client ID and Client Secret (OAuth Machine-to-Machine - Service Principal)
- In the Databricks Account Console, go to User management > Service principals
- Click Add service principal and create one
- Select the service principal, go to Secrets > Generate secret
- Copy the Client ID and Client Secret
- Add the service principal to your workspace and grant it the required permissions
Use the client ID and client secret as the client_id and client_secret secrets.
Where to Find Workspace Details
For the authoritative walkthrough -- including how to derive the ZeroBus endpoint from your workspace URL -- see Databricks's guide: Get your workspace URL and ZeroBus ingest endpoint.
- Server Hostname -- The host portion of your Databricks workspace URL (e.g.
adb-1234567890.azuredatabricks.net). - Workspace ID (ZeroBus) -- The numeric ID in the workspace URL (e.g.
https://adb-<workspace_id>.<n>.azuredatabricks.net) or under Workspace settings. - Region (ZeroBus) -- The AWS region of the workspace (e.g.
us-west-2).
Troubleshooting
Connection Issues
- Server hostname: Ensure the hostname is correct and reachable (e.g.
adb-1234567890.azuredatabricks.net) - ZeroBus endpoint: Verify the Workspace ID and Region -- a mismatch produces DNS or 404 errors at
<workspace_id>.zerobus.<region>.cloud.databricks.com
Authentication Errors
- 401 Unauthorized: Check that your OAuth credentials are valid and not expired
- Token request rejected (ZeroBus): The token endpoint validates the requested
authorization_details. MissingUSE CATALOG,USE SCHEMA, orSELECT/MODIFYon the target table will cause the token request to fail - OAuth M2M: Ensure the service principal is added to the workspace
Permission Errors
- USE SCHEMA denied: Grant
USE SCHEMAon the target schema to your principal - Volume access denied (Autoloader): Grant
READ VOLUMEandWRITE VOLUMEon the volume - Table privileges denied (ZeroBus): Grant
SELECT, MODIFYon the target table
Data Loading Issues
- Autoloader not picking up files: Verify your Autoloader pipeline reads from
/Volumes/<catalog>/<schema>/<volume>/. With a Monad Managed pipeline, check the pipeline and schedule job namedmonad_autoloader_<catalog>_<schema>_<volume>in your workspace - Managed pipeline setup failures: Errors during output setup usually indicate missing workspace access or
CREATE TABLEgrants for the service principal, or serverless compute not being enabled in the workspace - ZeroBus ingest failures: The error message includes the HTTP status and Databricks response -- common causes are schema mismatches against the target Delta table or revoked table privileges
- ZeroBus error 4024: The target table is on Unity Catalog's default/managed storage, which ZeroBus rejects. Recreate the table inside a catalog or schema with an explicit managed location backed by your own S3 / ADLS / GCS storage credentials
- Large batch failures: If uploads fail with 413 errors, reduce
data_sizein batch configuration - A run fails once on a new field, then succeeds: Expected with a Monad Managed pipeline under
addNewColumns-- see Column mapping and schema evolution DELTA_INVALID_CHARACTERS_IN_COLUMN_NAMESon a managed table: Managed tables use column mapping, so field names with special characters ingest normally. A table that this error stopped before Monad added column mapping to managed tables can keep failing on normal runs. To recover it, open the pipeline, open the dropdown next to Run pipeline, and choose Run pipeline with full table refresh. The full refresh rebuilds the table from every file still staged in the Volume, so it can take significant time and compute on a large Volume. Afterwards,SHOW TBLPROPERTIES <catalog>.<schema>.<table>showsdelta.columnMapping.modeset toname
Limitations
- Catalog and schema must exist before configuring the output
- Autoloader mode: Monad stages files and, with a Monad Managed ingestion pipeline, manages the pipeline lifecycle. With a Self Managed pipeline you are responsible for configuring Autoloader ingestion in Databricks
- Monad Managed pipeline: Edits to the managed notebook don't persist, because Monad re-imports it each time the output starts
- ZeroBus mode: The target table must already exist with a schema compatible with the incoming records; Monad does not create or evolve the table
- Table names in ZeroBus mode must match
^[A-Za-z0-9_-]+$
ZeroBus service limits
ZeroBus is a Databricks-managed direct-write API with strict per-request, per-stream, and per-workspace quotas. Monad's batch caps (5,000-50,000 records, 5-10 MB per batch) are sized to stay within these limits, but you should still review the upstream constraints before scaling up:
- Maximum request size -- the data plane rejects oversized payloads with a 413 / size-limit error
- Per-stream and per-workspace throughput -- ZeroBus throttles sustained writes beyond the documented QPS / MB-per-second limits
- Schema compatibility -- the target Delta table schema must already match the incoming records; ZeroBus does not auto-evolve schemas
- Storage backing -- target tables must live under a catalog or schema with an explicit managed location, not the metastore's default storage (see ZeroBus table requirements)
For the authoritative list of quotas, supported regions, and table requirements, see the Databricks docs: ZeroBus limits.
Best Practices
- Use default batch settings -- they are optimized for bulk loading throughput
- Pre-create catalog, schema, and (ZeroBus) target table -- Monad expects these to exist
- Use dedicated service principals with only the required permissions
- Pick Autoloader when you want Databricks to own ingestion scheduling and schema evolution
- Pick ZeroBus when you want lower end-to-end latency and direct writes without staging files
Related Articles
Monad
- Databricks output -- the standard Databricks output (Copy Into / Autoloader against a SQL warehouse)
Databricks -- ZeroBus
- ZeroBus ingest overview
- Get your workspace URL and ZeroBus ingest endpoint
- ZeroBus limits and quotas