Copy page

Outputs

Databricks Lakehouse

Stream data from your Monad pipeline into the Databricks Lakehouse -- either staging files for Databricks Autoloader or pushing records directly through the ZeroBus streaming protocol.

Overview

The Databricks Lakehouse output supports two write modes:

• Autoloader -- Stages compressed JSONL files to a Unity Catalog Volume for Databricks Autoloader (cloudFiles) to ingest. You can configure the Autoloader pipeline in Databricks yourself (Self Managed), or set the Ingestion Pipeline to Monad Managed to have Monad create and schedule it for you.
• ZeroBus -- Sends records directly to the ZeroBus direct-write data-plane endpoint, bypassing file staging. Records land in the target Delta table with low end-to-end latency.

Both modes use OAuth M2M (service principal) authentication and validate access during connection testing.

Requirements

1. Databricks Workspace with Unity Catalog enabled
2. Catalog and Schema must already exist in your workspace
3. Target Delta table must exist (ZeroBus mode) or an Autoloader pipeline configured (Autoloader mode -- Monad can create it for you, see Managed Autoloader pipeline). See ZeroBus table requirements for storage constraints
4. Volume for staging files (Autoloader mode only) -- Monad will create it if it doesn't exist
5. Service principal with OAuth M2M credentials and the required permissions

Setting Up Permissions

Autoloader mode

Autoloader only needs volume access -- table writes happen inside your Autoloader job:

Code
GRANT USE CATALOG ON CATALOG <your_catalog> TO `<principal>`;
GRANT USE SCHEMA ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
GRANT READ VOLUME, WRITE VOLUME ON VOLUME <your_catalog>.<your_schema>.<your_volume> TO `<principal>`;
GRANT CREATE VOLUME ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;

Managed Autoloader pipeline (Monad Managed)

When the Ingestion Pipeline is set to Monad Managed, Monad also creates a notebook, a Lakeflow (Delta Live Tables) pipeline, and a scheduled job in your workspace, and the pipeline writes a bronze table into the target schema. This goes beyond the volume-only grants above -- the service principal additionally needs:

Code
GRANT CREATE TABLE ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;

Plus, at the workspace level:

• Workspace access so the service principal can create the notebook under /Shared/monad/pipelines/, the pipeline, and the schedule job
• Serverless compute enabled in the workspace -- the managed pipeline runs on serverless

See the Databricks docs on service principals for granting workspace entitlements.

ZeroBus mode

ZeroBus writes directly to the target table, so it needs table-level privileges:

Code
GRANT USE CATALOG ON CATALOG <your_catalog> TO `<principal>`;
GRANT USE SCHEMA ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
GRANT SELECT, MODIFY ON TABLE <your_catalog>.<your_schema>.<your_table> TO `<principal>`;

Where <principal> is your service principal application ID.

Configuration

Settings

Setting	Type	Required	Default	Description
Server Hostname	string	Yes	-	Databricks workspace hostname (e.g. adb-1234567890.azuredatabricks.net)
Write Mode	object	Yes	autoloader	How data is loaded (see Write Modes)
Catalog	string	Yes	-	Unity Catalog name
Schema	string	Yes	-	Target schema within the catalog
Batch Config	object	No	See below	Batching configuration

Write Modes

Mode	Description
autoloader	Stages JSONL files to a Volume for Databricks Autoloader (cloudFiles) to ingest
zerobus	Streams records directly to the target Delta table via the ZeroBus direct-write API

Autoloader settings:

Setting	Type	Required	Description
Volume	string	Yes	Unity Catalog Volume used for staging JSONL files
Ingestion Pipeline	oneof	No	Who manages the Auto Loader pipeline: Self Managed (default) or Monad Managed (see Managed Autoloader pipeline)

Monad Managed pipeline settings:

Setting	Type	Required	Description
Interval (minutes)	int	No	How often the pipeline runs, in minutes. Must divide evenly into an hour (5, 6, 10, 12, 15, 20, 30) or be a whole number of hours, up to 1440 (24 hours). Defaults to 60 (hourly)
Table Name	string	No	The bronze table the pipeline writes into (letters, numbers, and underscores only). Defaults to bronze_<volume>

Managed Autoloader pipeline

When the Ingestion Pipeline is Monad Managed, Monad provisions the ingestion pipeline during output setup. Setup is idempotent -- the notebook is re-imported on each restart so configuration changes propagate, the existing pipeline is reused, and interval changes are applied to the existing schedule:

1. A Python notebook is created at /Shared/monad/pipelines/monad_autoloader_<catalog>_<schema>_<volume> that reads staged files from the volume via Auto Loader (cloudFiles) into the configured bronze table in the target schema, with schema inference and evolution enabled
2. A serverless Lakeflow (Delta Live Tables) pipeline named monad_autoloader_<catalog>_<schema>_<volume> is created for the notebook, targeting the configured catalog and schema
3. A scheduled job named monad_autoloader_<catalog>_<schema>_<volume>_schedule triggers a pipeline run at the configured interval

Changing the table name later points the pipeline at the new table; the previous table is left in place but is no longer updated. Keep the Ingestion Pipeline Self Managed if you prefer to manage your own Autoloader ingestion against the staging volume.

ZeroBus requires:

Setting	Type	Required	Description
Workspace ID	string	Yes	Numeric Databricks workspace ID -- used to scope the ZeroBus OAuth token and form the data-plane endpoint
Region	string	Yes	Workspace region (e.g. us-west-2) -- used to form the ZeroBus data-plane endpoint
Table Name	string	Yes	Target table name within the configured catalog and schema (alphanumeric, underscore, and hyphen only)

The ZeroBus data-plane endpoint is constructed as https://<workspace_id>.zerobus.<region>.cloud.databricks.com.

ZeroBus table requirements

ZeroBus does not support tables created against Unity Catalog's metastore-default (managed) storage. Attempting to write to such a table fails with error code 4024.

The target table must be a managed Delta table whose catalog or schema has an explicit managed location backed by your own cloud storage (S3 / ADLS / GCS) via a Unity Catalog external location and storage credential.

Batch Configuration

Batch caps are intentionally tighter than the standard Databricks Delta Table output because ZeroBus enforces strict per-request size limits on the data plane. See ZeroBus limits for the upstream constraints.

Setting	Default	Min	Max	Description
record_count	50,000	5,000	50,000	Maximum records per batch
data_size	10 MB	5 MB	10 MB	Maximum batch size
publish_rate	300s	300s	600s	Maximum time before sending a batch

Secrets

Setting	Type	Required	Description
Client ID	string	Yes	OAuth M2M client ID for service principal authentication
Client Secret	string	Yes	OAuth M2M client secret for service principal authentication

Generate Client ID and Client Secret (OAuth Machine-to-Machine - Service Principal)

1. In the Databricks Account Console, go to User management > Service principals
2. Click Add service principal and create one
3. Select the service principal, go to Secrets > Generate secret
4. Copy the Client ID and Client Secret
5. Add the service principal to your workspace and grant it the required permissions

Use the client ID and client secret as the client_id and client_secret secrets.

Where to Find Workspace Details

For the authoritative walkthrough -- including how to derive the ZeroBus endpoint from your workspace URL -- see Databricks's guide: Get your workspace URL and ZeroBus ingest endpoint.

• Server Hostname -- The host portion of your Databricks workspace URL (e.g. adb-1234567890.azuredatabricks.net).
• Workspace ID (ZeroBus) -- The numeric ID in the workspace URL (e.g. https://adb-<workspace_id>.<n>.azuredatabricks.net) or under Workspace settings.
• Region (ZeroBus) -- The AWS region of the workspace (e.g. us-west-2).

Troubleshooting

Connection Issues

• Server hostname: Ensure the hostname is correct and reachable (e.g. adb-1234567890.azuredatabricks.net)
• ZeroBus endpoint: Verify the Workspace ID and Region -- a mismatch produces DNS or 404 errors at <workspace_id>.zerobus.<region>.cloud.databricks.com

Authentication Errors

• 401 Unauthorized: Check that your OAuth credentials are valid and not expired
• Token request rejected (ZeroBus): The token endpoint validates the requested authorization_details. Missing USE CATALOG, USE SCHEMA, or SELECT/MODIFY on the target table will cause the token request to fail
• OAuth M2M: Ensure the service principal is added to the workspace

Permission Errors

• USE SCHEMA denied: Grant USE SCHEMA on the target schema to your principal
• Volume access denied (Autoloader): Grant READ VOLUME and WRITE VOLUME on the volume
• Table privileges denied (ZeroBus): Grant SELECT, MODIFY on the target table

Data Loading Issues

• Autoloader not picking up files: Verify your Autoloader pipeline reads from /Volumes/<catalog>/<schema>/<volume>/. With a Monad Managed pipeline, check the pipeline and schedule job named monad_autoloader_<catalog>_<schema>_<volume> in your workspace
• Managed pipeline setup failures: Errors during output setup usually indicate missing workspace access or CREATE TABLE grants for the service principal, or serverless compute not being enabled in the workspace
• ZeroBus ingest failures: The error message includes the HTTP status and Databricks response -- common causes are schema mismatches against the target Delta table or revoked table privileges
• ZeroBus error 4024: The target table is on Unity Catalog's default/managed storage, which ZeroBus rejects. Recreate the table inside a catalog or schema with an explicit managed location backed by your own S3 / ADLS / GCS storage credentials
• Large batch failures: If uploads fail with 413 errors, reduce data_size in batch configuration

Limitations

• Catalog and schema must exist before configuring the output
• Autoloader mode: Monad stages files and, with a Monad Managed ingestion pipeline, manages the pipeline lifecycle. With a Self Managed pipeline you are responsible for configuring Autoloader ingestion in Databricks
• ZeroBus mode: The target table must already exist with a schema compatible with the incoming records; Monad does not create or evolve the table
• Table names in ZeroBus mode must match ^[A-Za-z0-9_-]+$

ZeroBus service limits

ZeroBus is a Databricks-managed direct-write API with strict per-request, per-stream, and per-workspace quotas. Monad's batch caps (5,000-50,000 records, 5-10 MB per batch) are sized to stay within these limits, but you should still review the upstream constraints before scaling up:

• Maximum request size -- the data plane rejects oversized payloads with a 413 / size-limit error
• Per-stream and per-workspace throughput -- ZeroBus throttles sustained writes beyond the documented QPS / MB-per-second limits
• Schema compatibility -- the target Delta table schema must already match the incoming records; ZeroBus does not auto-evolve schemas
• Storage backing -- target tables must live under a catalog or schema with an explicit managed location, not the metastore's default storage (see ZeroBus table requirements)

For the authoritative list of quotas, supported regions, and table requirements, see the Databricks docs: ZeroBus limits.

Best Practices

1. Use default batch settings -- they are optimized for bulk loading throughput
2. Pre-create catalog, schema, and (ZeroBus) target table -- Monad expects these to exist
3. Use dedicated service principals with only the required permissions
4. Pick Autoloader when you want Databricks to own ingestion scheduling and schema evolution
5. Pick ZeroBus when you want lower end-to-end latency and direct writes without staging files

Related Articles

Monad

• Databricks output -- the standard Databricks output (Copy Into / Autoloader against a SQL warehouse)

Databricks -- ZeroBus

• ZeroBus ingest overview
• Get your workspace URL and ZeroBus ingest endpoint
• ZeroBus limits and quotas

Databricks -- Autoloader & Volumes

• Auto Loader (cloudFiles) overview
• Unity Catalog Volumes

Databricks -- Unity Catalog & Storage

• Managed tables
• Manage external locations and storage credentials
• Create catalogs

Databricks -- Authentication

• OAuth machine-to-machine (M2M) authentication
• Service principals