Copy page

Outputs

Databricks

Stream data from your Monad pipeline into Databricks Delta Lake tables via Unity Catalog, with automatic table creation, schema inference, and gzip-compressed staging.

Overview

The Databricks output supports three write modes:

• Copy Into -- Stages compressed JSONL files to a Unity Catalog Volume and uses COPY INTO to load them into a Delta table. Monad handles table creation, schema inference, and file cleanup automatically.
• Autoloader -- Stages compressed JSONL files to a Unity Catalog Volume for Databricks Autoloader (cloudFiles) to ingest. You configure the Autoloader job in Databricks to pick up files from the volume.
• ZeroBus -- Streams records directly into an existing Delta table via the Databricks ZeroBus protocol. No Volume staging and no SQL warehouse are required; the table must already exist with a compatible schema.

All three modes support OAuth M2M (service principal) authentication and validate permissions or connectivity during connection testing.

Requirements

1. Databricks Workspace with Unity Catalog enabled
2. SQL Warehouse running and accessible (only required for copy_into mode)
3. Catalog and Schema must already exist in your workspace
4. Volume for staging files (required for copy_into and autoloader modes; not used in zerobus mode) - Monad will create it if it doesn't exist
5. Target Delta table must already exist with a compatible schema (only required for zerobus mode)
6. Authentication credentials (see Authentication Methods)

Setting Up Permissions

The required permissions depend on which write mode you use.

Copy Into mode

Code
-- Replace placeholders with your actual values
GRANT USE CATALOG ON CATALOG <your_catalog> TO `<principal>`;
GRANT USE SCHEMA ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
GRANT CREATE TABLE ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
GRANT SELECT ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
GRANT MODIFY ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
GRANT READ VOLUME, WRITE VOLUME ON VOLUME <your_catalog>.<your_schema>.<your_volume> TO `<principal>`;
GRANT CREATE VOLUME ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;

Autoloader mode

Autoloader only needs volume access -- table permissions are managed by your Autoloader job:

Code
GRANT USE CATALOG ON CATALOG <your_catalog> TO `<principal>`;
GRANT USE SCHEMA ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
GRANT READ VOLUME, WRITE VOLUME ON VOLUME <your_catalog>.<your_schema>.<your_volume> TO `<principal>`;
GRANT CREATE VOLUME ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;

ZeroBus mode

ZeroBus writes directly to an existing Delta table -- it needs catalog/schema usage plus SELECT and MODIFY on the target table. These privileges are encoded into the OAuth token's authorization_details so the service principal must hold them in advance:

Code
GRANT USE CATALOG ON CATALOG <your_catalog> TO `<principal>`;
GRANT USE SCHEMA ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
GRANT SELECT, MODIFY ON TABLE <your_catalog>.<your_schema>.<your_table> TO `<principal>`;

Where <principal> is: Your service principal application ID

Monad verifies these permissions during Test Connection and will report any missing grants. For ZeroBus, Test Connection issues an OAuth token request against the scoped resource -- a token failure typically indicates a missing grant.

Configuration

Settings

Setting	Type	Required	Default	Description
Server Hostname	string	Yes	-	The Databricks workspace hostname (e.g. adb-1234567890.azuredatabricks.net)
Write Mode	object	Yes	-	How data is loaded (see Write Modes)
Catalog	string	Yes	-	The Unity Catalog name
Schema	string	Yes	-	The target schema within the catalog
Batch Config	object	No	See below	Batching configuration

Volume is configured per write mode (under copy_into or autoloader) -- see Write Modes. It is not used in zerobus mode.

Write Modes

Mode	Description
copy_into	Stages files to a Volume and uses COPY INTO to load data into a Delta table
autoloader	Stages files to a Volume for Databricks Autoloader (cloudFiles) to ingest
zerobus	Streams records directly into an existing Delta table via the ZeroBus protocol (no Volume, no SQL warehouse)

Copy Into requires three additional settings:

• Table Name -- the target Delta table name. If the table doesn't exist, Monad will create it automatically.
• HTTP Path -- the SQL warehouse HTTP path from connection details (e.g. /sql/1.0/warehouses/abc123).
• Volume -- the Unity Catalog Volume used for staging JSONL files before COPY INTO.

Autoloader requires one additional setting -- no SQL warehouse is involved:

• Volume -- the Unity Catalog Volume that your Autoloader (cloudFiles) job reads from.

ZeroBus requires three additional settings and writes directly into an existing table -- no Volume staging and no SQL warehouse are involved:

• Workspace ID -- the numeric Databricks workspace ID (e.g. 1234567890123456). Used to scope the OAuth token and to form the ZeroBus data-plane endpoint.
• Region -- the workspace region (e.g. us-west-2). Used to form the ZeroBus data-plane endpoint.
• Table Name -- the target Delta table name. The table must already exist with a compatible schema; Monad will not create it in this mode.

Batch Configuration

Defaults are tuned for bulk loading throughput -- larger batches mean fewer load operations.

Setting	Default	Min	Max	Description
record_count	50,000	10,000	100,000	Maximum records per batch
data_size	20 MB	10 MB	50 MB	Maximum batch size
publish_rate	300s	300s	600s	Maximum time before sending a batch

Secrets

Setting	Type	Required	Description
Client ID	string	Yes	OAuth M2M client ID for service principal authentication
Client Secret	string	Yes	OAuth M2M client secret for service principal authentication

Generate Client ID and Client Secret (OAuth Machine-to-Machine - Service Principal)

Recommended for production. Uses a service principal with client credentials:

1. In the Databricks Account Console, go to User management > Service principals
2. Click Add service principal and create one
3. Select the service principal, go to Secrets > Generate secret
4. Copy the Client ID and Client Secret
5. Add the service principal to your workspace and grant it the required permissions

Use the client ID and client secret as the client_id and client_secret secrets.

Where to Find Connection Details

1. In your Databricks workspace, go to SQL Warehouses
2. Select your warehouse and open the Connection details tab
3. Copy the Server hostname and HTTP path

Troubleshooting

Connection Issues

• Server hostname: Ensure the hostname is correct and accessible (e.g. adb-1234567890.azuredatabricks.net)
• HTTP path: Verify the SQL warehouse HTTP path from the connection details tab
• SQL warehouse: Make sure your warehouse is running -- Monad cannot start a stopped warehouse

Authentication Errors

• 401 Unauthorized: Check that your OAuth credentials are valid and not expired
• OAuth M2M: Ensure the service principal is added to the workspace and has the correct grants

Permission Errors

• USE SCHEMA denied: Grant USE SCHEMA on the target schema to your principal
• Missing permissions: Run Test Connection to see which specific permissions are missing
• Volume access denied: Grant READ VOLUME and WRITE VOLUME on the volume

Data Loading Issues

• COPY INTO failures (copy_into mode): Check that the volume exists and is accessible
• Schema mismatch (copy_into mode): mergeSchema is enabled, so new fields are added automatically. However, incompatible type changes may cause errors
• Large batch failures: If uploads fail with 413 errors, reduce the data_size in batch configuration
• Autoloader not picking up files: Verify your Autoloader job is configured to read from the correct volume path (/Volumes/<catalog>/<schema>/<volume>/)
• ZeroBus ingest rejected (zerobus mode): Confirm the target table exists in the workspace, the service principal has SELECT and MODIFY on it, and the record schema matches the table columns. ZeroBus does not auto-create tables or evolve schemas.
• ZeroBus token failures (zerobus mode): A failed Test Connection or failed to authenticate to ZeroBus typically means the service principal is missing one of the privileges encoded in authorization_details (USE CATALOG, USE SCHEMA, or table SELECT/MODIFY) or that the workspace ID/region is wrong.

Limitations

• Catalog and schema must exist before configuring the output
• Volume is created automatically if it doesn't exist (copy_into and autoloader modes)
• In copy_into mode, table schema is inferred from the data -- explicit schema definition is not supported, but you can pre-create the table with your desired schema
• In autoloader mode, Monad only stages files -- you are responsible for configuring the Autoloader job in Databricks
• In zerobus mode, the target table must already exist with a compatible schema -- Monad does not create or evolve the schema, and no Volume or SQL warehouse is used

Best Practices

1. Use default batch settings -- they are optimized for bulk loading throughput
2. Share volumes across connectors -- multiple tables can safely stage files in the same volume
3. Pre-create catalog and schema -- Monad expects these to exist
4. Use dedicated service principals with only the required permissions
5. Monitor warehouse usage (copy_into mode) -- each COPY INTO consumes SQL warehouse compute
6. Use autoloader mode when you want Databricks to control the ingestion schedule and schema evolution
7. Use zerobus mode when you need low-latency streaming into a fixed-schema Delta table and want to avoid running a SQL warehouse