Copy page

Outputs

Databricks

Stream data from your Monad pipeline into Databricks Delta Lake tables via Unity Catalog, with automatic table creation, schema inference, and gzip-compressed staging.

Overview

The Databricks output supports two write modes:

• Copy Into -- Stages compressed JSONL files to a Unity Catalog Volume and uses COPY INTO to load them into a Delta table. Monad handles table creation, schema inference, and file cleanup automatically.
• Autoloader -- Stages compressed JSONL files to a Unity Catalog Volume for Databricks Autoloader (cloudFiles) to ingest. You configure the Autoloader job in Databricks to pick up files from the volume.

Both modes support OAuth M2M (service principal) authentication and validate permissions during connection testing.

Requirements

1. Databricks Workspace with Unity Catalog enabled
2. SQL Warehouse running and accessible (only required for copy_into mode)
3. Catalog and Schema must already exist in your workspace
4. Volume for staging files - Monad will create it if it doesn't exist
5. Authentication credentials (see Authentication Methods)

Setting Up Permissions

The required permissions depend on which write mode you use.

Copy Into mode

Code
-- Replace placeholders with your actual values
GRANT USE CATALOG ON CATALOG <your_catalog> TO `<principal>`;
GRANT USE SCHEMA ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
GRANT CREATE TABLE ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
GRANT SELECT ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
GRANT MODIFY ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
GRANT READ VOLUME, WRITE VOLUME ON VOLUME <your_catalog>.<your_schema>.<your_volume> TO `<principal>`;
GRANT CREATE VOLUME ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;

Autoloader mode

Autoloader only needs volume access -- table permissions are managed by your Autoloader job:

Code
GRANT USE CATALOG ON CATALOG <your_catalog> TO `<principal>`;
GRANT USE SCHEMA ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
GRANT READ VOLUME, WRITE VOLUME ON VOLUME <your_catalog>.<your_schema>.<your_volume> TO `<principal>`;
GRANT CREATE VOLUME ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;

Where <principal> is: Your service principal application ID

Monad verifies all of these permissions during Test Connection and will report any missing grants.

Configuration

Settings

Setting	Type	Required	Default	Description
Server Hostname	string	Yes	-	The Databricks workspace hostname (e.g. adb-1234567890.azuredatabricks.net)
Write Mode	object	Yes	-	How data is loaded (see Write Modes)
Catalog	string	Yes	-	The Unity Catalog name
Schema	string	Yes	-	The target schema within the catalog
Volume	string	Yes	-	The Unity Catalog Volume used for staging JSONL files
Batch Config	object	No	See below	Batching configuration

Write Modes

Mode	Description
copy_into	Stages files to a Volume and uses COPY INTO to load data into a Delta table
autoloader	Stages files to a Volume for Databricks Autoloader (cloudFiles) to ingest

Copy Into requires two additional settings:

• Table Name -- the target Delta table name. If the table doesn't exist, Monad will create it automatically.
• HTTP Path -- the SQL warehouse HTTP path from connection details (e.g. /sql/1.0/warehouses/abc123).

Autoloader has no additional settings. Files are uploaded to the volume and left for your Autoloader job to pick up -- no SQL warehouse is required.

Batch Configuration

Defaults are tuned for bulk loading throughput -- larger batches mean fewer load operations.

Setting	Default	Min	Max	Description
record_count	50,000	10,000	100,000	Maximum records per batch
data_size	20 MB	10 MB	50 MB	Maximum batch size
publish_rate	300s	300s	600s	Maximum time before sending a batch

Secrets

Setting	Type	Required	Description
Client ID	string	Yes	OAuth M2M client ID for service principal authentication
Client Secret	string	Yes	OAuth M2M client secret for service principal authentication

Generate Client ID and Client Secret (OAuth Machine-to-Machine - Service Principal)

Recommended for production. Uses a service principal with client credentials:

1. In the Databricks Account Console, go to User management > Service principals
2. Click Add service principal and create one
3. Select the service principal, go to Secrets > Generate secret
4. Copy the Client ID and Client Secret
5. Add the service principal to your workspace and grant it the required permissions

Use the client ID and client secret as the client_id and client_secret secrets.

Where to Find Connection Details

1. In your Databricks workspace, go to SQL Warehouses
2. Select your warehouse and open the Connection details tab
3. Copy the Server hostname and HTTP path

Troubleshooting

Connection Issues

• Server hostname: Ensure the hostname is correct and accessible (e.g. adb-1234567890.azuredatabricks.net)
• HTTP path: Verify the SQL warehouse HTTP path from the connection details tab
• SQL warehouse: Make sure your warehouse is running -- Monad cannot start a stopped warehouse

Authentication Errors

• 401 Unauthorized: Check that your OAuth credentials are valid and not expired
• OAuth M2M: Ensure the service principal is added to the workspace and has the correct grants

Permission Errors

• USE SCHEMA denied: Grant USE SCHEMA on the target schema to your principal
• Missing permissions: Run Test Connection to see which specific permissions are missing
• Volume access denied: Grant READ VOLUME and WRITE VOLUME on the volume

Data Loading Issues

• COPY INTO failures (copy_into mode): Check that the volume exists and is accessible
• Schema mismatch (copy_into mode): mergeSchema is enabled, so new fields are added automatically. However, incompatible type changes may cause errors
• Large batch failures: If uploads fail with 413 errors, reduce the data_size in batch configuration
• Autoloader not picking up files: Verify your Autoloader job is configured to read from the correct volume path (/Volumes/<catalog>/<schema>/<volume>/)

Limitations

• Catalog and schema must exist before configuring the output
• Volume is created automatically if it doesn't exist
• In copy_into mode, table schema is inferred from the data -- explicit schema definition is not supported, but you can pre-create the table with your desired schema
• In autoloader mode, Monad only stages files -- you are responsible for configuring the Autoloader job in Databricks

Best Practices

1. Use default batch settings -- they are optimized for bulk loading throughput
2. Share volumes across connectors -- multiple tables can safely stage files in the same volume
3. Pre-create catalog and schema -- Monad expects these to exist
4. Use dedicated service principals with only the required permissions
5. Monitor warehouse usage (copy_into mode) -- each COPY INTO consumes SQL warehouse compute
6. Use autoloader mode when you want Databricks to control the ingestion schedule and schema evolution