# Databricks

Stream data from your Monad pipeline into Databricks Delta Lake tables via Unity Catalog, with automatic table creation, schema inference, and gzip-compressed staging.

## Overview

The Databricks output supports three write modes:

- **Copy Into** -- Stages compressed JSONL files to a Unity Catalog Volume and uses `COPY INTO` to load them into a Delta table. Monad handles table creation, schema inference, and file cleanup automatically.
- **Autoloader** -- Stages compressed JSONL files to a Unity Catalog Volume for Databricks Autoloader (`cloudFiles`) to ingest. You configure the Autoloader job in Databricks to pick up files from the volume.
- **ZeroBus** -- Streams records directly into an existing Delta table via the Databricks ZeroBus protocol. No Volume staging and no SQL warehouse are required; the table must already exist with a compatible schema.

All three modes support OAuth M2M (service principal) authentication and validate permissions or connectivity during connection testing.

## Requirements

1. **Databricks Workspace** with Unity Catalog enabled
2. **SQL Warehouse** running and accessible (only required for `copy_into` mode)
3. **Catalog and Schema** must already exist in your workspace
4. **Volume** for staging files (required for `copy_into` and `autoloader` modes; not used in `zerobus` mode) - Monad will create it if it doesn't exist
5. **Target Delta table** must already exist with a compatible schema (only required for `zerobus` mode)
6. **Authentication credentials** (see [Authentication Methods](#authentication-methods))

### Setting Up Permissions

The required permissions depend on which write mode you use.

#### Copy Into mode

```sql
-- Replace placeholders with your actual values
GRANT USE CATALOG ON CATALOG <your_catalog> TO `<principal>`;
GRANT USE SCHEMA ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
GRANT CREATE TABLE ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
GRANT SELECT ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
GRANT MODIFY ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
GRANT READ VOLUME, WRITE VOLUME ON VOLUME <your_catalog>.<your_schema>.<your_volume> TO `<principal>`;
GRANT CREATE VOLUME ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
```

#### Autoloader mode

Autoloader only needs volume access -- table permissions are managed by your Autoloader job:

```sql
GRANT USE CATALOG ON CATALOG <your_catalog> TO `<principal>`;
GRANT USE SCHEMA ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
GRANT READ VOLUME, WRITE VOLUME ON VOLUME <your_catalog>.<your_schema>.<your_volume> TO `<principal>`;
GRANT CREATE VOLUME ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
```

#### ZeroBus mode

ZeroBus writes directly to an existing Delta table -- it needs catalog/schema usage plus `SELECT` and `MODIFY` on the target table. These privileges are encoded into the OAuth token's `authorization_details` so the service principal must hold them in advance:

```sql
GRANT USE CATALOG ON CATALOG <your_catalog> TO `<principal>`;
GRANT USE SCHEMA ON SCHEMA <your_catalog>.<your_schema> TO `<principal>`;
GRANT SELECT, MODIFY ON TABLE <your_catalog>.<your_schema>.<your_table> TO `<principal>`;
```

Where `<principal>` is: Your **service principal application ID**

Monad verifies these permissions during **Test Connection** and will report any missing grants. For ZeroBus, Test Connection issues an OAuth token request against the scoped resource -- a token failure typically indicates a missing grant.

## Configuration

### Settings

| Setting         | Type   | Required | Default   | Description                                                                   |
| --------------- | ------ | -------- | --------- | ----------------------------------------------------------------------------- |
| Server Hostname | string | Yes      | -         | The Databricks workspace hostname (e.g. `adb-1234567890.azuredatabricks.net`) |
| Write Mode      | object | Yes      | -         | How data is loaded (see [Write Modes](#write-modes))                          |
| Catalog         | string | Yes      | -         | The Unity Catalog name                                                        |
| Schema          | string | Yes      | -         | The target schema within the catalog                                          |
| Batch Config    | object | No       | See below | Batching configuration                                                        |

`Volume` is configured per write mode (under `copy_into` or `autoloader`) -- see [Write Modes](#write-modes). It is not used in `zerobus` mode.

#### Write Modes

| Mode         | Description                                                                                          |
| ------------ | ---------------------------------------------------------------------------------------------------- |
| `copy_into`  | Stages files to a Volume and uses `COPY INTO` to load data into a Delta table                        |
| `autoloader` | Stages files to a Volume for Databricks Autoloader (`cloudFiles`) to ingest                          |
| `zerobus`    | Streams records directly into an existing Delta table via the ZeroBus protocol (no Volume, no SQL warehouse) |

**Copy Into** requires three additional settings:

- **Table Name** -- the target Delta table name. If the table doesn't exist, Monad will create it automatically.
- **HTTP Path** -- the SQL warehouse HTTP path from connection details (e.g. `/sql/1.0/warehouses/abc123`).
- **Volume** -- the Unity Catalog Volume used for staging JSONL files before `COPY INTO`.

**Autoloader** requires one additional setting -- no SQL warehouse is involved:

- **Volume** -- the Unity Catalog Volume that your Autoloader (`cloudFiles`) job reads from.

**ZeroBus** requires three additional settings and writes directly into an existing table -- no Volume staging and no SQL warehouse are involved:

- **Workspace ID** -- the numeric Databricks workspace ID (e.g. `1234567890123456`). Used to scope the OAuth token and to form the ZeroBus data-plane endpoint.
- **Region** -- the workspace region (e.g. `us-west-2`). Used to form the ZeroBus data-plane endpoint.
- **Table Name** -- the target Delta table name. The table must already exist with a compatible schema; Monad will not create it in this mode.

#### Batch Configuration

Defaults are tuned for bulk loading throughput -- larger batches mean fewer load operations.

| Setting        | Default | Min    | Max     | Description                         |
| -------------- | ------- | ------ | ------- | ----------------------------------- |
| `record_count` | 50,000  | 10,000 | 100,000 | Maximum records per batch           |
| `data_size`    | 20 MB   | 10 MB  | 50 MB   | Maximum batch size                  |
| `publish_rate` | 300s    | 300s   | 600s    | Maximum time before sending a batch |

### Secrets

| Setting       | Type   | Required | Description                                                  |
| ------------- | ------ | -------- | ------------------------------------------------------------ |
| Client ID     | string | Yes      | OAuth M2M client ID for service principal authentication     |
| Client Secret | string | Yes      | OAuth M2M client secret for service principal authentication |

## Generate Client ID and Client Secret (OAuth Machine-to-Machine - Service Principal)

Recommended for production. Uses a service principal with client credentials:

1. In the **Databricks Account Console**, go to **User management** > **Service principals**
2. Click **Add service principal** and create one
3. Select the service principal, go to **Secrets** > **Generate secret**
4. Copy the **Client ID** and **Client Secret**
5. Add the service principal to your workspace and grant it the [required permissions](#setting-up-permissions)

Use the client ID and client secret as the `client_id` and `client_secret` secrets.

## Where to Find Connection Details

1. In your Databricks workspace, go to **SQL Warehouses**
2. Select your warehouse and open the **Connection details** tab
3. Copy the **Server hostname** and **HTTP path**

## Troubleshooting

### Connection Issues

- **Server hostname**: Ensure the hostname is correct and accessible (e.g. `adb-1234567890.azuredatabricks.net`)
- **HTTP path**: Verify the SQL warehouse HTTP path from the connection details tab
- **SQL warehouse**: Make sure your warehouse is running -- Monad cannot start a stopped warehouse

### Authentication Errors

- **401 Unauthorized**: Check that your OAuth credentials are valid and not expired
- **OAuth M2M**: Ensure the service principal is added to the workspace and has the correct grants

### Permission Errors

- **USE SCHEMA denied**: Grant `USE SCHEMA` on the target schema to your principal
- **Missing permissions**: Run Test Connection to see which specific permissions are missing
- **Volume access denied**: Grant `READ VOLUME` and `WRITE VOLUME` on the volume

### Data Loading Issues

- **COPY INTO failures** (copy_into mode): Check that the volume exists and is accessible
- **Schema mismatch** (copy_into mode): `mergeSchema` is enabled, so new fields are added automatically. However, incompatible type changes may cause errors
- **Large batch failures**: If uploads fail with 413 errors, reduce the `data_size` in batch configuration
- **Autoloader not picking up files**: Verify your Autoloader job is configured to read from the correct volume path (`/Volumes/<catalog>/<schema>/<volume>/`)
- **ZeroBus ingest rejected** (zerobus mode): Confirm the target table exists in the workspace, the service principal has `SELECT` and `MODIFY` on it, and the record schema matches the table columns. ZeroBus does not auto-create tables or evolve schemas.
- **ZeroBus token failures** (zerobus mode): A failed Test Connection or `failed to authenticate to ZeroBus` typically means the service principal is missing one of the privileges encoded in `authorization_details` (`USE CATALOG`, `USE SCHEMA`, or table `SELECT`/`MODIFY`) or that the workspace ID/region is wrong.

## Limitations

- Catalog and schema must exist before configuring the output
- Volume is created automatically if it doesn't exist (copy_into and autoloader modes)
- In copy_into mode, table schema is inferred from the data -- explicit schema definition is not supported, but you can pre-create the table with your desired schema
- In autoloader mode, Monad only stages files -- you are responsible for configuring the Autoloader job in Databricks
- In zerobus mode, the target table must already exist with a compatible schema -- Monad does not create or evolve the schema, and no Volume or SQL warehouse is used

## Best Practices

1. **Use default batch settings** -- they are optimized for bulk loading throughput
2. **Share volumes across connectors** -- multiple tables can safely stage files in the same volume
3. **Pre-create catalog and schema** -- Monad expects these to exist
4. **Use dedicated service principals** with only the required permissions
5. **Monitor warehouse usage** (copy_into mode) -- each `COPY INTO` consumes SQL warehouse compute
6. **Use autoloader mode** when you want Databricks to control the ingestion schedule and schema evolution
7. **Use zerobus mode** when you need low-latency streaming into a fixed-schema Delta table and want to avoid running a SQL warehouse