# Hydrolix

Streams records into Hydrolix via the HTTP streaming ingest endpoint (`/ingest/event`).

## Requirements

1. **Hydrolix cluster** reachable over HTTPS at a stable hostname (e.g. `mycluster.hydrolix.live`).
2. **Target table** already created in Hydrolix in `<project>.<table>` form. Monad does not create tables.
3. **Authentication credentials**
   - A **service-account bearer token** (recommended for production), or
   - A **Hydrolix user's username and password**.
4. *(Optional)* A **transform** defined in Hydrolix if you want the server to reshape records on ingest and you don't want Monad to perform the transformation upstream.
5. *(Optional)* A **table-scoped access token** if the target table requires additional ingest controls.

### Creating a service-account token

1. In your Hydrolix cluster's UI, open **Service Accounts** and either pick an existing one or create a new service account with permission to write to the target table.
2. Generate a bearer token for the service account (Hydrolix's `POST /config/v1/service_accounts/{uuid}/tokens/` endpoint, or the UI equivalent).
3. Copy the token value. You won't be able to see it again. Use it as the **Bearer Token** secret in Monad.

Service-account tokens are long-lived and are the recommended path for machine-to-machine ingest.

### Using username/password

If you don't have a service-account token, Monad can sign in with a Hydrolix user's credentials. Prefer service-account tokens for automation - user credentials tie ingest to an individual and are subject to the user's session policies.

## Configuration

### Settings

| Setting     | Type   | Required | Default | Description                                                                                                                                       |
| ----------- | ------ | -------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| Host        | string | Yes      | -       | Hostname of the Hydrolix cluster, no scheme. Example: `mycluster.hydrolix.live`.                                                                  |
| Table       | string | Yes      | -       | Fully-qualified target table in the form `project.table`.                                    |
| Auth        | object | Yes      | -       | Authentication method - either `service_account` or `password` (see [Authentication](#authentication)).                                            |
| Transform   | string | No       | -       | Name of a Hydrolix transform to apply to this stream.    |
| Table Token | secret | No       | -       | Optional table access token for tables that require an additional ingest control.  |

#### Authentication

Pick exactly one method under **Authentication**.

**Service Account Token**: long-lived bearer, recommended for production:

| Setting      | Type   | Required | Description                                                                                     |
| ------------ | ------ | -------- | ----------------------------------------------------------------------------------------------- |
| Bearer Token | secret | Yes      | Service-account bearer token. Sent as `Authorization: Bearer <token>` on every ingest request. |

**Username & Password**: Monad exchanges these for a 24-hour access token and keeps it refreshed:

| Setting  | Type   | Required | Description                                    |
| -------- | ------ | -------- | ---------------------------------------------- |
| Username | string | Yes      | Hydrolix user account username.                |
| Password | secret | Yes      | Hydrolix user account password.                |

## Gotchas

Things worth knowing before you configure the connector:

- **Timestamp handling is the #1 source of silent data quality issues in Hydrolix.** Ingest never fails on a bad timestamp — the record just lands in the wrong time bucket and query results become quietly wrong. Normalize timestamps to a single field name and format (ISO 8601 / RFC 3339 preferred) *upstream* of this output. Configure the corresponding timestamp column and format in your pipeline transform.
- **No deduplication.** Hydrolix does not deduplicate records at ingest or storage time. If Monad retries a batch that the server actually accepted before the network dropped, both copies will be stored. If you need exactly-once, deduplicate upstream (record ID + a merge query on read) or accept at-least-once semantics.
- **User tokens expire in 24 hours; service-account tokens default to 365 days.** Password auth silently transitions to failing ingest at the 24-hour mark. Prefer service-account tokens for anything that needs to run longer than a shift.
- **Firewall / WAF stripping `X-Hdx-*` headers.** Any proxy in front of Hydrolix that drops custom `X-Hdx-*` or `Authorization` headers produces 400/401 errors that look like config bugs but aren't. Allowlist those headers on intermediate hops.
- **Table, project, and transform must exist first.** Monad does not auto-create them, and Hydrolix returns 4xx (not a helpful onboarding hint) if any are missing.
- **CSV / Parquet payloads not supported here.** Monad's Hydrolix output only sends gzipped JSON. If you need CSV or Parquet ingest, ingest via a different path - this connector will not encode records as anything other than NDJSON.
- **Backpressure on sustained 429s.** After exhausting retries, Monad fails the batch and the pipeline re-queues it. Sustained rate-limiting means the Hydrolix ingest tier can't keep up - lower the pipeline's throughput or ask Hydrolix to raise the ingest quota, rather than tuning retries higher.
- **429 waits count against pipeline throughput.** Because retries can hold a batch for up to ~5 minutes, prolonged rate-limiting will reduce the effective publish rate you observe from Monad even though no records are lost.

## Where to Find Connection Details

- **Host**: the hostname you use to reach the Hydrolix UI or the API (without `https://`). For a hosted cluster this looks like `mycluster.hydrolix.live`.
- **Table**: in the Hydrolix UI, open the target table - the header shows `project_name.table_name`. Use that value exactly.
- **Transform** *(optional)*: in the Hydrolix UI under the target table's **Transforms**. Use the transform's `name`.

## Test Connection

**Test Connection** sends a single canned JSON record to `/ingest/event` using the configured auth, table, and any optional headers. A successful test proves:

- The host is reachable and the TLS cert is valid.
- The credentials are accepted (for password auth, that `/config/v1/login` succeeded and returned an access token).
- The service account or user has permission to write to the target table.
- Any configured `Transform` / `Table Token` values are accepted.

A test failure typically indicates a bad hostname, expired token, wrong table name, or missing write permission on the target table.

## Troubleshooting

### Connection Issues

- **DNS / TLS errors**: Verify the hostname is correct and does not include a scheme or path (e.g. `mycluster.hydrolix.live`, not `https://mycluster.hydrolix.live/`).
- **Timeouts**: Monad aborts each ingest request at 29 seconds to stay under Hydrolix's 30-second streaming cutoff. Persistent timeouts usually mean the cluster is overloaded or the batch is too large - reduce `record_count`.

### Authentication Errors

- **401 Unauthorized (service account)**: The bearer token is invalid or has been revoked. Generate a new token in Hydrolix and update the **Bearer Token** secret.
- **401 Unauthorized (password)**: The username or password is wrong, or the user's password has been rotated. Update the **Password** secret. If MFA is enforced on the account, use a service-account token instead - the streaming login endpoint doesn't support MFA.
- **`hydrolix login request failed`**: The `POST /config/v1/login` call returned a non-2xx response. Check the credentials, then check whether the user account is locked or disabled.
- **`missing auth_token.access_token`**: Hydrolix returned an unexpected login response shape. This usually indicates a cluster misconfiguration; contact your Hydrolix administrator.

### Permission Errors

- **403 Forbidden**: The service account or user cannot write to `<project>.<table>`. Grant ingest permission on the target table.
- **`X-Hdx-Token` required**: The target table requires a table-scoped ingest token. Set the **Table Token** setting.

### Data Loading Issues

- **400 Bad Request with schema errors**: A record's shape does not match the table (or the configured transform). Fix the upstream shape, or configure a transform in Hydrolix that accepts the incoming shape and set it in **Transform**.
- **429 Too Many Requests**: Monad automatically retries with exponential backoff (1s, 2s, 4s, ...) for up to 8 retries and honors the server's `Retry-After` header.

## Limitations

- **Table must exist**: Monad does not create Hydrolix tables or transforms - both must be defined in Hydrolix beforehand.
- **1 MiB uncompressed batch cap**: Enforced by Hydrolix, not Monad. Larger batches will be rejected with 413.
- **29-second request timeout**: Monad aborts requests just before Hydrolix's 30-second server cutoff. Slow single requests will fail — reduce batch size rather than raise this limit.
- **MFA-enabled users**: Password authentication cannot complete an MFA challenge. Use a service-account token for any MFA-protected account.

## Best Practices

1. **Prefer service-account tokens** - long-lived, scoped to a machine identity, and unaffected by user password rotation or MFA policies.
2. **Set an explicit `Transform`** if the incoming record shape doesn't match the table's default transform, so schema drift is caught at ingest instead of silently dropping fields.
4. **Rotate service-account tokens periodically** in Hydrolix and update the secret in Monad in the same window to minimize the exposure of a leaked token.
5. **Use a dedicated service account per pipeline** so that revoking a compromised token doesn't take down other ingest flows.