# Google Cloud Storage via Pub/Sub Enables ingestion of Google Cloud Storage objects via Cloud Storage `OBJECT_FINALIZE` notifications delivered to a Pub/Sub subscription. **Sync Type: Streaming** ## Details The Google Cloud Storage via Pub/Sub connector is the Google Cloud equivalent of the AWS [SQS S3](../aws/aws-sqs-s3) and [Azure Blob Storage Queue](../microsoft/azure-blob-storage-queue) inputs. Instead of polling a bucket and listing its objects (as the [Google Cloud Storage](./google-cloud-storage) input does), it consumes a Pub/Sub subscription fed by the bucket's notifications: whenever an object is created in your bucket, Cloud Storage publishes an `OBJECT_FINALIZE` notification to a Pub/Sub topic, and this connector reads it from a subscription on that topic. The connector then downloads the referenced object, processes it according to your configured format and compression settings, and extracts records for your pipeline. This notification-driven approach is the pattern Google recommends for higher-volume ingestion, since it avoids repeatedly listing bucket contents. Records are passed through as they appear in your files; no schema mapping is applied. ## How It Works 1. **Object Notification**: When an object (or a new version of an object) is written to your bucket, Cloud Storage publishes an `OBJECT_FINALIZE` notification to your Pub/Sub topic. 2. **Message Delivery**: The topic delivers the notification to your subscription, where the connector receives it. 3. **Event Filtering**: Only `OBJECT_FINALIZE` notifications are acted on. Other events (`OBJECT_DELETE`, `OBJECT_ARCHIVE`, `OBJECT_METADATA_UPDATE`, `OBJECT_INITIALIZE`) are acknowledged and skipped. 4. **Object Download**: The connector reads the bucket, object name, and generation (version) from the notification's attributes and downloads **that exact version** of the object. 5. **Data Processing**: Downloaded objects are decompressed (if needed) and parsed according to your format settings. 6. **Record Extraction**: Individual records are extracted and sent to your data pipeline. 7. **Acknowledgment**: The notification is acknowledged only after the whole object has been processed. On a failure that may succeed later, the notification is returned to Pub/Sub, and **the subscription's retry policy decides when it is redelivered** (see [Retries and Failures](#retries-and-failures)). ## Prerequisites Before setting up the Google Cloud Storage via Pub/Sub input, ensure you have: 1. A Google Cloud project with the **Cloud Storage** and **Cloud Pub/Sub** APIs enabled. 2. A Cloud Storage bucket that receives your source data. 3. A Pub/Sub topic and a **pull** subscription on it, with its retry policy set to **Retry after exponential backoff delay** (required; see [Retries and Failures](#retries-and-failures)). 4. A Cloud Storage notification configuration that publishes the bucket's notifications to the topic. 5. A service account for Monad with the permissions below, and a JSON key for it. ## Required Permissions Two identities are involved: - **Monad's service account** (the one whose JSON key you provide) needs: - **Pub/Sub Subscriber** (`roles/pubsub.subscriber`) on the subscription, to receive and acknowledge notifications. - **Storage Object Viewer** (`roles/storage.objectViewer`) on the bucket, to download objects. If notifications from several buckets reach the subscription, grant this on each bucket. - **The Cloud Storage service agent** (Google-managed, `service-PROJECT_NUMBER@gs-project-accounts.iam.gserviceaccount.com`) needs **Pub/Sub Publisher** (`roles/pubsub.publisher`) on the topic, so the bucket can publish notifications. This is a Google-managed account; you cannot create a key for it, and Monad does not use it. ## Setup Instructions ### Step 1: Create a Bucket (if needed) 1. Open the [Google Cloud Console](https://console.cloud.google.com/) and select your project. 2. Go to **Cloud Storage** → **Buckets** → **Create**. 3. Enter a name (e.g., `security-logs`), choose a location and storage class, and create the bucket. ### Step 2: Create a Topic and a Subscription 1. Go to **Pub/Sub** → **Topics** → **Create topic**. 2. Enter a **Topic ID** (e.g., `gcs-notifications`), clear **Add a default subscription**, and click **Create**. 3. Go to **Pub/Sub** → **Subscriptions** → **Create subscription**: - **Subscription ID**: e.g., `gcs-notifications-sub`. - **Select a Cloud Pub/Sub topic**: the topic you just created. - **Delivery type**: **Pull**. - **Retry policy** (**required**): select **Retry after exponential backoff delay**. The defaults (10 seconds minimum, 600 seconds maximum) work well. - **Dead lettering** (recommended): enable it and choose a dead-letter topic, so notifications that keep failing are kept for inspection instead of expiring. 4. Click **Create**. :::warning[Set the retry policy] The connector does not delay retries itself; it relies on the subscription's retry policy. The default policy, **Retry immediately**, makes a failing object retry in a tight loop. See [Retries and Failures](#retries-and-failures) for the consequences and how to change the policy on an existing subscription. ::: ### Step 3: Let Cloud Storage Publish to the Topic 1. Find the Cloud Storage service agent's email: go to **Cloud Storage** → **Settings** → **Project Access** and copy the **Cloud Storage Service Account** (or run `gcloud storage service-agent --project=PROJECT_ID`). 2. Go to **Pub/Sub** → **Topics**, select the checkbox next to your topic, and in the **Permissions** panel click **Add principal**. 3. Enter the service agent's email, select the **Pub/Sub Publisher** role, and click **Save**. ### Step 4: Create the Bucket Notification Configuration Notification configurations cannot be managed from the Google Cloud Console. Create one from [Cloud Shell](https://cloud.google.com/shell) (the terminal icon in the Console) or any machine with the `gcloud` CLI: ```bash gcloud storage buckets notifications create gs://BUCKET_NAME \ --topic=projects/PROJECT_ID/topics/TOPIC_ID \ --payload-format=json \ --event-types=OBJECT_FINALIZE ``` - `--event-types=OBJECT_FINALIZE` is optional but recommended: the connector only acts on `OBJECT_FINALIZE` and skips everything else. - `--payload-format=json` (`JSON_API_V1`) and `--payload-format=none` are both supported; the connector reads everything it needs from the message attributes. - Create **one** notification configuration per bucket and topic. Each configuration publishes its own message, so two configurations on the same bucket and topic would ingest every object twice. - Notifications can take up to 30 seconds to start after the configuration is created, and only objects written afterwards produce notifications. Verify it with `gcloud storage buckets notifications list gs://BUCKET_NAME`. ### Step 5: Create a Service Account for Monad 1. Go to **IAM & Admin** → **Service Accounts** → **Create service account**. 2. Enter a name (e.g., `monad-gcs-pubsub`), click **Create and continue**, skip the project-level roles, and click **Done**. 3. Grant the subscription role: go to **Pub/Sub** → **Subscriptions**, select your subscription, and in the **Permissions** panel click **Add principal**. Enter the service account's email, select **Pub/Sub Subscriber**, and click **Save**. 4. Grant the bucket role: go to **Cloud Storage** → **Buckets** → your bucket → **Permissions** → **Grant access**. Enter the service account's email, select **Storage Object Viewer**, and click **Save**. 5. Create a key: open the service account → **Keys** → **Add key** → **Create new key** → **JSON** → **Create**. Store the downloaded file securely. IAM changes can take a few minutes to take effect. ## Configuration ### Settings | Setting | Type | Required | Description | | ------- | ---- | -------- | ----------- | | Project ID | string | Yes | The GCP project ID that contains the Pub/Sub subscription. | | Subscription | string | Yes | The Pub/Sub subscription that receives the bucket's notifications. Accepts a subscription ID (e.g., `gcs-notifications-sub`), which is looked up in the project set in Project ID, or a full `projects/PROJECT_ID/subscriptions/SUBSCRIPTION_ID` path. With a full path, the project in the path is used and Project ID is ignored. | | Format | string | Yes | File format of the referenced objects. Supported values: `json`, `jsonl`, `csv`, `delimited`, `wsv` (default: `json`). | | Compression | string | Yes | Compression format of the referenced objects. Supported values: `auto`, `gzip`, `none`. `auto` detects gzip from the file contents. | | Record Location | string | No | Location of the record in the JSON object. JSON only; leave empty for the entire record. See [Record Location](../../guides/record-location) for syntax and examples. | | Key Filter | object | No | Optional filter applied to the object name from each notification, before fetch. See [Key Filter](#key-filter) below. | | With Metadata | boolean | No | When enabled, annotates each record with its source bucket, object name, and generation under `_monad_metadata`. Default: `false`. | ### Secrets | Secret | Type | Required | Description | | ------ | ---- | -------- | ----------- | | Service Account Credentials | string | Yes | The contents of the service account's JSON key file (raw JSON or base64-encoded). Only service account keys (`"type": "service_account"`) are accepted. | ### Key Filter Filters the object name (from the notification) before the connector fetches it. Useful when the subscription receives notifications for a wider set of objects than this connector should process. Matching is case-insensitive on both the object name and the configured value. Key Filter is a discriminated union on the `type` field — the empty case is `{"type": "none"}` rather than an empty `{mode, operator, value}` object, so the per-field rules only apply when filtering is actually requested: | Field | Type | Required | Description | | ----- | ---- | -------- | ----------- | | `type` | string | Yes | `none` passes every object through; `filter` enables the rules in `filter` below. | | `filter` | object | When `type` is `filter` | The match parameters described below. Omit when `type` is `none`. | When `type` is `filter`, the nested `filter` object takes: | Field | Type | Required | Description | | ----- | ---- | -------- | ----------- | | `mode` | string | Yes | `include` processes only matching objects; `exclude` skips matching objects. | | `operator` | string | Yes | One of `contains`, `ends_with`, `starts_with`. | | `value` | string | Yes | The substring (or suffix/prefix) to match against the object name. | Examples: - No filtering — `{ "type": "none" }` - Process only gzipped JSON files — `{ "type": "filter", "filter": { "mode": "include", "operator": "ends_with", "value": ".json.gz" } }` - Skip a noisy prefix — `{ "type": "filter", "filter": { "mode": "exclude", "operator": "starts_with", "value": "internal/" } }` When the field is omitted entirely (or set to `{"type": "none"}`), every object referenced by the subscription is processed. Objects skipped by the filter are acknowledged, not retried. ## Example Notification Message Each notification carries the object reference in its **attributes**, which are present for both payload formats: ```json { "notificationConfig": "projects/_/buckets/security-logs/notificationConfigs/1", "eventType": "OBJECT_FINALIZE", "payloadFormat": "JSON_API_V1", "bucketId": "security-logs", "objectId": "app/2026/10/29/events.jsonl.gz", "objectGeneration": "1761730000123456", "eventTime": "2026-10-29T09:00:01.123456Z" } ``` With `JSON_API_V1`, the message body also contains the object's metadata (`name`, `bucket`, `generation`, `size`, `contentType`, `timeCreated`, …); with `NONE`, the body is empty. The connector uses only the attributes. The record timestamp is the notification's `eventTime`. ## Example Record With **With Metadata** enabled, each record from the file is emitted with the source object appended: ```json { "event_id": "e-1001", "timestamp": "2026-10-29T09:00:01Z", "actor": "alice@example.com", "action": "login", "result": "success", "_monad_metadata": { "bucket": "security-logs", "object": "app/2026/10/29/events.jsonl.gz", "generation": 1761730000123456 } } ``` ## Retries and Failures | Outcome | What the connector does | | ------- | ----------------------- | | Object processed | Acknowledges the notification. | | Not an `OBJECT_FINALIZE` event, or skipped by the key filter | Acknowledges the notification. | | Malformed notification (missing bucket, object, or generation), or a message that is not a Cloud Storage notification | Logs a warning and acknowledges it; retrying could never succeed. | | The object version no longer exists (overwritten or deleted before it was processed) | Logs a warning and acknowledges it. A newer version has its own notification. | | A `json` or `jsonl` object contains invalid records (malformed JSON, or values that are not JSON objects such as `42` or `[1, 2]`) | Skips the invalid records, ingests every valid record in the object, then acknowledges the notification. Logs a warning with the object, the number of records skipped, and the first line numbers (`jsonl`) or array positions (`json`) of the skipped records. Retrying could not fix the skipped records, so the object is not retried. | | Download, permission, or other parse error (for example, a Record Location that does not exist in the object, or a malformed `csv` row) | Logs a warning and returns the notification to Pub/Sub. **When it is redelivered is decided entirely by the subscription's retry policy**; the connector adds no delay of its own. | :::warning[A retry policy is required] The connector does not delay retries itself. When processing an object fails, the notification is returned to Pub/Sub straight away, and the subscription's **retry policy** controls when Pub/Sub delivers it again. Pub/Sub's default policy is **Retry immediately**. If you keep it, every failing notification is redelivered within milliseconds and retried in a tight loop until the cause is fixed. For example, if the service account loses read access to a bucket, every notification from that bucket is retried continuously, which: - downloads, or tries to download, the same objects over and over, adding Cloud Storage requests and cost; - writes a warning to the pipeline logs on every attempt, burying other messages; - uses up a dead-letter topic's delivery attempts within seconds, so notifications are dead-lettered almost immediately instead of being retried over time; - competes with healthy notifications, including those from other buckets on the same subscription, for the connector's capacity. Set the subscription's retry policy to **Retry after exponential backoff delay** before starting the pipeline. The connector cannot read or change the policy, so the connection test cannot warn you if it is missing. ::: To set the retry policy on an existing subscription, go to **Pub/Sub** → **Subscriptions**, open the subscription, click **Edit**, set **Retry policy** to **Retry after exponential backoff delay**, and click **Update**. Or run: ```bash gcloud pubsub subscriptions update SUBSCRIPTION_ID \ --min-retry-delay=10s \ --max-retry-delay=600s ``` We also recommend **dead lettering**: a dead-letter topic (with its own subscription) so notifications that keep failing are kept for inspection. Without one, an unacknowledged notification is retried until the subscription's message retention period (7 days by default) expires, then dropped. If an object fails partway through, the records already emitted from it are emitted again when it is retried. ## Important Notes - **Event Filtering**: Only `OBJECT_FINALIZE` notifications are processed. Copying, rewriting, or restoring an object also produces `OBJECT_FINALIZE`, so those objects are ingested again. - **Exact Versions**: The connector downloads the object generation named in the notification, so an object overwritten shortly after upload is never read as its newer contents, and each version is read once per notification. - **Gzip and `Content-Encoding`**: Objects are downloaded as stored, without Cloud Storage's automatic decompression for objects with `Content-Encoding: gzip`. Set **Compression** to `auto` (or `gzip`) for gzipped objects. - **At-least-once Delivery**: Cloud Storage and Pub/Sub both deliver at least once, and notifications are not ordered. The same object can occasionally be ingested twice. - **Multiple Buckets**: The bucket is read from each notification, so a single connector serves every bucket whose notifications reach the subscription, as long as the service account has **Storage Object Viewer** on each. - **Test Connection**: The connection test checks that the credentials are valid and the service account can consume from the subscription, without receiving any messages. A subscription that does not exist and a missing **Pub/Sub Subscriber** role report the same error. Bucket access cannot be tested in advance, because buckets are only known from notifications; missing bucket access shows up as retried download errors. :::warning[One subscription per pipeline] Do not point two pipelines at the same subscription, whether they share this input or each have their own input configured with the same subscription. Pub/Sub delivers each message on a subscription to only one of its readers, so each pipeline silently receives only part of the data. To send this data to several destinations, add more outputs to a single pipeline, or create one subscription per pipeline on the same topic; every subscription receives every message. See [Using One Input in Multiple Pipelines](../../guides/shared-inputs.mdx#inputs-that-read-from-a-queue). ::: ## Related Articles - [Pub/Sub notifications for Cloud Storage](https://cloud.google.com/storage/docs/pubsub-notifications) - [Configure Pub/Sub notifications for Cloud Storage](https://cloud.google.com/storage/docs/reporting-changes) - [Subscription retry policy](https://cloud.google.com/pubsub/docs/subscription-retry-policy) - [Dead-letter topics](https://cloud.google.com/pubsub/docs/handling-failures) - [Pub/Sub access control](https://cloud.google.com/pubsub/docs/access-control) - [Decompressive transcoding](https://cloud.google.com/storage/docs/transcoding) ## Sync frequency This input consumes from the source queue or subscription, ingesting records at whatever rate the source delivers them rather than on a fixed poll. See [Input Sync Frequency](../../guides/sync-frequency) for details.