Google Cloud Storage via Pub/Sub
Enables ingestion of Google Cloud Storage objects via Cloud Storage OBJECT_FINALIZE notifications delivered to a Pub/Sub subscription.
Sync Type: Streaming
Details
The Google Cloud Storage via Pub/Sub connector is the Google Cloud equivalent of the AWS SQS S3 and Azure Blob Storage Queue inputs. Instead of polling a bucket and listing its objects (as the Google Cloud Storage input does), it consumes a Pub/Sub subscription fed by the bucket's notifications: whenever an object is created in your bucket, Cloud Storage publishes an OBJECT_FINALIZE notification to a Pub/Sub topic, and this connector reads it from a subscription on that topic. The connector then downloads the referenced object, processes it according to your configured format and compression settings, and extracts records for your pipeline.
This notification-driven approach is the pattern Google recommends for higher-volume ingestion, since it avoids repeatedly listing bucket contents.
Records are passed through as they appear in your files; no schema mapping is applied.
How It Works
- Object Notification: When an object (or a new version of an object) is written to your bucket, Cloud Storage publishes an
OBJECT_FINALIZEnotification to your Pub/Sub topic. - Message Delivery: The topic delivers the notification to your subscription, where the connector receives it.
- Event Filtering: Only
OBJECT_FINALIZEnotifications are acted on. Other events (OBJECT_DELETE,OBJECT_ARCHIVE,OBJECT_METADATA_UPDATE,OBJECT_INITIALIZE) are acknowledged and skipped. - Object Download: The connector reads the bucket, object name, and generation (version) from the notification's attributes and downloads that exact version of the object.
- Data Processing: Downloaded objects are decompressed (if needed) and parsed according to your format settings.
- Record Extraction: Individual records are extracted and sent to your data pipeline.
- Acknowledgment: The notification is acknowledged only after the whole object has been processed. On a failure that may succeed later, the notification is returned to Pub/Sub, and the subscription's retry policy decides when it is redelivered (see Retries and Failures).
Prerequisites
Before setting up the Google Cloud Storage via Pub/Sub input, ensure you have:
- A Google Cloud project with the Cloud Storage and Cloud Pub/Sub APIs enabled.
- A Cloud Storage bucket that receives your source data.
- A Pub/Sub topic and a pull subscription on it, with its retry policy set to Retry after exponential backoff delay (required; see Retries and Failures).
- A Cloud Storage notification configuration that publishes the bucket's notifications to the topic.
- A service account for Monad with the permissions below, and a JSON key for it.
Required Permissions
Two identities are involved:
- Monad's service account (the one whose JSON key you provide) needs:
- Pub/Sub Subscriber (
roles/pubsub.subscriber) on the subscription, to receive and acknowledge notifications. - Storage Object Viewer (
roles/storage.objectViewer) on the bucket, to download objects. If notifications from several buckets reach the subscription, grant this on each bucket.
- Pub/Sub Subscriber (
- The Cloud Storage service agent (Google-managed,
service-PROJECT_NUMBER@gs-project-accounts.iam.gserviceaccount.com) needs Pub/Sub Publisher (roles/pubsub.publisher) on the topic, so the bucket can publish notifications. This is a Google-managed account; you cannot create a key for it, and Monad does not use it.
Setup Instructions
Step 1: Create a Bucket (if needed)
- Open the Google Cloud Console and select your project.
- Go to Cloud Storage → Buckets → Create.
- Enter a name (e.g.,
security-logs), choose a location and storage class, and create the bucket.
Step 2: Create a Topic and a Subscription
- Go to Pub/Sub → Topics → Create topic.
- Enter a Topic ID (e.g.,
gcs-notifications), clear Add a default subscription, and click Create. - Go to Pub/Sub → Subscriptions → Create subscription:
- Subscription ID: e.g.,
gcs-notifications-sub. - Select a Cloud Pub/Sub topic: the topic you just created.
- Delivery type: Pull.
- Retry policy (required): select Retry after exponential backoff delay. The defaults (10 seconds minimum, 600 seconds maximum) work well.
- Dead lettering (recommended): enable it and choose a dead-letter topic, so notifications that keep failing are kept for inspection instead of expiring.
- Subscription ID: e.g.,
- Click Create.
Set the retry policy
The connector does not delay retries itself; it relies on the subscription's retry policy. The default policy, Retry immediately, makes a failing object retry in a tight loop. See Retries and Failures for the consequences and how to change the policy on an existing subscription.
Step 3: Let Cloud Storage Publish to the Topic
- Find the Cloud Storage service agent's email: go to Cloud Storage → Settings → Project Access and copy the Cloud Storage Service Account (or run
gcloud storage service-agent --project=PROJECT_ID). - Go to Pub/Sub → Topics, select the checkbox next to your topic, and in the Permissions panel click Add principal.
- Enter the service agent's email, select the Pub/Sub Publisher role, and click Save.
Step 4: Create the Bucket Notification Configuration
Notification configurations cannot be managed from the Google Cloud Console. Create one from Cloud Shell (the terminal icon in the Console) or any machine with the gcloud CLI:
Code
--event-types=OBJECT_FINALIZEis optional but recommended: the connector only acts onOBJECT_FINALIZEand skips everything else.--payload-format=json(JSON_API_V1) and--payload-format=noneare both supported; the connector reads everything it needs from the message attributes.- Create one notification configuration per bucket and topic. Each configuration publishes its own message, so two configurations on the same bucket and topic would ingest every object twice.
- Notifications can take up to 30 seconds to start after the configuration is created, and only objects written afterwards produce notifications.
Verify it with gcloud storage buckets notifications list gs://BUCKET_NAME.
Step 5: Create a Service Account for Monad
- Go to IAM & Admin → Service Accounts → Create service account.
- Enter a name (e.g.,
monad-gcs-pubsub), click Create and continue, skip the project-level roles, and click Done. - Grant the subscription role: go to Pub/Sub → Subscriptions, select your subscription, and in the Permissions panel click Add principal. Enter the service account's email, select Pub/Sub Subscriber, and click Save.
- Grant the bucket role: go to Cloud Storage → Buckets → your bucket → Permissions → Grant access. Enter the service account's email, select Storage Object Viewer, and click Save.
- Create a key: open the service account → Keys → Add key → Create new key → JSON → Create. Store the downloaded file securely.
IAM changes can take a few minutes to take effect.
Configuration
Settings
| Setting | Type | Required | Description |
|---|---|---|---|
| Project ID | string | Yes | The GCP project ID that contains the Pub/Sub subscription. |
| Subscription | string | Yes | The Pub/Sub subscription that receives the bucket's notifications. Accepts a subscription ID (e.g., gcs-notifications-sub), which is looked up in the project set in Project ID, or a full projects/PROJECT_ID/subscriptions/SUBSCRIPTION_ID path. With a full path, the project in the path is used and Project ID is ignored. |
| Format | string | Yes | File format of the referenced objects. Supported values: json, jsonl, csv, delimited, wsv (default: json). |
| Compression | string | Yes | Compression format of the referenced objects. Supported values: auto, gzip, none. auto detects gzip from the file contents. |
| Record Location | string | No | Location of the record in the JSON object. JSON only; leave empty for the entire record. See Record Location for syntax and examples. |
| Key Filter | object | No | Optional filter applied to the object name from each notification, before fetch. See Key Filter below. |
| With Metadata | boolean | No | When enabled, annotates each record with its source bucket, object name, and generation under _monad_metadata. Default: false. |
Secrets
| Secret | Type | Required | Description |
|---|---|---|---|
| Service Account Credentials | string | Yes | The contents of the service account's JSON key file (raw JSON or base64-encoded). Only service account keys ("type": "service_account") are accepted. |
Key Filter
Filters the object name (from the notification) before the connector fetches it. Useful when the subscription receives notifications for a wider set of objects than this connector should process. Matching is case-insensitive on both the object name and the configured value.
Key Filter is a discriminated union on the type field — the empty case is {"type": "none"} rather than an empty {mode, operator, value} object, so the per-field rules only apply when filtering is actually requested:
| Field | Type | Required | Description |
|---|---|---|---|
type | string | Yes | none passes every object through; filter enables the rules in filter below. |
filter | object | When type is filter | The match parameters described below. Omit when type is none. |
When type is filter, the nested filter object takes:
| Field | Type | Required | Description |
|---|---|---|---|
mode | string | Yes | include processes only matching objects; exclude skips matching objects. |
operator | string | Yes | One of contains, ends_with, starts_with. |
value | string | Yes | The substring (or suffix/prefix) to match against the object name. |
Examples:
- No filtering —
{ "type": "none" } - Process only gzipped JSON files —
{ "type": "filter", "filter": { "mode": "include", "operator": "ends_with", "value": ".json.gz" } } - Skip a noisy prefix —
{ "type": "filter", "filter": { "mode": "exclude", "operator": "starts_with", "value": "internal/" } }
When the field is omitted entirely (or set to {"type": "none"}), every object referenced by the subscription is processed. Objects skipped by the filter are acknowledged, not retried.
Example Notification Message
Each notification carries the object reference in its attributes, which are present for both payload formats:
Code
With JSON_API_V1, the message body also contains the object's metadata (name, bucket, generation, size, contentType, timeCreated, …); with NONE, the body is empty. The connector uses only the attributes. The record timestamp is the notification's eventTime.
Example Record
With With Metadata enabled, each record from the file is emitted with the source object appended:
Code
Retries and Failures
| Outcome | What the connector does |
|---|---|
| Object processed | Acknowledges the notification. |
Not an OBJECT_FINALIZE event, or skipped by the key filter | Acknowledges the notification. |
| Malformed notification (missing bucket, object, or generation), or a message that is not a Cloud Storage notification | Logs a warning and acknowledges it; retrying could never succeed. |
| The object version no longer exists (overwritten or deleted before it was processed) | Logs a warning and acknowledges it. A newer version has its own notification. |
A json or jsonl object contains invalid records (malformed JSON, or values that are not JSON objects such as 42 or [1, 2]) | Skips the invalid records, ingests every valid record in the object, then acknowledges the notification. Logs a warning with the object, the number of records skipped, and the first line numbers (jsonl) or array positions (json) of the skipped records. Retrying could not fix the skipped records, so the object is not retried. |
Download, permission, or other parse error (for example, a Record Location that does not exist in the object, or a malformed csv row) | Logs a warning and returns the notification to Pub/Sub. When it is redelivered is decided entirely by the subscription's retry policy; the connector adds no delay of its own. |
A retry policy is required
The connector does not delay retries itself. When processing an object fails, the notification is returned to Pub/Sub straight away, and the subscription's retry policy controls when Pub/Sub delivers it again.
Pub/Sub's default policy is Retry immediately. If you keep it, every failing notification is redelivered within milliseconds and retried in a tight loop until the cause is fixed. For example, if the service account loses read access to a bucket, every notification from that bucket is retried continuously, which:
- downloads, or tries to download, the same objects over and over, adding Cloud Storage requests and cost;
- writes a warning to the pipeline logs on every attempt, burying other messages;
- uses up a dead-letter topic's delivery attempts within seconds, so notifications are dead-lettered almost immediately instead of being retried over time;
- competes with healthy notifications, including those from other buckets on the same subscription, for the connector's capacity.
Set the subscription's retry policy to Retry after exponential backoff delay before starting the pipeline. The connector cannot read or change the policy, so the connection test cannot warn you if it is missing.
To set the retry policy on an existing subscription, go to Pub/Sub → Subscriptions, open the subscription, click Edit, set Retry policy to Retry after exponential backoff delay, and click Update. Or run:
Code
We also recommend dead lettering: a dead-letter topic (with its own subscription) so notifications that keep failing are kept for inspection. Without one, an unacknowledged notification is retried until the subscription's message retention period (7 days by default) expires, then dropped.
If an object fails partway through, the records already emitted from it are emitted again when it is retried.
Important Notes
- Event Filtering: Only
OBJECT_FINALIZEnotifications are processed. Copying, rewriting, or restoring an object also producesOBJECT_FINALIZE, so those objects are ingested again. - Exact Versions: The connector downloads the object generation named in the notification, so an object overwritten shortly after upload is never read as its newer contents, and each version is read once per notification.
- Gzip and
Content-Encoding: Objects are downloaded as stored, without Cloud Storage's automatic decompression for objects withContent-Encoding: gzip. Set Compression toauto(orgzip) for gzipped objects. - At-least-once Delivery: Cloud Storage and Pub/Sub both deliver at least once, and notifications are not ordered. The same object can occasionally be ingested twice.
- Multiple Buckets: The bucket is read from each notification, so a single connector serves every bucket whose notifications reach the subscription, as long as the service account has Storage Object Viewer on each.
- Test Connection: The connection test checks that the credentials are valid and the service account can consume from the subscription, without receiving any messages. A subscription that does not exist and a missing Pub/Sub Subscriber role report the same error. Bucket access cannot be tested in advance, because buckets are only known from notifications; missing bucket access shows up as retried download errors.
One subscription per pipeline
Do not point two pipelines at the same subscription, whether they share this input or each have their own input configured with the same subscription. Pub/Sub delivers each message on a subscription to only one of its readers, so each pipeline silently receives only part of the data. To send this data to several destinations, add more outputs to a single pipeline, or create one subscription per pipeline on the same topic; every subscription receives every message. See Using One Input in Multiple Pipelines.
Related Articles
- Pub/Sub notifications for Cloud Storage
- Configure Pub/Sub notifications for Cloud Storage
- Subscription retry policy
- Dead-letter topics
- Pub/Sub access control
- Decompressive transcoding
Sync frequency
This input consumes from the source queue or subscription, ingesting records at whatever rate the source delivers them rather than on a fixed poll. See Input Sync Frequency for details.