Copy page

Cloud Security

Google Cloud Storage

Collects and ingests data from a Google Cloud Storage bucket.

Sync Type: Incremental

Details

The Google Cloud Storage input allows you to collect and ingest data from a Google Cloud Storage bucket. You can specify which bucket to monitor and configure how data should be processed based on its format and organization.

Requirements

Before setting up the Google Cloud Storage input, you need to:

1. Have a Google Cloud Platform account with access to the desired project.
2. Create a service account with the necessary permissions.
3. Generate a JSON key for the service account.

Setup Instructions

You can set up the Google Cloud Storage input using either the Google Cloud Console UI or command-line interface.

Option 1: Using Google Cloud Console

1. Navigate to the Google Cloud Console
2. Select your project
3. Open "IAM & Admin" > "Service Accounts"
4. Create a new service account:
   • Click "Create Service Account"
   • Provide a name for the service account
   • Click "Create"
5. Assign the required role:
   • Add the "Storage Object Viewer" role
   • Also add the "Storage Legacy Bucket Reader" role on the bucket (or a custom role with storage.buckets.get); Test Connection needs it
   • Click "Continue"
   • Click "Done"
6. Generate credentials:
   • Select the newly created service account
   • Go to the "Keys" tab
   • Click "Add Key" > "Create new key"
   • Select JSON format
   • Click "Create" to download the key file
   • Store this file securely - you'll need its contents later

Option 2: Using Command Line

1. Set your project ID:

Code
export PROJECT_ID="your-project-id"
gcloud config set project $PROJECT_ID

2. Create a service account:

Code
gcloud iam service-accounts create monad-gcs-input-connector \
  --display-name="Monad GCS Input Connector" \
  --description="Service account for GCS input connector"

3. Assign the required roles:

Code
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:monad-gcs-input-connector@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/storage.objectViewer"

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:monad-gcs-input-connector@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/serviceusage.serviceUsageConsumer"

Test Connection also needs storage.buckets.get on the bucket, which Storage Object Viewer does not include. Grant roles/storage.legacyBucketReader on the bucket if the test fails with a permission error.

4. Generate and download the service account key:

Code
gcloud iam service-accounts keys create monad-gcs-key.json \
  --iam-account=monad-gcs-input-connector@$PROJECT_ID.iam.gserviceaccount.com 

This creates a monad-gcs-key.json file in your current directory. Use the contents of this file as the value for the credentials_json secret in your input configuration.

Important: Store this credentials file securely and never commit it to version control.

Bucket Structure

Your bucket should follow one of these partition formats:

1. Simple Date format (YYYY/MM/DD):

Code
bucket/
  2026/
    06/
      02/
        data.json

2. Hive format (year=YYYY/month=MM/day=DD):

Code
bucket/
  year=2026/
    month=06/
      day=02/
        data.json

3. Flat Hive Compliant format (dt=YYYY-MM-DD):

Code
bucket/
  dt=2026-06-02/
    data.json

4. Compact Date format (eventDay=YYYYMMDD):

Code
bucket/
  eventDay=20260602/
    data.json

You can optionally include a prefix for better organization:

Code
bucket/
  data/
    device-logs/
      2026/
        06/
          02/
            data.json

Details

On the first run the input starts from the current time and does not read existing objects unless Backfill Start Time is set. Set Backfill Start Time to the date of the oldest partition you want ingested. State is checkpointed only after an entire date prefix has been successfully processed, saving:

• The latest object creation time encountered
• The lexicographically greatest object name for that date

On subsequent runs, the processor performs an incremental sync starting from the last checkpointed timestamp. In the event of a failure, the processor resumes from the start of the last checkpointed date prefix. This means that if a failure occurs mid-prefix, the entire date prefix will be reprocessed from the beginning, which can represent a large number of objects or blobs.

• To avoid large-scale reprocessing on failure, we recommend publishing blob data to a queue that can be consumed from instead.

• Please also note we rescan and drop all data based on our deduplication logic on every single sync which occurs in a day prefix. This means that for larger buckets, this may lead to hitting rate limits since we will be scanning the same data a large number of times in a day. To avoid this, we recommend publishing blob data to a queue that can be consumed from to avoid such scenarios. You can also slow the polling cadence with the Cron Schedule setting and/or cap the request rate with the API Rate Limit setting (see Configuration).

• The partition layout must be one of simple date, hive compliant, flat hive compliant, or compact date. Anything else can cause unexpected behavior in the input.

• Each object's creation time should fall on the same date (UTC) as the partition it is written to. So any object that lands in the 2025/08/10 prefix should be created on 2025/08/10. Not doing so can cause unexpected behavior in the input.

Configuration

Settings

Setting	Type	Required	Default	Description
project_id	string	Yes	-	The Google Cloud project ID to use
bucket_name	string	Yes	-	The name of the Google Cloud Storage bucket to use
prefix	string	No	-	The prefix to use when reading from the bucket. This is used to filter objects in the bucket.
compression	string	Yes	-	Compression format of the objects. Allowed values: auto, gzip, none.
partition_format	string	Yes	simple date	Partition format of your GCS bucket. Options: hive compliant ('year=2024/month=01/day=01'), flat hive compliant ('dt=2024-01-01'), or simple date ('2024/01/01'). Allowed values: hive compliant, flat hive compliant, simple date, compact date.
format	string	Yes	json	File format of the Google Cloud Storage objects. Allowed values: delimited, json, jsonl, wsv, csv.
record_location	string	No	-	Location of the record in the object. Applies only for JSON objects. Leave empty for the entire record.
cron	string (cron)	No	-	Optional cron string to control how often the bucket is polled. Leave blank to keep the default continuous polling. eg: 0 * * * * - hourly, or */15 * * * * - every 15 minutes.
rate_limit	object	No	-	Maximum outbound source-API request rate. A blank or zero value falls back to the connector's default rate-limiting behavior. Maximum 1000 requests per second. See API Rate Limiting for the field format.
backfill_start_time	date	No	-	The date to start fetching data from. If not specified, no past records will be fetched.

Secrets

Secret	Type	Required	Description
credentials_json	secret	No	JSON credentials to authenticate with Google Cloud (required for service account auth).

Setting up the Input

1. In the Monad UI, go to the "Inputs" section.
2. Click "Add Input" and select "Google Cloud Storage".
3. Configure the input settings:
   • Project ID: Your Google Cloud project ID
   • Bucket Name: The name of the bucket you want to monitor
   • Prefix (optional): The path above the date partition (e.g., "data/device-logs"); the date partition is appended under it, so do not include the date in the prefix
   • Compression: Select the compression format of your files
   • Partition Format: Choose how your data is organized: simple date, hive compliant, flat hive compliant, or compact date
   • Format: Select the format of your files (e.g., "json", "csv")
   • Record Location: Specify where to find records in JSON files — see Record Location (default: "")
   • Cron Schedule (optional): A cron string controlling how often the bucket is polled (e.g. 0 * * * * for hourly). Leave blank to keep the default polling cadence.
   • API Rate Limit (optional): Cap the connector's outbound request rate to the Cloud Storage API. Leave blank to use the default. See API Rate Limiting.
4. In the "Secrets" section, provide the contents of your service account JSON key file.

Working with Prefix and Partition Format

The combination of prefix and partition_format determines how the input navigates your bucket's folder structure to find files.

Simple Date Format

The Simple Date format uses a date-based folder structure in the format YYYY/MM/DD.

• Without Prefix: Files are fetched directly from date-formatted folders

  Code
  bucket/
    2026/
      06/
        02/
          data.json

• With Prefix: Files are fetched from date-formatted folders under the specified prefix (e.g. data/device-logs)

  Code
  bucket/
    data/
      device-logs/
        2026/
          06/
            02/
              data.json

Hive Format

The Hive format uses a more explicit folder structure in the format year=YYYY/month=MM/day=DD.

• Without Prefix: Files are fetched from Hive-formatted folders

  Code
  bucket/
    year=2026/
      month=06/
        day=02/
          data.json

• With Prefix: Files are fetched from Hive-formatted folders under the specified prefix

  Code
  bucket/
    data/
      device-logs/
        year=2026/
          month=06/
            day=02/
              data.json

Troubleshooting

Common Issues

1. Access Denied: Ensure your service account has the correct IAM roles assigned.
2. No Files Found: Verify that the bucket name, prefix, and partition format match your bucket structure.
3. Invalid Credentials: Make sure the credentials_json is correct.
4. Parse Errors: Ensure the file format and record location settings match your data structure.

Related Articles

• Google Cloud Storage Documentation
• Managing Service Accounts
• Working with Cloud Storage Objects

Sync frequency

This input polls on a connector-specific interval of 10 seconds. Set the optional Cron Schedule (cron) setting to override this cadence and poll on a fixed schedule instead (e.g. 0 * * * * for hourly); leaving it blank keeps the default polling behavior. A cron schedule configured on the pipeline also overrides this cadence. See Input Sync Frequency for details.

To reduce load against the source API instead of (or in addition to) slowing the cadence, use the optional API Rate Limit (rate_limit) setting to cap the connector's outbound request rate. See API Rate Limiting.