Copy page

Google

Cloud Storage

Collects and ingests data from a Google Cloud Storage bucket.

Sync Type: Incremental

Details

The Google Cloud Storage input allows you to collect and ingest data from a Google Cloud Storage bucket. You can specify which bucket to monitor and configure how data should be processed based on its format and organization.

Requirements

Before setting up the Google Cloud Storage input, you need to:

1. Have a Google Cloud Platform account with access to the desired project.
2. Create a service account with the necessary permissions.
3. Generate a JSON key for the service account.

Setup Instructions

You can set up the Google Cloud Storage input using either the Google Cloud Console UI or command-line interface.

Option 1: Using Google Cloud Console

1. Navigate to the Google Cloud Console
2. Select your project
3. Open "IAM & Admin" > "Service Accounts"
4. Create a new service account:
   • Click "Create Service Account"
   • Provide a name for the service account
   • Click "Create"
5. Assign the required role:
   • Add the "Storage Object Viewer" role
   • Click "Continue"
   • Click "Done"
6. Generate credentials:
   • Select the newly created service account
   • Go to the "Keys" tab
   • Click "Add Key" > "Create new key"
   • Select JSON format
   • Click "Create" to download the key file
   • Store this file securely - you'll need its contents later

Option 2: Using Command Line

1. Set your project ID:

Code
export PROJECT_ID="your-project-id"
gcloud config set project $PROJECT_ID

2. Create a service account:

Code
gcloud iam service-accounts create monad-gcs-input-connector \
  --display-name="Monad GCS Input Connector" \
  --description="Service account for GCS input connector"

3. Assign the required roles:

Code
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:monad-gcs-input-connector@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/storage.objectViewer"

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:monad-gcs-input-connector@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/serviceusage.serviceUsageConsumer"

4. Generate and download the service account key:

Code
gcloud iam service-accounts keys create monad-gcs-key.json \
  --iam-account=monad-gcs-input-connector@$PROJECT_ID.iam.gserviceaccount.com 

This creates a monad-gcs-key.json file in your current directory. Use the contents of this file as the value for the credentials_json secret in your input configuration.

Important: Store this credentials file securely and never commit it to version control.

Bucket Structure

Your bucket should follow one of these partition formats:

1. Simple Date format (YYYY/MM/DD):

Code
bucket/
  2026/
    06/
      02/
        data.json

2. Hive format (year=YYYY/month=MM/day=DD):

Code
bucket/
  year=2026/
    month=06/
      day=02/
        data.json

3. Flat Hive Compliant format (dt=YYYY-MM-DD):

Code
bucket/
  dt=2026-06-02/
    data.json

You can optionally include a prefix for better organization:

Code
bucket/
  data/
    device-logs/
      2026/
        06/
          02/
            data.json

Details

When the input is run for the first time, it performs a full sync of all files in the specified bucket-prefix. State is checkpointed only after an entire date prefix has been successfully processed, saving:

• The highest LastModified timestamp encountered
• The lexicographically greatest Blob key at that timestamp

On subsequent runs, the processor performs an incremental sync starting from the last checkpointed timestamp. In the event of a failure, the processor resumes from the start of the last checkpointed date prefix. This means that if a failure occurs mid-prefix, the entire date prefix will be reprocessed from the beginning, which can represent a large number of objects or blobs.

• To avoid large-scale reprocessing on failure, we recommend publishing blob data to a queue that can be consumed from instead.

• Please also note we rescan and drop all data based on our deduplication logic on every single sync which occurs in a day prefix. This means that for larger buckets, this may lead to hitting rate limits since we will be scanning the same data a large number of times in a day. To avoid this, we recommend publishing blob data to a queue that can be consumed from to avoid such scenarios. You can also slow the polling cadence with the Cron Schedule setting and/or cap the request rate with the API Rate Limit setting (see Configuration).

• Prefixes must be hive compliant/flat hive compliant/simple date always. Anything other than this can cause unexpected behavior in the input.

• Each log's last updated time should be on the same date as the logical prefix itself. So any object that lands in the 2025/08/10 prefix should have a last updated time of 2025/08/10 (in its ISO8601 format). Not doing so can cause unexpected behavior in the input.

Configuration

Settings

Setting	Type	Required	Default	Description
project_id	string	Yes	-	The Google Cloud project ID to use
bucket_name	string	Yes	-	The name of the Google Cloud Storage bucket to use
compression	string	Yes	-	Compression format of the objects (e.g., "gzip", "none")
partition_format	string	Yes	"Simple Date"	Specifies how data is organized in the bucket. Options: hive compliant, flat hive compliant, or simple date.
format	string	Yes	"json"	The format of the files in the bucket (e.g., "json", "csv")
prefix	string	No	-	The prefix to filter objects within the bucket
record_location	string	No	""	Location of the record in the JSON object. See Record Location for syntax and examples.
cron	string	No	-	Optional cron schedule controlling how often the bucket is polled. Leave blank to keep the default polling cadence (see Sync frequency below). e.g., 0 * * * * for hourly.
rate_limit	object	No	-	Optional limit on the connector's outbound request rate to the Google Cloud Storage API. Leave blank to use the connector's default behavior. See API Rate Limiting for the field format, limits, and how to choose a value.

Secrets

Setting	Type	Required	Description
credentials_json	string	Yes	Service account JSON key file contents as a string

Setting up the Input

1. In the Monad UI, go to the "Inputs" section.
2. Click "Add Input" and select "Google Cloud Storage".
3. Configure the input settings:
   • Project ID: Your Google Cloud project ID
   • Bucket Name: The name of the bucket you want to monitor
   • Prefix (optional): Filter objects in the bucket by prefix (e.g., "data/2023/")
   • Compression: Select the compression format of your files
   • Partition Format: Choose how your data is organized (e.g., "Simple Date" or "Hive")
   • Format: Select the format of your files (e.g., "json", "csv")
   • Record Location: Specify where to find records in JSON files — see Record Location (default: "")
   • Cron Schedule (optional): A cron string controlling how often the bucket is polled (e.g. 0 * * * * for hourly). Leave blank to keep the default polling cadence.
   • API Rate Limit (optional): Cap the connector's outbound request rate to the Cloud Storage API. Leave blank to use the default. See API Rate Limiting.
4. In the "Secrets" section, provide the contents of your service account JSON key file.

Working with Prefix and Partition Format

The combination of prefix and partition_format determines how the input navigates your bucket's folder structure to find files.

Simple Date Format

The Simple Date format uses a date-based folder structure in the format YYYY/MM/DD.

• Without Prefix: Files are fetched directly from date-formatted folders

  Code
  bucket/
    2026/
      06/
        02/
          data.json

• With Prefix: Files are fetched from date-formatted folders under the specified prefix (e.g. data/device-logs)

  Code
  bucket/
    data/
      device-logs/
        2026/
          06/
            02/
              data.json

Hive Format

The Hive format uses a more explicit folder structure in the format year=YYYY/month=MM/day=DD.

• Without Prefix: Files are fetched from Hive-formatted folders

  Code
  bucket/
    year=2026/
      month=06/
        day=02/
          data.json

• With Prefix: Files are fetched from Hive-formatted folders under the specified prefix

  Code
  bucket/
    data/
      device-logs/
        year=2026/
          month=06/
            day=02/
              data.json

Troubleshooting

Common Issues

1. Access Denied: Ensure your service account has the correct IAM roles assigned.
2. No Files Found: Verify that the bucket name, prefix, and partition format match your bucket structure.
3. Invalid Credentials: Make sure the credentials_json is correct.
4. Parse Errors: Ensure the file format and record location settings match your data structure.

Related Articles

• Google Cloud Storage Documentation
• Managing Service Accounts
• Working with Cloud Storage Objects

Sync frequency

This input polls on a connector-specific interval, adjusting between about 10 seconds and 5 minutes depending on the backlog. Set the optional Cron Schedule (cron) setting to override this cadence and poll on a fixed schedule instead (e.g. 0 * * * * for hourly); leaving it blank keeps the default polling behavior. A cron schedule configured on the pipeline also overrides this cadence. See Input Sync Frequency for details.

To reduce load against the source API instead of (or in addition to) slowing the cadence, use the optional API Rate Limit (rate_limit) setting to cap the connector's outbound request rate. See API Rate Limiting.