Skip to main content

Pub/Sub

Ingests messages from a Google Cloud Pub/Sub subscription.

Sync Type: Incremental

Overview​

The Google Cloud Pub/Sub input allows you to consume streaming messages from a Google Cloud Pub/Sub subscription. This input is ideal for ingesting real-time event data, logs, and messages published to a Pub/Sub topic by Google Cloud services or your own applications.

Key Features​

  • Real-time Message Consumption: Pulls messages as they arrive in the subscription
  • Automatic Acknowledgment: Messages are automatically acknowledged after processing
  • State Tracking: Maintains checkpoint state for reliable message processing
  • Configurable Subscription: Specify any Pub/Sub subscription within your GCP project

How It Works​

The input continuously pulls messages from the specified Pub/Sub subscription:

  1. Connection: Connects to the subscription using GCP credentials
  2. Message Consumption: Pulls messages as they arrive with a max outstanding message limit of 1
  3. Processing: Each message is passed through with its metadata (ID, publish timestamp)
  4. Checkpointing: State is saved periodically to track processing progress
  5. Acknowledgment: Messages are automatically acknowledged after successful processing

Requirements​

Before setting up the Google Cloud Pub/Sub input, you need to:

  1. Have a Google Cloud Platform account with access to the desired project
  2. Have an existing Pub/Sub subscription created and ready to receive messages
  3. Create a service account with the necessary permissions
  4. Generate a JSON key for the service account

Details​

Authentication​

The input uses Google Cloud service account credentials for authentication. You must provide a JSON service account key that has permissions to pull messages from your Pub/Sub subscription.

Message Delivery​

  • Messages are pulled from the subscription with a maximum of 1 outstanding message at a time
  • Each message includes:
    • Message ID (unique identifier)
    • Publish timestamp (when the message was published)
    • Message data (the actual content)
    • Attributes (key-value pairs attached to the message)
  • Messages are automatically acknowledged after processing

Checkpointing​

The input saves checkpoint state every 100 messages to ensure reliable processing. If the connection is interrupted, the input will resume from the last checkpoint.

Prerequisites​

Service Account Setup​

  1. Navigate to the Google Cloud Console
  2. Select your project from the dropdown at the top
  3. Go to IAM & Admin > Service Accounts
  4. Click Create Service Account
  5. Enter a name (e.g., "monad-pubsub-input") and click Create
  6. Assign the following role to the service account:
    • Pub/Sub Subscriber - Required to pull messages from subscriptions
  7. Click Continue and then Done
  8. Find the service account in the list and click on it
  9. Go to the Keys tab and click Add Key > Create new key
  10. Choose JSON as the key type and click Create
  11. The JSON key file will be downloaded - store it securely

Subscription Requirements​

Ensure that:

  1. You have an existing Pub/Sub subscription created
  2. The service account has the "Pub/Sub Subscriber" role on the project
  3. The subscription is active and can receive messages
  4. You know the subscription's name (format: typically just the name, like "my-subscription")

Configuration​

Settings​

SettingTypeRequiredDescription
project_idstringYesThe GCP project ID that contains the Pub/Sub subscription
subscription_idstringYesThe Pub/Sub subscription name to pull messages from

Secrets​

SecretTypeRequiredDescription
credentials_jsonstringYesJSON credentials from your service account key file

Setting up the Input​

  1. In the Monad UI, go to the Inputs section
  2. Click Add Input and select Google Cloud Pub/Sub
  3. Configure the input settings:
    • Project ID: Your Google Cloud project ID (e.g., my-project-123)
    • Subscription ID: The name of your Pub/Sub subscription (e.g., my-subscription)
  4. In the Secrets section, paste the entire contents of your service account JSON key file into the Credentials JSON field
  5. Click Save to configure the input

Rate Limits​

Google Cloud Pub/Sub has the following rate limits and considerations:

ScopeLimitNotes
Pull Requests1,000 requests per second per subscriptionAggregated across all connections
Message RateBased on subscription quotaConfigured in your GCP project
Concurrent ConnectionsConfigurable per projectDefault allows multiple concurrent subscribers