# Pub/Sub

Ingests messages from a Google Cloud Pub/Sub subscription.

**Sync Type: Incremental**

## Overview

The Google Cloud Pub/Sub input allows you to consume streaming messages from a Google Cloud Pub/Sub subscription. This input is ideal for ingesting real-time event data, logs, and messages published to a Pub/Sub topic by Google Cloud services or your own applications.

### Key Features

- **Real-time Message Consumption**: Pulls messages as they arrive in the subscription
- **Automatic Acknowledgment**: Messages are automatically acknowledged after processing
- **State Tracking**: Maintains checkpoint state for reliable message processing
- **Configurable Subscription**: Specify any Pub/Sub subscription within your GCP project

### How It Works

The input continuously pulls messages from the specified Pub/Sub subscription:

1. **Connection**: Connects to the subscription using GCP credentials
2. **Message Consumption**: Pulls messages as they arrive with a max outstanding message limit of 1
3. **Processing**: Each message is passed through with its metadata (ID, publish timestamp)
4. **Checkpointing**: State is saved periodically to track processing progress
5. **Acknowledgment**: Messages are automatically acknowledged after successful processing

## Requirements

Before setting up the Google Cloud Pub/Sub input, you need to:

1. Have a Google Cloud Platform account with access to the desired project
2. Have an existing Pub/Sub subscription created and ready to receive messages
3. Create a service account with the necessary permissions
4. Generate a JSON key for the service account

## Details

### Authentication

The input uses Google Cloud service account credentials for authentication. You must provide a JSON service account key that has permissions to pull messages from your Pub/Sub subscription.

### Message Delivery

- Messages are pulled from the subscription with a maximum of 1 outstanding message at a time
- Each message includes:
  - Message ID (unique identifier)
  - Publish timestamp (when the message was published)
  - Message data (the actual content)
  - Attributes (key-value pairs attached to the message)
- Messages are automatically acknowledged after processing

### Checkpointing

The input saves checkpoint state every 100 messages to ensure reliable processing. If the connection is interrupted, the input will resume from the last checkpoint.

## Prerequisites

### Service Account Setup

1. Navigate to the [Google Cloud Console](https://console.cloud.google.com/)
2. Select your project from the dropdown at the top
3. Go to **IAM & Admin** > **Service Accounts**
4. Click **Create Service Account**
5. Enter a name (e.g., "monad-pubsub-input") and click **Create**
6. Assign the following role to the service account:
   - **Pub/Sub Subscriber** - Required to pull messages from subscriptions
7. Click **Continue** and then **Done**
8. Find the service account in the list and click on it
9. Go to the **Keys** tab and click **Add Key** > **Create new key**
10. Choose **JSON** as the key type and click **Create**
11. The JSON key file will be downloaded - store it securely

### Subscription Requirements

Ensure that:

1. You have an existing Pub/Sub subscription created
2. The service account has the "Pub/Sub Subscriber" role on the project
3. The subscription is active and can receive messages
4. You know the subscription's name (format: typically just the name, like "my-subscription")

## Configuration

### Settings

| Setting | Type | Required | Description |
|---------|------|----------|-------------|
| project_id | string | Yes | The GCP project ID that contains the Pub/Sub subscription |
| subscription_id | string | Yes | The Pub/Sub subscription name to pull messages from |

### Secrets

| Secret | Type | Required | Description |
|--------|------|----------|-------------|
| credentials_json | string | Yes | JSON credentials from your service account key file |

## Setting up the Input

1. In the Monad UI, go to the **Inputs** section
2. Click **Add Input** and select **Google Cloud Pub/Sub**
3. Configure the input settings:
   - **Project ID**: Your Google Cloud project ID (e.g., `my-project-123`)
   - **Subscription ID**: The name of your Pub/Sub subscription (e.g., `my-subscription`)
4. In the **Secrets** section, paste the entire contents of your service account JSON key file into the **Credentials JSON** field
5. Click **Save** to configure the input

## Rate Limits

Google Cloud Pub/Sub has the following rate limits and considerations:

| Scope | Limit | Notes |
|-------|-------|-------|
| Pull Requests | 1,000 requests per second per subscription | Aggregated across all connections |
| Message Rate | Based on subscription quota | Configured in your GCP project |
| Concurrent Connections | Configurable per project | Default allows multiple concurrent subscribers |

## Related Articles

- [Google Cloud Pub/Sub Documentation](https://cloud.google.com/pubsub/docs)
- [Creating and Managing Subscriptions](https://cloud.google.com/pubsub/docs/create-subscription)
- [Subscribing to Topics](https://cloud.google.com/pubsub/docs/subscriber)
- [Access Control with IAM](https://cloud.google.com/pubsub/docs/access-control)