Google Cloud Logs
Collects and ingests records from the Google Cloud Logging API.
Sync Type: Incremental
High volume: route through Pub/Sub
This connector reads directly from the Cloud Logging API: simple to set up, but reads are capped at 60,000 entries per minute per project with no Google-managed checkpoint, so at high volume entries can lag or drop. For high-volume or no-loss collection, route logs through a Log Router sink into Pub/Sub and use the Pub/Sub input instead.
Requirements
Before setting up the Google Cloud Logs input, you need to:
- Have a Google Cloud Platform account with access to the desired project.
- Create a service account with the necessary permissions.
- Generate a JSON key for the service account.
Details
The Google Cloud Logs input allows you to collect and ingest log data from Google Cloud Platform (GCP) services using the Google Cloud Logging API.
Delivery delay
Cloud Logging does not make an entry queryable the instant it is received, so the connector reads on a short trailing delay. Each sync collects entries by their receiveTimestamp (the time Google ingested the entry) up to roughly 2 minutes behind the current time, which lets late-arriving entries settle before they are read so none are skipped. In practice, records are delivered about 2 minutes after Google ingests them.
First run and history
On the first run the input starts about 2 minutes behind the current time. Entries Google received before the input was enabled are never read, and there is no backfill setting. If a run is interrupted, the open window is re-read on the next run, so duplicates are possible but entries are not lost.
Filters
The Filter setting is appended to the connector's own receiveTimestamp window with AND.
In the Logging query language AND binds tighter than OR, so wrap a filter that uses a top-level OR in parentheses; otherwise the OR branch escapes the time window and is re-read on every sync.
Proto payload parsing
With Enable Proto Payload Parsing on (the default), protoPayload entries of type google.cloud.audit.AuditLog and google.appengine.logging.v1.RequestLog are decoded into JSON.
Entries with any other protoPayload type are logged as unprocessable and dropped.
Turn the setting off to receive those entries with the payload left as an encoded Any value.
Configuration
Settings
| Setting | Type | Required | Default | Description |
|---|---|---|---|---|
resource_names | array of string | Yes | - | The resources to query logs from. |
filter | string | No | - | The filter to apply to the logs. |
enable_proto_payload_parsing | bool | No | true | Enables automatic parsing of embedded protocol buffer payloads within the input. |
use_synthetic_data | bool | No | false | Generate synthetic data for testing, instead of connecting to a real data source. Defaults to an hourly cron schedule for cron-based inputs. |
Secrets
| Secret | Type | Required | Description |
|---|---|---|---|
credentials_json | secret | No | JSON credentials to authenticate with Google Cloud (required for service account auth). |
Setting up the Service Account
- Go to the Google Cloud Console.
- Select your project from the project dropdown at the top of the page.
- Navigate to "IAM & Admin" > "Service Accounts".
- Click "Create Service Account".
- Enter a name for the service account and click "Create".
- Assign the following role to the service account:
- Logs Viewer
- Private Logs Viewer (
roles/logging.privateLogViewer), only if your filter includes Data Access audit logs
- Click "Continue" and then "Done".
- Find the newly created service account in the list and click on it.
- Go to the "Keys" tab and click "Add Key" > "Create new key".
- Choose JSON as the key type and click "Create".
- Save the downloaded JSON key file securely.
Setting up the Input
- In the Monad UI, go to the "Inputs" section.
- Click "Add Input" and select "Google Cloud Logs".
- Configure the input settings:
- Resource Name: A list of Google Cloud resources you want to collect logs from (e.g.,
["projects/my-project", "folders/my-folder", "organizations/my-org"]) - Filter: A Google Cloud log filter to specify which logs to collect (e.g.,
severity>=WARNING)
- Resource Name: A list of Google Cloud resources you want to collect logs from (e.g.,
- In the "Secrets" section, paste your service account JSON key file, either as raw JSON or base64-encoded.
Config
Code
Related Articles
Record format
Each record is one Cloud Logging entry encoded from the API's protobuf representation, not the REST JSON shown in the Cloud Console.
Keys are snake_case (insert_id, log_name, receive_timestamp), timestamp and receive_timestamp are objects with seconds and nanos, and severity is the numeric enum value (for example 400 for WARNING).
The payload is under Payload: decoded audit payloads appear as Payload.ProtoPayload.value, text payloads as Payload.TextPayload, and JSON payloads as Payload.JsonPayload in protobuf Struct form.
Sample Record
Code
Sync frequency
By default this input polls approximately every 10 seconds, with each sync beginning after the previous one completes. A cron schedule configured on the pipeline overrides this cadence. See Input Sync Frequency for details.