Aiven Service Logs
Collects and ingests service logs from Aiven, providing access to application and service-level logs for monitoring, debugging, and analysis.
Sync Type: Incremental
Overview
The Aiven Service Logs input connector retrieves operational logs from your Aiven services, including application logs, system logs, and service-specific diagnostic information. This connector enables you to centralize log data from your Aiven-managed services (such as PostgreSQL, MySQL, Kafka, OpenSearch, and others) into your Monad pipeline for analysis, monitoring, and compliance.
Prerequisites
Before configuring this input, you need:
-
Active Aiven Account - You must have an Aiven account with at least one active service. Both free and paid tier accounts are supported.
-
Project and Service Information:
- Identify the Aiven project name where your service is located. Projects are organizational units that group services together.
- Identify the service name within that project. This is the specific service (e.g., PostgreSQL, Kafka) from which you want to collect logs.
- You can find both in the Aiven Console by navigating to your service and checking the URL or service details page.
-
Authentication Token Creation:
- You need to create a personal authentication token with appropriate permissions.
- See the Authentication Setup section below for detailed instructions.
-
Understanding Aiven's Log Retention:
- Aiven retains service logs for 4 days by default in the platform.
- The connector can access logs within this retention window.
Authentication Setup
Aiven uses token-based authentication. Follow these steps to create and configure your authentication token:
Step 1: Generate Authentication Token
- Log into your Aiven account at console.aiven.io
- Click on your user profile in the top-right corner
- Select User Profile from the dropdown menu
- Navigate to the Authentication tab
- Under Personal tokens, click Generate token
- Configure the token:
- Token description: Enter a descriptive name (e.g., "Monad Service Logs Connector")
- Expiration: Choose an appropriate expiration time
- Click Generate token
- Copy the generated token immediately - you will not be able to access it again
Step 2: Verify Token Permissions
- In the Aiven Console, navigate to your Project
- Go to Settings > Members
- Find the user or application associated with your token
- Verify the role includes log reading permissions:
- Admin: Full access including logs
- Operator: Can read logs and manage services
- Developer: Can read logs
- Custom role: Must explicitly include
service:logs:read
Token Security Best Practices
- Store tokens securely and never commit them to version control
- Rotate tokens periodically (every 90-180 days recommended)
- Create separate tokens for different applications or environments
- Revoke tokens immediately if compromised
- Use tokens with minimal required permissions
Configuration Reference
The following configuration defines the input parameters. Each field's specifications are detailed below.
Settings
| Setting | Type | Required | Default | Description |
|---|---|---|---|---|
project | string | Yes | - | The Aiven project name |
service | string | Yes | - | The Aiven service name |
use_synthetic_data | bool | No | false | Generate synthetic data for testing, instead of connecting to a real data source. Defaults to an hourly cron schedule for cron-based inputs. |
backfill_start_time | date | No | - | The date to start fetching data from. If not specified, no past records will be fetched. |
rate_limit | object | No | - | Maximum outbound source-API request rate. A blank or zero value falls back to the connector's default rate-limiting behavior. Maximum 10 requests per second. See API Rate Limiting for the field format. |
Secrets
| Secret | Type | Required | Description |
|---|---|---|---|
auth_token | secret | Yes | The authentication token |
Example Log Entry
Code
Troubleshooting
Common issues and solutions:
Authentication Errors
Error: HTTP 401 Unauthorized or Invalid credentials
Possible Causes:
- Token is invalid or expired
- Token format is incorrect
- Token doesn't have required permissions
Solutions:
- Verify your token is still active in the Aiven Console (User Profile > Authentication > Personal tokens)
- Check that your user has
service:logs:readpermission for the specified project - Generate a new token if the current one has expired
Service or Project Not Found
Error: HTTP 404 Not Found or Service not found
Possible Causes:
- Project name is incorrect or misspelled
- Service name is incorrect or misspelled
- Service has been deleted or renamed
- Token doesn't have access to the specified project
Solutions:
- Verify the exact project name in the Aiven Console (case-sensitive)
- Verify the exact service name in the Aiven Console (case-sensitive)
- Check that your token's associated user has access to the project
No Logs Returned
Error: Connector runs successfully but no logs are ingested
Possible Causes:
- Service hasn't generated logs in the specified time range
- Backfill start time is too recent (no logs exist in that window)
- Backfill start time is beyond the 4-day retention window
- Service is not actively running
Solutions:
- Verify the service is in a "Running" state in the Aiven Console
- Check the service has activity (database queries, message processing, etc.)
- Adjust the backfill start time to a more recent period within the 4-day retention window
- Set an earlier Backfill Start Time to fetch older logs (without one, the input starts from when it was created)
- Manually check logs in the Aiven Console (Logs tab) to confirm logs exist
References
- Aiven API Documentation - Official Aiven REST API reference
- Aiven Authentication Tokens - Token types and management
- Create Personal Tokens - Step-by-step token creation guide
- Aiven Roles and Permissions - Permission model and role details
- Access Service Logs - Aiven Console log access guide
- Organizations, Units, and Projects - Understanding Aiven hierarchy
- Aiven Service Management - Managing Aiven services
Sync frequency
By default this input polls approximately every 10 seconds, with each sync beginning after the previous one completes. A cron schedule configured on the pipeline overrides this cadence. See Input Sync Frequency for details.
Community transforms
The Monad Transforms Repository has ready-made transforms for this input. Import one into your organization, then add it to a pipeline after this input.
| Format | Transform |
|---|---|
| OCSF v1.9.0 | Aiven Service Logs to OCSF v1.9.0 |
| ECS v9.5.0 | Aiven Service Logs to ECS v9.5.0 |
| CEF | Aiven Service Logs to CEF |
CEF transforms emit an intermediate record; add the CEF Serializer transform after one to produce the CEF:0 string.