Copy page

Aiven

Service Logs

Collects and ingests service logs from Aiven, providing access to application and service-level logs for monitoring, debugging, and analysis.

Sync Type: Incremental

Overview

The Aiven Service Logs input connector retrieves operational logs from your Aiven services, including application logs, system logs, and service-specific diagnostic information. This connector enables you to centralize log data from your Aiven-managed services (such as PostgreSQL, MySQL, Kafka, OpenSearch, and others) into your Monad pipeline for analysis, monitoring, and compliance.

Prerequisites

Before configuring this input, you need:

1. Active Aiven Account - You must have an Aiven account with at least one active service. Both free and paid tier accounts are supported.

2. Project and Service Information:

   • Identify the Aiven project name where your service is located. Projects are organizational units that group services together.
   • Identify the service name within that project. This is the specific service (e.g., PostgreSQL, Kafka) from which you want to collect logs.
   • You can find both in the Aiven Console by navigating to your service and checking the URL or service details page.

3. Authentication Token Creation:

   • You need to create a personal authentication token with appropriate permissions.
   • See the Authentication Setup section below for detailed instructions.

4. Understanding Aiven's Log Retention:

   • Aiven retains service logs for 4 days by default in the platform.
   • The connector can access logs within this retention window.

Authentication Setup

Aiven uses token-based authentication. Follow these steps to create and configure your authentication token:

Step 1: Generate Authentication Token

1. Log into your Aiven account at console.aiven.io
2. Click on your user profile in the top-right corner
3. Select User Profile from the dropdown menu
4. Navigate to the Authentication tab
5. Under Personal tokens, click Generate token
6. Configure the token:
   • Token description: Enter a descriptive name (e.g., "Monad Service Logs Connector")
   • Expiration: Choose an appropriate expiration time
7. Click Generate token
8. Copy the generated token immediately - you will not be able to access it again

Step 2: Verify Token Permissions

1. In the Aiven Console, navigate to your Project
2. Go to Settings > Members
3. Find the user or application associated with your token
4. Verify the role includes log reading permissions:
   • Admin: Full access including logs
   • Operator: Can read logs and manage services
   • Developer: Can read logs
   • Custom role: Must explicitly include service:logs:read

Token Security Best Practices

• Store tokens securely and never commit them to version control
• Rotate tokens periodically (every 90-180 days recommended)
• Create separate tokens for different applications or environments
• Revoke tokens immediately if compromised
• Use tokens with minimal required permissions

Configuration Reference

The following configuration defines the input parameters. Each field's specifications are detailed below.

Settings

Setting	Type	Required	Default	Description
Project	string	Yes	-	The Aiven project name containing your service. Projects organize services and control access.
Service	string	Yes	-	The Aiven service name within the project. This is the specific service instance (e.g., "my-postgres-db", "kafka-production") from which to collect logs.
Backfill Start Time	string	No	-	The date to start fetching data from. If not specified, no past records will be fetched.
API Rate Limit	object	No	-	Optional limit on the connector's outbound request rate to the source API. Leave blank to use the connector's default behavior. See API Rate Limiting for the field format, limits, and how to choose a value.

Secrets

Secret	Type	Required	Description
Auth Token	string	Yes	Aiven authentication token with service:logs:read permission.

Example Log Entry

Code
{
  "time": "2025-07-15T13:40:16.454037Z",
  "hostname": "pg-production-1",
  "msg": "user=admin,db=myapp,app=web,client=10.0.1.5 LOG: listening on IPv6 address \"::\", port 5432",
  "unit": "postgresql-14.service"
}

Troubleshooting

Common issues and solutions:

Authentication Errors

Error: HTTP 401 Unauthorized or Invalid credentials

Possible Causes:

• Token is invalid or expired
• Token format is incorrect
• Token doesn't have required permissions

Solutions:

1. Verify your token is still active in the Aiven Console (User Profile > Authentication > Personal tokens)
2. Check that your user has service:logs:read permission for the specified project
3. Generate a new token if the current one has expired

Service or Project Not Found

Error: HTTP 404 Not Found or Service not found

Possible Causes:

• Project name is incorrect or misspelled
• Service name is incorrect or misspelled
• Service has been deleted or renamed
• Token doesn't have access to the specified project

Solutions:

1. Verify the exact project name in the Aiven Console (case-sensitive)
2. Verify the exact service name in the Aiven Console (case-sensitive)
3. Check that your token's associated user has access to the project

No Logs Returned

Error: Connector runs successfully but no logs are ingested

Possible Causes:

• Service hasn't generated logs in the specified time range
• Backfill start time is too recent (no logs exist in that window)
• Backfill start time is beyond the 4-day retention window
• Service is not actively running

Solutions:

1. Verify the service is in a "Running" state in the Aiven Console
2. Check the service has activity (database queries, message processing, etc.)
3. Adjust the backfill start time to a more recent period within the 4-day retention window
4. Remove the backfill start time to fetch all available logs
5. Manually check logs in the Aiven Console (Logs tab) to confirm logs exist

References

• Aiven API Documentation - Official Aiven REST API reference
• Aiven Authentication Tokens - Token types and management
• Create Personal Tokens - Step-by-step token creation guide
• Aiven Roles and Permissions - Permission model and role details
• Access Service Logs - Aiven Console log access guide
• Organizations, Units, and Projects - Understanding Aiven hierarchy
• Aiven Service Management - Managing Aiven services