Skip to main content

Service Logs

Collects and ingests service logs from Aiven, providing access to application and service-level logs for monitoring, debugging, and analysis.

Sync Type: Incremental

Overview​

The Aiven Service Logs input connector retrieves operational logs from your Aiven services, including application logs, system logs, and service-specific diagnostic information. This connector enables you to centralize log data from your Aiven-managed services (such as PostgreSQL, MySQL, Kafka, OpenSearch, and others) into your Monad pipeline for analysis, monitoring, and compliance.

Prerequisites​

Before configuring this input, you need:

  1. Active Aiven Account - You must have an Aiven account with at least one active service. Both free and paid tier accounts are supported.

  2. Project and Service Information:

    • Identify the Aiven project name where your service is located. Projects are organizational units that group services together.
    • Identify the service name within that project. This is the specific service (e.g., PostgreSQL, Kafka) from which you want to collect logs.
    • You can find both in the Aiven Console by navigating to your service and checking the URL or service details page.

  3. Authentication Token Creation:

    • You need to create a personal authentication token with appropriate permissions.
    • See the Authentication Setup section below for detailed instructions.

  4. Understanding Aiven's Log Retention:

    • Aiven retains service logs for 4 days by default in the platform.
    • The connector can access logs within this retention window.

Authentication Setup​

Aiven uses token-based authentication. Follow these steps to create and configure your authentication token:

Step 1: Generate Authentication Token​

  1. Log into your Aiven account at console.aiven.io
  2. Click on your user profile in the top-right corner
  3. Select User Profile from the dropdown menu
  4. Navigate to the Authentication tab
  5. Under Personal tokens, click Generate token
  6. Configure the token:
    • Token description: Enter a descriptive name (e.g., "Monad Service Logs Connector")
    • Expiration: Choose an appropriate expiration time
  7. Click Generate token
  8. Copy the generated token immediately - you will not be able to access it again

Step 2: Verify Token Permissions​

  1. In the Aiven Console, navigate to your Project
  2. Go to Settings > Members
  3. Find the user or application associated with your token
  4. Verify the role includes log reading permissions:
    • Admin: Full access including logs
    • Operator: Can read logs and manage services
    • Developer: Can read logs
    • Custom role: Must explicitly include service:logs:read

Token Security Best Practices​

  • Store tokens securely and never commit them to version control
  • Rotate tokens periodically (every 90-180 days recommended)
  • Create separate tokens for different applications or environments
  • Revoke tokens immediately if compromised
  • Use tokens with minimal required permissions

Configuration Reference​

The following configuration defines the input parameters. Each field's specifications are detailed below.

Settings​

SettingTypeRequiredDefaultDescription
ProjectstringYes-The Aiven project name containing your service. Projects organize services and control access.
ServicestringYes-The Aiven service name within the project. This is the specific service instance (e.g., "my-postgres-db", "kafka-production") from which to collect logs.
Backfill Start TimestringNo-The date to start fetching data from. If not specified, no past records will be fetched.

Secrets​

SecretTypeRequiredDescription
Auth TokenstringYesAiven authentication token with service:logs:read permission.

Example Log Entry​

{
  "time": "2025-07-15T13:40:16.454037Z",
  "hostname": "pg-production-1",
  "msg": "user=admin,db=myapp,app=web,client=10.0.1.5 LOG: listening on IPv6 address \"::\", port 5432",
  "unit": "postgresql-14.service"
}

Troubleshooting​

Common issues and solutions:

Authentication Errors​

Error: HTTP 401 Unauthorized or Invalid credentials

Possible Causes:

  • Token is invalid or expired
  • Token format is incorrect
  • Token doesn't have required permissions

Solutions:

  1. Verify your token is still active in the Aiven Console (User Profile > Authentication > Personal tokens)
  2. Check that your user has service:logs:read permission for the specified project
  3. Generate a new token if the current one has expired

Service or Project Not Found​

Error: HTTP 404 Not Found or Service not found

Possible Causes:

  • Project name is incorrect or misspelled
  • Service name is incorrect or misspelled
  • Service has been deleted or renamed
  • Token doesn't have access to the specified project

Solutions:

  1. Verify the exact project name in the Aiven Console (case-sensitive)
  2. Verify the exact service name in the Aiven Console (case-sensitive)
  3. Check that your token's associated user has access to the project

No Logs Returned​

Error: Connector runs successfully but no logs are ingested

Possible Causes:

  • Service hasn't generated logs in the specified time range
  • Backfill start time is too recent (no logs exist in that window)
  • Backfill start time is beyond the 4-day retention window
  • Service is not actively running

Solutions:

  1. Verify the service is in a "Running" state in the Aiven Console
  2. Check the service has activity (database queries, message processing, etc.)
  3. Adjust the backfill start time to a more recent period within the 4-day retention window
  4. Remove the backfill start time to fetch all available logs
  5. Manually check logs in the Aiven Console (Logs tab) to confirm logs exist

References​