# Service Logs

Collects and ingests service logs from Aiven, providing access to application and service-level logs for monitoring, debugging, and analysis.

**Sync Type: Incremental**

## Overview

The Aiven Service Logs input connector retrieves operational logs from your Aiven services, including application logs, system logs, and service-specific diagnostic information. This connector enables you to centralize log data from your Aiven-managed services (such as PostgreSQL, MySQL, Kafka, OpenSearch, and others) into your Monad pipeline for analysis, monitoring, and compliance.

## Prerequisites

Before configuring this input, you need:

1. **Active Aiven Account** - You must have an Aiven account with at least one active service. Both free and paid tier accounts are supported. 

2. **Project and Service Information**:
   - Identify the Aiven **project name** where your service is located. Projects are organizational units that group services together.
   - Identify the **service name** within that project. This is the specific service (e.g., PostgreSQL, Kafka) from which you want to collect logs.
   - You can find both in the Aiven Console by navigating to your service and checking the URL or service details page.

3. **Authentication Token Creation**:
   - You need to create a personal authentication token with appropriate permissions.
   - See the [Authentication Setup](#authentication-setup) section below for detailed instructions.

4. **Understanding Aiven's Log Retention**:
   - Aiven retains service logs for **4 days by default** in the platform.
   - The connector can access logs within this retention window.

## Authentication Setup

Aiven uses token-based authentication. Follow these steps to create and configure your authentication token:

### Step 1: Generate Authentication Token

1. Log into your Aiven account at [console.aiven.io](https://console.aiven.io)
2. Click on your user profile in the top-right corner
3. Select **User Profile** from the dropdown menu
4. Navigate to the **Authentication** tab
5. Under **Personal tokens**, click **Generate token**
6. Configure the token:
   - **Token description**: Enter a descriptive name (e.g., "Monad Service Logs Connector")
   - **Expiration**: Choose an appropriate expiration time
7. Click **Generate token**
8. Copy the generated token immediately - you will not be able to access it again

### Step 2: Verify Token Permissions

1. In the Aiven Console, navigate to your **Project**
2. Go to **Settings** > **Members**
3. Find the user or application associated with your token
4. Verify the role includes log reading permissions:
   - **Admin**: Full access including logs
   - **Operator**: Can read logs and manage services
   - **Developer**: Can read logs
   - Custom role: Must explicitly include `service:logs:read`

### Token Security Best Practices

- Store tokens securely and never commit them to version control
- Rotate tokens periodically (every 90-180 days recommended)
- Create separate tokens for different applications or environments
- Revoke tokens immediately if compromised
- Use tokens with minimal required permissions

## Configuration Reference

The following configuration defines the input parameters. Each field's specifications are detailed below.

### Settings

| Setting | Type | Required | Default | Description |
|---------|------|----------|---------|-------------|
| **Project** | string | Yes | - | The Aiven project name containing your service. Projects organize services and control access. |
| **Service** | string | Yes | - | The Aiven service name within the project. This is the specific service instance (e.g., "my-postgres-db", "kafka-production") from which to collect logs. |
| **Backfill Start Time** | string | No | - | The date to start fetching data from. If not specified, no past records will be fetched. |
| API Rate Limit | object | No | - | Optional limit on the connector's outbound request rate to the source API. Leave blank to use the connector's default behavior. See [API Rate Limiting](../../../guides/rate-limiting) for the field format, limits, and how to choose a value. |

### Secrets

| Secret | Type | Required | Description |
|---------|------|----------|-------------|
| **Auth Token** | string | Yes | Aiven authentication token with `service:logs:read` permission. |

## Example Log Entry

```json
{
  "time": "2025-07-15T13:40:16.454037Z",
  "hostname": "pg-production-1",
  "msg": "user=admin,db=myapp,app=web,client=10.0.1.5 LOG: listening on IPv6 address \"::\", port 5432",
  "unit": "postgresql-14.service"
}
```

## Troubleshooting

Common issues and solutions:

### Authentication Errors

**Error**: `HTTP 401 Unauthorized` or `Invalid credentials`

**Possible Causes**:
- Token is invalid or expired
- Token format is incorrect
- Token doesn't have required permissions

**Solutions**:
1. Verify your token is still active in the Aiven Console (User Profile > Authentication > Personal tokens)
2. Check that your user has `service:logs:read` permission for the specified project
3. Generate a new token if the current one has expired

### Service or Project Not Found

**Error**: `HTTP 404 Not Found` or `Service not found`

**Possible Causes**:
- Project name is incorrect or misspelled
- Service name is incorrect or misspelled
- Service has been deleted or renamed
- Token doesn't have access to the specified project

**Solutions**:
1. Verify the exact project name in the Aiven Console (case-sensitive)
2. Verify the exact service name in the Aiven Console (case-sensitive)
3. Check that your token's associated user has access to the project

### No Logs Returned

**Error**: Connector runs successfully but no logs are ingested

**Possible Causes**:
- Service hasn't generated logs in the specified time range
- Backfill start time is too recent (no logs exist in that window)
- Backfill start time is beyond the 4-day retention window
- Service is not actively running

**Solutions**:
1. Verify the service is in a "Running" state in the Aiven Console
2. Check the service has activity (database queries, message processing, etc.)
3. Adjust the backfill start time to a more recent period within the 4-day retention window
4. Remove the backfill start time to fetch all available logs
5. Manually check logs in the Aiven Console (Logs tab) to confirm logs exist

## References

- [Aiven API Documentation](https://api.aiven.io/doc/) - Official Aiven REST API reference
- [Aiven Authentication Tokens](https://aiven.io/docs/platform/concepts/authentication-tokens) - Token types and management
- [Create Personal Tokens](https://aiven.io/docs/platform/howto/create_authentication_token) - Step-by-step token creation guide
- [Aiven Roles and Permissions](https://aiven.io/docs/platform/concepts/permissions) - Permission model and role details
- [Access Service Logs](https://aiven.io/docs/platform/howto/access-service-logs) - Aiven Console log access guide
- [Organizations, Units, and Projects](https://aiven.io/docs/platform/concepts/orgs-units-projects) - Understanding Aiven hierarchy
- [Aiven Service Management](https://aiven.io/docs/platform/howto/list-service) - Managing Aiven services

## Sync frequency

This input consumes from the source queue or subscription, ingesting records at whatever rate the source delivers them rather than on a fixed poll. See [Input Sync Frequency](../../guides/sync-frequency) for details.