Copy page

AWS

RDS Audit Logs

Ingests audit logs from an AWS RDS DB instance using the RDS log files API.

Sync Type: Incremental

Overview

The AWS RDS Audit Logs input reads audit log files directly from an RDS DB instance using the DescribeDBLogFiles and DownloadDBLogFilePortion APIs. Each non-empty line in the downloaded log files is emitted as a record. Because log line formats differ across database engines, records are emitted as raw log lines rather than parsed into individual fields.

How It Works

1. Monad calls DescribeDBLogFiles to list log files available on the configured DB instance, filtered to the engine-appropriate file prefix (audit/ for MySQL and MariaDB; error/postgresql.log.* for Postgres).
2. For each new or updated log file, Monad calls DownloadDBLogFilePortion to stream the file contents in pages of 1,000 lines.
3. Monad keeps a checkpoint per log file (position + last-written timestamp) so that subsequent runs resume from where they left off. The checkpoint is saved after each page of 1,000 lines, so if a run is interrupted, up to one page of lines can be emitted again on the next run.
4. If a run is interrupted mid-download, the next run detects that the stored LastWritten value is below the live value and resumes from the saved marker position.

Supported Engines

Engine	Audit log file prefix
MySQL	audit/
MariaDB	audit/
Postgres	error/postgresql.log.*

Only RDS instances whose engine is exactly MySQL, MariaDB or PostgreSQL are supported. Amazon Aurora (aurora-mysql, aurora-postgresql) and other engines are not supported by this input; it fails with an unsupported-engine error.

For PostgreSQL, Monad ingests every line of the PostgreSQL server log, not only pgaudit lines. Errors, connection messages and other server log output are emitted as records alongside the audit entries; filter on the message content (for example, lines containing AUDIT:) downstream if you want audit entries only.

Note: If the RDS instance has audit logs routed to CloudWatch (i.e., audit appears in the instance's EnabledCloudwatchLogsExports), this input will return an error and you should use the CloudWatch Logs input instead.

Requirements

Before using this connector, ensure the following prerequisites are met.

1. Enable Audit Logging on the RDS Instance

Audit logging must be enabled on the DB instance and configured to write to RDS log files (not CloudWatch Logs).

MySQL / MariaDB — Enable the MariaDB Audit Plugin via a custom DB option group:

1. Create a custom option group for your engine and major version.
2. Add the MARIADB_AUDIT_PLUGIN option to the option group, and set any option settings you need (for example, SERVER_AUDIT_EVENTS).
3. Apply the option group to your DB instance.

See MariaDB Audit Plugin support for MySQL in the AWS documentation for the available option settings.

PostgreSQL — Enable pgaudit via a custom parameter group:

1. Add pgaudit to the shared_preload_libraries parameter.
2. Set pgaudit.log to the statement classes you want to capture (e.g., ddl, write, role).
3. Apply the parameter group and reboot if required.

For full setup instructions, see the Amazon RDS Database Log Files documentation.

2. Grant IAM Permissions

The role or credentials used must have permission to describe and download log files for the target DB instance.

Minimum required permissions:

Code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RDSAuditLogsPermissions",
      "Effect": "Allow",
      "Action": [
        "rds:DescribeDBInstances",
        "rds:DescribeDBLogFiles",
        "rds:DownloadDBLogFilePortion"
      ],
      "Resource": "arn:aws:rds:<region>:<account-id>:db:<db-instance-identifier>"
    }
  ]
}

Replace <region>, <account-id>, and <db-instance-identifier> with your values. To scope the policy to all instances in the account, replace the Resource value with "*".

For Role ARN Authentication:

1. Create or update an IAM role in your AWS account.
2. Add the above permissions to the role.
3. Update the trust relationship to allow Monad to assume it (see AWS Authentication Methods).

For Static Credentials:

1. Create or use an IAM user with programmatic access.
2. Attach the above policy.
3. Generate or use an existing Access Key ID and Secret Access Key.

Configuration

Settings

Setting	Type	Required	Default	Description
role_arn	string	No	-	The ARN of the IAM role to assume for accessing AWS resources.
region	string	Yes	us-east-1	The AWS region to use. Allowed values: us-east-1, us-east-2, us-west-1, us-west-2, af-south-1, ap-east-1, ap-south-1, ap-northeast-1, ap-northeast-2, ap-northeast-3, ap-southeast-1, ap-southeast-2, ca-central-1, ca-west-1, cn-north-1, cn-northwest-1, eu-central-1, eu-west-1, eu-west-2, eu-west-3, eu-north-1, eu-south-1, me-south-1, sa-east-1, us-gov-east-1, us-gov-west-1.
db_instance_identifier	string	Yes	-	Identifier of the RDS DB instance to fetch audit logs from.
backfill_start_time	date	No	-	Date to start fetching data from, in ISO 8601 format. If not specified, every audit log file RDS still retains is read on the first sync. All syncs thereafter are incremental.
use_synthetic_data	bool	No	false	Generate synthetic data for testing, instead of connecting to a real data source. Defaults to an hourly cron schedule for cron-based inputs.

Secrets (Static Credentials Only)

Secret	Type	Required	Description
access_key	secret	No	The Access Key from AWS to authenticate with AWS
secret_key	secret	No	The Secret Key from AWS to authenticate with AWS

Authentication Note: Choose either Role ARN (recommended) or static credentials (Access Key + Secret Key). See AWS Authentication Methods for detailed setup instructions.

Output Record Schema

Each record corresponds to one non-empty log line. Records are not parsed per field because the format differs across engines.

Field	Type	Description
engine	string	Database engine (mysql, mariadb, or postgres).
instance_id	string	RDS DB instance identifier.
log_file_name	string	Name of the log file the line was read from (e.g., audit/server_audit.log).
message	string	The raw log line.
region	string	AWS region of the DB instance.
source	string	Always rds_log_file.

Sample Record

Code
{
  "engine": "mysql",
  "instance_id": "prod-mysql-01",
  "log_file_name": "audit/server_audit.log",
  "message": "20240115 10:30:45,ip-10-0-1-5,admin,prod_db,52,1001,QUERY,`SELECT * FROM users WHERE id = 1`,,0",
  "region": "us-east-1",
  "source": "rds_log_file"
}

Related Articles

• Amazon RDS Database Log Files
• RDS DescribeDBLogFiles API
• RDS DownloadDBLogFilePortion API
• Using pgaudit with Amazon RDS for PostgreSQL
• MariaDB Audit Plugin for Amazon RDS

Sync frequency

By default this input polls approximately every 10 seconds, with each sync beginning after the previous one completes. A cron schedule configured on the pipeline overrides this cadence. See Input Sync Frequency for details.