AWS Redshift Audit Logs
Ingests Amazon Redshift database audit logs (connection and user logs) delivered to an Amazon S3 bucket. These logs help you monitor authentication, connection activity, and changes to database user definitions for security and compliance.
Sync Type: Incremental
Overview
Amazon Redshift can publish database audit logs to an S3 bucket. This connector reads those log files from S3, parses them, and emits one record per log line. It is a focused wrapper over the S3 input: you choose which audit log type to ingest, and Monad handles the bucket layout, file format, and incremental syncing for you.
How It Works
- Redshift delivers gzip-compressed, pipe-separated audit log files to your S3 bucket under
AWSLogs/{AccountID}/redshift/{Region}/{Year}/{Month}/{Day}/. - Each file name carries the log type, e.g.
..._connectionlog_2013-10-29T18:01.gz. - Based on the selected Log Type, Monad processes only the matching files and parses each line using the documented column layout for that log type.
- An incremental cursor (based on object last-modified time and key ordering) tracks progress, so only new objects are processed on subsequent runs.
Requirements
1. Enable Audit Logging to S3 on Your Redshift Cluster
Audit logging is not enabled by default. Enable it and configure S3 as the destination:
- Navigate to Amazon Redshift in the AWS Console.
- Select your cluster, then go to Properties → Database configurations → Edit audit logging.
- Turn on audit logging and choose S3 as the log export type.
- Select or create a destination bucket (and optional key prefix).
For details, see Database audit logging.
2. Grant IAM Permissions
The role or credentials used must be able to list and read objects in the audit log bucket.
Code
If the bucket uses SSE-KMS, the role also needs kms:Decrypt on that key.
See AWS Authentication Methods for role-assumption vs. static-credential setup.
Configuration
Settings
| Setting | Type | Required | Default | Description |
|---|---|---|---|---|
region | string | No | - | The region of the S3 bucket. If left blank, the region will be auto-detected. Allowed values: us-east-1, us-east-2, us-west-1, us-west-2, af-south-1, ap-east-1, ap-south-1, ap-northeast-1, ap-northeast-2, ap-northeast-3, ap-southeast-1, ap-southeast-2, ca-central-1, ca-west-1, cn-north-1, cn-northwest-1, eu-central-1, eu-west-1, eu-west-2, eu-west-3, eu-north-1, eu-south-1, me-south-1, sa-east-1, us-gov-east-1, us-gov-west-1. |
bucket | string | Yes | - | Name of the S3 bucket that receives Redshift audit logs. |
prefix | string | No | - | Prefix of the audit log keys, up to (but not including) the date partition. If you set a custom S3 key prefix when enabling audit logging, include it here. |
log_type | string | Yes | connectionlog | Which Amazon Redshift audit log to ingest. Allowed values: connectionlog (Connection Log), userlog (User Log). See Log Types. |
role_arn | string | No | - | The ARN of the IAM role to assume for accessing AWS resources. |
backfill_start_time | date | No | - | The date to start fetching data from. If not specified, no past records will be fetched. |
Secrets
| Secret | Type | Required | Description |
|---|---|---|---|
access_key | secret | No | The Access Key from AWS to authenticate with AWS |
secret_key | secret | No | The Secret Key from AWS to authenticate with AWS |
Authentication Note: Choose either Role ARN (recommended) or static credentials (Access Key + Secret Key). See AWS Authentication Methods.
Log Types
Select a single log type per input. Each type has a fixed column layout; Monad emits one JSON record per log line with these fields.
Connection Log
Logs authentication attempts, connections, and disconnections.
| Field | Description |
|---|---|
event | Connection or authentication event. |
recordtime | Time the event occurred. |
remotehost | Name or IP address of remote host. |
remoteport | Port number for remote host. |
pid | Process ID associated with the statement. |
dbname | Database name. |
username | User name. |
authmethod | Authentication method. |
duration | Duration of connection in microseconds. |
sslversion | SSL version. |
sslcipher | SSL cipher. |
mtu | Maximum transmission unit. |
sslcompression | SSL compression type. |
sslexpansion | SSL expansion type. |
iamauthguid | IAM authentication ID for the CloudTrail request. |
application_name | Initial or updated application name for a session. |
os_version | Operating system version of the client machine. |
driver_version | ODBC/JDBC driver version. |
plugin_name | Name of the plugin used to connect. |
protocol_version | Internal protocol version used by the driver. |
sessionid | Globally unique identifier for the session. |
compression | Compression algorithm in use for the connection. |
User Log
Records changes to database user definitions (create, drop, rename, alter).
| Field | Description |
|---|---|
userid | ID of the user affected by the change. |
username | User name of the user affected by the change. |
oldusername | For a rename, the original user name; otherwise empty. |
action | Action that occurred. |
usecreatedb | If true (1), the user has create-database permissions. |
usesuper | If true (1), the user is a superuser. |
usecatupd | If true (1), the user can update system catalogs. |
valuntil | Password expiration date. |
pid | Process ID. |
xid | Transaction ID. |
recordtime | Time in UTC that the query started. |
Details
Incremental Syncing
- First sync: if Backfill Start Time is not set, no existing objects are processed; ingestion starts with log files written after the input first runs. To ingest history, set a Backfill Start Time and the first sync processes objects from that date onward.
- Subsequent syncs: only objects newer than the last checkpoint are processed, tracked by last-modified time and lexicographic key ordering.
- Restart behavior: on interruption, the next run resumes from the day prefix of the last checkpointed timestamp. You will not lose records, though some objects on the in-progress page may be reprocessed after a catastrophic failure. A log file that cannot be parsed is an exception: it stops the input at that file (see Troubleshooting).
Bucket Structure
Redshift organizes audit logs as:
Code
Set Prefix to the path up to the region (e.g. AWSLogs/123456789012/redshift/us-east-1), prepending any custom key prefix you configured. The date partition ({Year}/{Month}/{Day}) is handled automatically.
Log Format
Audit log files are gzip-compressed, headerless, pipe-separated values. Monad applies the column layout for the selected log type, so no header row is expected in the files.
Troubleshooting
Input stops progressing on one log file
If a log file cannot be decompressed or parsed (for example, a line whose number of columns does not match the layout for the selected Log Type), the input fails on that file. The input does not skip it: every following sync resumes from the same checkpoint and fails on the same file, so no newer logs are ingested. To recover, remove the file from the prefix (for example, move it to another location in the bucket), and the input continues from there.
Related Articles
Sync frequency
By default this input polls the bucket every 5 minutes. A cron schedule configured on the pipeline overrides this cadence. See Input Sync Frequency for details.
Community transforms
The Monad Transforms Repository has ready-made transforms for this input. Import one into your organization, then add it to a pipeline after this input.
| Format | Transform |
|---|---|
| OCSF v1.9.0 | AWS Redshift Audit Logs to OCSF v1.9.0 |
| ECS v9.5.0 | AWS Redshift Audit Logs to ECS v9.5.0 |
| CEF | AWS Redshift Audit Logs to CEF |
CEF transforms emit an intermediate record; add the CEF Serializer transform after one to produce the CEF:0 string.