Copy page

Outputs

Scanner (S3 Source)

This output writes logs to an Amazon S3 bucket in your AWS account that Scanner indexes for search and detection. Scanner does not host any log buckets — Monad writes logs to a bucket in your account, and Scanner indexes them in place.

Prerequisites

1. Integrate your AWS account with Scanner. Deploying Scanner's Terraform stack creates the IAM role that grants Scanner read access to your log buckets, along with an index-files bucket in your account.
2. Create a Custom Logs AWS S3 source in Scanner pointing at the bucket (and prefix) this output writes to, so Scanner indexes those objects. Make sure the format and compression you configure on the Scanner source match what you select here.

Configuration

Configuring the Scanner (S3 Source) connector follows the same flow as Monad's AWS S3 Output Connector, with a config surface tailored to what Scanner can ingest.

Authenticate with either an IAM role (provide the AWS IAM Role ARN setting) or static credentials (provide the Access Key and Secret Key secrets). The prefix should match the prefix configured on your Scanner S3 source.

Settings

Setting	Type	Required	Default	Description
S3 Bucket Name	string	Yes	-	The S3 bucket in your AWS account that Scanner indexes.
AWS Region	string	Yes	us-east-1	The AWS region where the S3 bucket is located.
S3 Object Prefix	string	No	-	Optional prefix for S3 object keys. Should match the prefix configured on your Scanner S3 source.
AWS IAM Role ARN	string	No	-	The ARN of the IAM role Monad assumes to write to the bucket. Provide this or static credentials (Secrets).
Format	object	Yes	json	The on-disk format Scanner indexes — see Format Options below.
Compression	string	Yes	gzip	Compression applied before upload. One of none or gzip; Scanner indexes both.
Partition Format	string	Yes	simple date	Directory structure used to partition stored objects. Scanner indexes objects regardless of layout.
Batch Configuration	object	No	See defaults	Controls when a batch is written to S3 — see Batch Configuration below.

Format Options

Scanner indexes a subset of the S3 output's formats. Configure exactly one; see the Formats reference for full details.

• JSON — JSON Lines (one object per line) or a JSON array of records.
• Parquet — Apache Parquet, written with the schema you provide.
• Delimited (CSV) — delimited text with a configurable delimiter and optional headers.

The S3 output's "nested" JSON shape is intentionally not offered here — Scanner's S3 source indexes JSON Lines and JSON arrays, so a single wrapping object would not index cleanly.

Partition Format Options

1. Simple Date Format (simple date): {prefix}/{YYYY}/{MM}/{DD}/{filename}
2. Hive Compliant Format (hive compliant): {prefix}/year={YYYY}/month={MM}/day={DD}/{filename} — compatible with Athena, Hive, and similar query engines.
3. Flat Hive Compliant Format (flat hive compliant): {prefix}/dt={YYYY-MM-DD}/{filename} — single-key Hive-style partitioning.

All partition formats use UTC time. Scanner indexes objects regardless of the layout you choose.

Batch Configuration

Setting	Type	Default	Min	Max	Description
Record Count	integer	100,000	500	1,000,000	Maximum number of records per file
Data Size	integer	10 MB	1 MB	25 MB	Maximum uncompressed data size per file
Publish Rate	integer	45 seconds	1 second	60 seconds	Maximum time before flushing a batch

Secrets

Required only when authenticating with static credentials instead of an IAM role.

Secret	Type	Required	Description
Access Key	string	No	AWS access key ID. Provide this and the secret key, or use an IAM role instead.
Secret Key	string	No	AWS secret access key.