Copy page

Solution Guides

Sending Data to Panther via S3

This guide shows how to deliver data from a Monad pipeline to Panther through an Amazon S3 bucket. Monad's Amazon S3 output writes files to the bucket, and Panther's S3 source picks up each new file and ingests it.

---

When to Use S3 Instead of HTTP

Monad can also send data straight to Panther with the Panther HTTP output. HTTP is simpler to set up, but Panther's HTTP source accepts at most 1 MB per request.

Use S3 when:

• Individual records can be larger than about 1 MB.
• You want larger, compressed files instead of many small requests.
• You already land security data in S3 and want Panther to read from the same place.

Panther states that data is ingested within minutes through any of its sources, so S3 doesn't add meaningful latency compared to HTTP.

The S3 path requires infrastructure in your own AWS account: an S3 bucket, an SNS topic, and IAM roles for both Monad and Panther.

---

How It Works

1. The Monad S3 output writes a batch of records to your bucket as a file.
2. S3 sends an object-created event to an SNS topic.
3. The SNS topic forwards the event to Panther's notifications queue.
4. Panther assumes a read-only IAM role in your account and reads the new file.

---

Before You Begin

You need:

• An AWS account where you can create S3 buckets, SNS topics, and IAM roles.
• Access to the Panther Console with permission to create log sources.
• A Monad pipeline with the input you want to send to Panther.
• Your Monad organization ID, used as the external ID in the Monad IAM role's trust policy.

---

Step 1 — Create the S3 Bucket

1. Create an S3 bucket in the AWS region of your choice, or pick an existing one.
2. Note the bucket name and region.
3. Choose a prefix for the Monad data, such as monad/okta. Use a separate prefix for each log type you plan to send, so Panther can apply a different schema to each.

---

Step 2 — Give Monad Write Access to the Bucket

Create an IAM role that Monad can assume to write files to the bucket. Follow the Requirements section of the Amazon S3 output page for the trust policy and permissions.

Scope the role's permissions to the bucket from Step 1.

---

Step 3 — Create the Panther S3 Source

In the Panther Console:

1. In the left-hand navigation, go to Log Sources and click Create New.
2. Select the AWS S3 Bucket tile.
3. Fill in the source details:
   • Name: a descriptive name, such as Monad - Okta.
   • AWS Account ID: the 12-digit ID of the account that owns the bucket.
   • Bucket Name: the bucket from Step 1.
   • KMS Key ARN: only if the bucket uses KMS encryption.
4. Click Configure Prefixes & Schemas and map the prefix from Step 1 to the schema for that log type. Leaving the prefix blank applies the schema to the whole bucket.
5. Create the IAM role Panther uses to read the bucket. Panther offers three options: launching a CloudFormation stack from the console, downloading a CloudFormation or Terraform template, or creating the role yourself.

See Panther's S3 source documentation for the full setup flow.

---

Step 4 — Send Bucket Notifications to Panther

Panther only ingests files it's notified about, so the bucket must publish object-created events to Panther. If your Panther role setup didn't create the notification topic for you, set it up manually:

1. Create an SNS topic in the same region as the bucket.
2. Update the topic's access policy so that:
   • S3 can publish events from your bucket to the topic.
   • Panther's AWS account can subscribe to the topic (sns:Subscribe).
3. Create a subscription on the topic:
   • Protocol: Amazon SQS.
   • Endpoint: arn:aws:sqs:<PantherRegion>:<PantherAccountId>:panther-input-data-notifications-queue.
   • Enable raw message delivery: leave this unchecked. Panther requires raw message delivery to be off.
4. On the bucket, create an event notification:
   • Event types: All object create events.
   • Prefix: the prefix from Step 1, so only Monad's files trigger notifications.
   • Destination: the SNS topic.

Panther's S3 source documentation has the exact topic policy and the values to use for your Panther deployment.

---

Step 5 — Configure the Monad S3 Output

Create an Amazon S3 output in Monad with these settings:

Setting	Recommended value	Why
AWS IAM Role ARN	The role from Step 2	Lets Monad write to the bucket.
S3 Bucket Name	The bucket from Step 1
AWS Region	The bucket's region
S3 Object Prefix	The prefix from Step 1, such as monad/okta	Must match the prefix mapped in Panther and the bucket notification filter.
Format	json
JSON Type	line	Writes one record per line, which Panther reads as individual events.
Compression Method	gzip	Panther reads gzip-compressed files, and compression reduces storage and transfer.
Partition Format	simple date	Any format works, because Panther reads every new file under the prefix.

The default batch settings are a good starting point. The output writes a file when a batch reaches 100,000 records, 10 MB of uncompressed data, or 45 seconds, whichever comes first. Lower Maximum Flush Interval if you want data to reach Panther sooner, at the cost of more, smaller files.

---

Step 6 — Connect and Verify

1. Add the S3 output to your pipeline and enable it.
2. Check that files start to appear in the bucket under your prefix. With the default settings, the first file arrives within about a minute once records flow.
3. In the Panther Console, open the S3 source and confirm it is receiving data, then search for the new events.

---

Troubleshooting

The Monad output reports an access denied error

Monad can't write to the bucket. Check the trust policy and permissions on the role from Step 2, and confirm the bucket name and region in the output match the bucket.

Files appear in the bucket but not in Panther

Panther isn't being notified about new files, or can't read them. Check that:

• The bucket event notification covers all object create events, and its prefix matches the Monad output's prefix.
• The SNS topic is subscribed to Panther's notifications queue, with raw message delivery turned off.
• The topic policy allows Panther's account to subscribe.
• Panther's IAM role can read objects in the bucket, and can use the KMS key if the bucket is encrypted.

Events land in Panther but fail to parse

The schema mapped to the prefix doesn't match the records. Make sure the output uses the json format with JSON Type line, and that the Panther schema matches the fields in your records.

---

Related Articles

• Amazon S3 output
• Panther HTTP output
• Panther: S3 source
• Panther: Data transports