Skip to main content

KV Lookup Output

Stores key-value pairs from pipeline records for use with the KV Lookup Enrichment.

Requirements​

  1. An active Monad subscription
  2. A pipeline with records containing a field suitable for use as a lookup key

Setting up​

The KV Lookup Output works in conjunction with the KV Lookup Enrichment to enable cross-pipeline data joins. Follow these steps to get started:

  1. Identify Your Use Case

    • Determine which data you want to make available for enrichment (e.g., user records, asset inventory, threat intelligence)
    • Identify the field that will serve as the unique lookup key (e.g., user_id, ip_address, hostname)

  2. Configure the KV Output

    • Create a new pipeline or modify an existing one
    • Add the KV Lookup Output as a destination
    • Configure the Key Field, Value Field, and TTL settings (see Configuration section below)

  3. Run the Pipeline

    • Execute the pipeline to populate the KV store
    • Records will be stored and available for lookup by other pipelines

  4. Configure the KV Lookup Enrichment

    • In a separate pipeline, add the KV Lookup Enrichment
    • Select this KV Output as the data source
    • Configure the Join Key and Destination Key to enrich incoming records

Functionality​

The KV Lookup Output processes and stores data using the following workflow:

  1. Receives processed events from the pipeline
  2. Extracts the key from each record using the configured Key Field
  3. Extracts the value using the configured Value Field (or stores the entire record if not specified)
  4. Stores the key-value pair with the configured TTL

Best Practices​

  • Key Selection:

    • Choose a field with unique, consistent values across records
    • Avoid fields that may be null or empty
    • Use stable identifiers (e.g., user_id rather than email which may change)

  • Value Field Usage:

    • Leave empty to store the entire record when you need full context during enrichment
    • Specify a field to store only that value when you need minimal storage footprint

  • TTL Configuration:

    • Set TTL based on how frequently your source data updates
    • Use shorter TTLs for frequently changing data to avoid stale lookups
    • Use longer TTLs for relatively static reference data

Limitations​

  • Minimum TTL: 5 seconds
  • Maximum TTL: 48 hours
  • The KV store must be populated before the KV Lookup Enrichment can retrieve data
  • No built-in UI for viewing stored key-value pairs (visibility improvements are planned)

Troubleshooting​

  • Enrichment Returns No Data:

  • Verify the KV Output pipeline has run successfully

  • Check that the Key Field exists and has values in your source records

  • Ensure the TTL has not expired for the records you're trying to look up

  • Key Not Found Errors:

  • Confirm the Join Key in the enrichment matches the Key Field values stored by the output

  • Check for data type mismatches (e.g., string vs. integer)

  • Verify the source pipeline is still running and populating the KV store

  • Stale Data:

  • Review your TTL settings and adjust if data is expiring too quickly

  • Ensure the source pipeline runs frequently enough to keep data fresh

Configuration​

The KV Lookup Output is configured using the following settings:

Settings​

SettingTypeRequiredDefaultDescription
Key FieldstringYes-The field in the incoming record to use as the lookup key in the KV store.
Value FieldstringNo-The field in the incoming record to store as the value. If not provided, the entire record will be stored.
TTLintNo300 (5 min)Time-to-live in seconds for stored key-value pairs. Minimum: 5 seconds, maximum: 48 hours (172,800 seconds).

Secrets​

This output type does not require any secrets.