Skip to main content

KV Lookup

Description​

Enriches your records by joining them with data stored in a Monad KV Lookup output, allowing you to augment incoming security events with previously stored contextual information.

Details​

The KV Lookup enrichment enables you to enrich your security data by associating incoming records with data stored in a KV Lookup output. This enrichment performs a join operation using a specified key from your incoming record, looks up corresponding values in your KV store, and adds this enrichment data to your record at a configurable destination path.

This capability is particularly useful for:

  • Adding context from reference data (e.g., asset information, user details)
  • Correlating events with historical data
  • Implementing lookup tables for threat intelligence
  • Building relationships between different data sources

How It Works​

When a record passes through this enrichment:

  1. The enrichment extracts the value from the specified Join Key in your incoming record
  2. This value is hashed to create a lookup key
  3. The enrichment checks its cache for the key (improving performance for repeated lookups)
  4. If not found in cache, it queries the KV store using the hashed key
  5. When a matching entry is found, its value is added to your record at the specified Destination Key
  6. If no matching entry exists, the behavior depends on your Error on Missing Key setting

The enrichment maintains an in-memory cache with a 10-second TTL to optimize performance for frequently accessed keys.

Data format​

The resulting lookup data is added as a key in an object in the existing record. The data itself is stored in the data object in the lookup response struct.

{
  "foo": "bar",
  "enrichments.my_enrichment_name": {
    "join_key": "my_join_key",
    "message": "key found",
    "code": "success",
    "data": {
      "name": "example lookup data",
      "additional looup data": "stored in this object"
    }
  }
}

Configuration​

The following configuration defines the enrichment parameters:

Settings​

SettingTypeRequiredDescription
KV Lookup Output IDstringYesThe ID of the KV lookup output to join with. This must reference an existing KV Lookup output in your Monad workspace.
Join KeystringYesThe key in the record whose value will be used for lookups (GJSON path syntax). This value is hashed before lookup.
Destination KeystringYesThe path where the result will be stored in the record. Use '.' for the root object, 'field.subfield' for nested properties, 'array[0]' for array elements, or combined paths like 'users[0].name'.
Error on Missing KeybooleanNoIf true, the enrichment will throw an error when the lookup key is not found in the KV store. If false, the record is passed through unchanged when no matching key exists. Default is false.

Use Cases​

Asset Enrichment​

Enrich security alerts with comprehensive asset information by joining with asset data stored in a KV Lookup output.

Example configuration:

{
  "settings": {
    "kv_lookup_output_id": "asset-inventory",
    "join_key": "asset.id",
    "destination_key": "enrichment.asset_details",
    "error_on_missing_key": false
  }
}

User Context​

Add user information to authentication events by looking up user details from a KV store.

Example configuration:

{
  "settings": {
    "kv_lookup_output_id": "user-directory",
    "join_key": "user.email",
    "destination_key": "enrichment.user_context",
    "error_on_missing_key": false
  }
}

Threat Intelligence​

Correlate network events with known threat indicators by joining with threat intelligence data.

Example configuration:

{
  "settings": {
    "kv_lookup_output_id": "threat-intel",
    "join_key": "network.src_ip",
    "destination_key": "enrichment.threat_intel",
    "error_on_missing_key": false
  }
}

Troubleshooting​

Missing Join Key​

If your Join Key doesn't exist in the incoming record or is empty, the enrichment will either:

  • Log a warning and continue (when Error on Missing Key is false)
  • Generate an error that may block pipeline processing (when Error on Missing Key is true)

KV Store Initialization​

If you encounter an error like "failed to get KV store," ensure that:

  • The specified KV Lookup Output ID exists and is correctly referenced
  • The KV Lookup Output has successfully pulled data at least once
  • Your Monad configuration has appropriate permissions

Performance Considerations​

  • The enrichment uses an internal cache to optimize performance
  • Consider the memory impact for very large KV stores
  • For extremely high-volume pipelines, monitor system performance
  • KV Lookup Output: Required to store the data used for enrichment
  • JQ Transform: Can be used before or after the KV Lookup to prepare keys or process enriched data