Copy page

Transforms

JQ

Apply JQ queries to transform JSON records using the full power of JQ syntax.

Overview

The JQ transform allows you to filter, reshape, and manipulate JSON data using JQ query syntax. It supports both direct output of query results and storing results under specified keys in the original record.

Queries are executed by gojq v0.12.19, a pure-Go jq implementation, rather than by the jq binary. Its behavior differs from jq in a few documented ways, so check the gojq documentation if a query behaves differently here than on your command line.

Example

This example shows how to use the JQ transformation to filter and reshape JSON records using JQ query syntax.

Code
{
  "operation": "jq",
  "arguments": {
    "query": ".users[] | select(.type == \"active\") | {id, name, email: .contact.email}",
    "key": "active_users"
  }
}

Given the input record:

Code
{
  "users": [
    {
      "id": 1,
      "name": "Alice",
      "type": "active",
      "contact": {
        "email": "alice@example.com",
        "phone": "123-456-7890"
      },
      "metadata": {
        "last_login": "2024-01-01"
      }
    },
    {
      "id": 2,
      "name": "Bob",
      "type": "inactive",
      "contact": {
        "email": "bob@example.com",
        "phone": "123-456-7891"
      },
      "metadata": {
        "last_login": "2023-12-01"
      }
    }
  ]
}

With key specified, the output record will be:

Code
{
  "users": [...],
  "active_users": [
    {
      "id": 1,
      "name": "Alice",
      "email": "alice@example.com"
    }
  ]
}

Without key specified, the output will be just the query result:

Code
[
  {
    "id": 1,
    "name": "Alice",
    "email": "alice@example.com"
  }
]

Configuration

Field	Type	Required	Description
query	string	Yes	The JQ query to execute
key	string	No	Optional key to store the result under in the original record
prevent_data_dropping	boolean	No	Defaults to false. When true, the transform errors instead of dropping the record if the query produces no output. Has no effect when key is set

More Examples

1. Delete sensitive fields:

Code
{
  "operation": "jq",
  "arguments": {
    "query": "del(.password, .ssn)"
  }
}

2. Compute new fields and store under a key:

Code
{
  "operation": "jq",
  "arguments": {
    "query": ".items | map(.price) | add",
    "key": "total_price"
  }
}

3. Complex data reshaping:

Code
{
  "operation": "jq",
  "arguments": {
    "query": "{customer_info: {name: .name, email: .contact.email}, orders: [.orders[] | {id, total: (.items | map(.price) | add)}]}"
  }
}

4. Array aggregation with nested filtering:

Code
{
  "operation": "jq",
  "arguments": {
    "query": "[.departments[] | .employees[] | select(.role == \"developer\") | {name, department}]",
    "key": "developers"
  }
}

Output Behavior

1. Direct Output (no key specified):

   • Query must return a JSON object or array
   • Single results are returned directly
   • Multiple results are wrapped in an array
   • A query producing no output drops the record, which is how queries such as select(...) filter data. Set prevent_data_dropping to true to error instead
   • Non-object/array results (strings, numbers, booleans) will cause an error

2. Key Storage (key specified):

   • Query result is stored under the specified key
   • Original record structure is preserved
   • Supports all JSON value types (objects, arrays, strings, numbers, booleans, null)
   • Nested key paths are supported (e.g., "metadata.calculated.total")
   • Overwrites any existing value at the key path
   • A record is never dropped: a query producing no output writes null at the key. prevent_data_dropping therefore has no effect here

Error Handling

The transform will fail with an error in these cases:

1. Missing or empty query
2. Query fails to parse or compile
3. Query execution error
4. Direct output (no key) returns a non-object/array value
5. Invalid JSON in input record
6. Invalid JSON output from query
7. Direct output (no key) produces no output while prevent_data_dropping is true

Preventing Silent Drops

A jq query that produces no output for a record drops that record. This is intentional when the query is filtering, but it is easy to do by accident: a typo in a field name, or a select condition that never matches, silently removes data with no error.

Set prevent_data_dropping to true when the query is only meant to reshape records, never to filter them:

Code
{
  "operation": "jq",
  "arguments": {
    "query": "{id: .event_id, host: .host.name}",
    "prevent_data_dropping": true
  }
}

If a record reaches this transform and the query yields nothing, the pipeline node reports an error rather than dropping the record. Leave the field unset (or false) for queries that are meant to filter, such as select(.severity == "high").

Records that fail this check are retried, so a query that drops records consistently will keep the node in an erroring state until the query or the setting is corrected.

Common JQ Patterns

• Filter objects: select(.field == "value")
• Access nested fields: .parent.child.field
• Create objects: {new_field: .old_field}
• Array operations: map, add, length
• Delete fields: del(.field_name)
• String operations: split, join, ascii_downcase
• Mathematical operations: add, multiply

One Record In, Many Records Out

Without key, every value the query emits becomes its own output record. A query of .items[] against a record whose items array holds three elements produces three records, each continuing through the pipeline independently:

Code
{
  "operation": "jq",
  "arguments": {
    "query": ".items[]"
  }
}

Code
{ "items": [{ "id": 1 }, { "id": 2 }, { "id": 3 }] }

produces three records:

Code
{ "id": 1 }
{ "id": 2 }
{ "id": 3 }

This is the idiomatic way to split a batched payload — a vendor API that returns many events wrapped in one envelope — into individual records.

With key set, the record count never changes: a single result is written to that key as-is, and multiple results are written to it as an array.