JQ

Apply JQ queries to transform JSON records using the full power of JQ syntax.

Overview

The JQ transform allows you to filter, reshape, and manipulate JSON data using JQ query syntax. It supports both direct output of query results and storing results under specified keys in the original record.

Example

This example shows how to use the JQ transformation to filter and reshape JSON records using JQ query syntax.

{
  "operation": "jq",
  "arguments": {
    "query": ".users[] | select(.type == \"active\") | {id, name, email: .contact.email}",
    "key": "active_users"
  }
}

Given the input record:

{
  "users": [
    {
      "id": 1,
      "name": "Alice",
      "type": "active",
      "contact": {
        "email": "alice@example.com",
        "phone": "123-456-7890"
      },
      "metadata": {
        "last_login": "2024-01-01"
      }
    },
    {
      "id": 2,
      "name": "Bob",
      "type": "inactive",
      "contact": {
        "email": "bob@example.com",
        "phone": "123-456-7891"
      },
      "metadata": {
        "last_login": "2023-12-01"
      }
    }
  ]
}

With key specified, the output record will be:

{
  "users": [...],
  "active_users": [
    {
      "id": 1,
      "name": "Alice",
      "email": "alice@example.com"
    }
  ]
}

Without key specified, the output will be just the query result:

[
  {
    "id": 1,
    "name": "Alice",
    "email": "alice@example.com"
  }
]

Configuration

Field	Type	Required	Description
query	string	Yes	The JQ query to execute
key	string	No	Optional key to store the result under in the original record

More Examples

1. Delete sensitive fields:

{
  "operation": "jq",
  "arguments": {
    "query": "del(.password, .ssn)"
  }
}

2. Compute new fields and store under a key:

{
  "operation": "jq",
  "arguments": {
    "query": ".items | map(.price) | add",
    "key": "total_price"
  }
}

3. Complex data reshaping:

{
  "operation": "jq",
  "arguments": {
    "query": "{customer_info: {name: .name, email: .contact.email}, orders: [.orders[] | {id, total: (.items | map(.price) | add)}]}"
  }
}

4. Array aggregation with nested filtering:

{
  "operation": "jq",
  "arguments": {
    "query": "[.departments[] | .employees[] | select(.role == \"developer\") | {name, department}]",
    "key": "developers"
  }
}

Output Behavior

1. Direct Output (no key specified):

   • Query must return a JSON object or array
   • Single results are returned directly
   • Multiple results are wrapped in an array
   • Empty results return an empty array
   • Non-object/array results (strings, numbers, booleans) will cause an error

2. Key Storage (key specified):

   • Query result is stored under the specified key
   • Original record structure is preserved
   • Supports all JSON value types (objects, arrays, strings, numbers, booleans, null)
   • Nested key paths are supported (e.g., "metadata.calculated.total")
   • Overwrites any existing value at the key path

Error Handling

The transform will fail with an error in these cases:

1. Missing or empty query
2. Query fails to parse or compile
3. Query execution error
4. Direct output (no key) returns a non-object/array value
5. Invalid JSON in input record
6. Invalid JSON output from query

Common JQ Patterns

• Filter objects: select(.field == "value")
• Access nested fields: .parent.child.field
• Create objects: {new_field: .old_field}
• Array operations: map, add, length
• Delete fields: del(.field_name)
• String operations: split, join, ascii_downcase
• Mathematical operations: add, multiply