Copy page

User Guides

Working with Record Location

The record_location setting specifies where to find records within a JSON object fetched by an input connector. It uses GJSON path syntax to traverse and extract data from nested JSON structures.

When to Use

Use record_location when your JSON data wraps the actual records inside a nested structure. For example, if your JSON looks like this:

Code
{
  "metadata": { "timestamp": "2024-01-01T10:00:00Z", "version": "1.0" },
  "data": {
    "events": [
      { "id": 1, "type": "login" },
      { "id": 2, "type": "logout" }
    ]
  }
}

Setting record_location to data.events extracts the events array, and each element becomes an individual record in your pipeline.

If your records are already at the root level (either a single JSON object or a root-level array), you can leave record_location empty, or set it to @this.

GJSON Path Syntax

Record Location uses GJSON path syntax. Here's a quick reference:

Dot Notation

Use dots to traverse nested objects:

Code
data.events           → access the "events" key inside "data"
store.location.items  → access deeply nested keys

Given this JSON:

Code
{
  "data": {
    "events": [
      { "id": 1, "action": "login" },
      { "id": 2, "action": "logout" }
    ]
  }
}

Path	Result
data	{"events": [{"id": 1, ...}, {"id": 2, ...}]}
data.events	[{"id": 1, "action": "login"}, {"id": 2, "action": "logout"}]

Array Access

Access array elements by index:

Code
data.events.0         → first element of the events array

Using the same JSON above:

Path	Result
data.events.0	{"id": 1, "action": "login"}
data.events.1	{"id": 2, "action": "logout"}

Wildcards

Use * and ? for pattern matching on keys:

Code
child*.0              → matches keys starting with "child" (e.g., "children")
c?ildren.0            → matches single-character wildcard

Given this JSON:

Code
{
  "children_us": [{ "name": "Alice" }, { "name": "Bob" }],
  "children_eu": [{ "name": "Clara" }, { "name": "Dan" }],
  "pets": [{ "name": "Rex" }]
}

Path	Result
children*.0	{"name": "Alice"} (first match for keys starting with "children")
children_??.0	{"name": "Alice"} (matches "children_us" — two-char wildcard)

Array Queries

Query arrays for matching elements

Given this JSON:

Code
{
  "alerts": [
    { "severity": "high", "source": "waf", "message": "SQL injection attempt" },
    { "severity": "low", "source": "ids", "message": "Port scan detected" },
    { "severity": "high", "source": "edr", "message": "Malware detected" },
    { "severity": "medium", "source": "waf", "message": "XSS attempt" }
  ]
}

Path	Result
alerts.#(severity=="high")	{"severity": "high", "source": "waf", "message": "SQL injection attempt"} (first match)
alerts.#(severity=="high")#	[{"severity":"high","source":"waf",...}, {"severity":"high","source":"edr",...}] (all matches)
alerts.#(source=="waf")#.message	["SQL injection attempt", "XSS attempt"]

Note: #(...) returns the first match. #(...)# returns all matches.

Array Iterator (#.)

Use #. to iterate over an array and extract a field from each element:

Code
#.name                → extract "name" from each element in a root-level array
data.items.#.id       → extract "id" from each element in "data.items"

Given this root-level array:

Code
[
  {
    "id": "vm-1",
    "tags": ["prod", "us-east"],
    "owner": { "email": "alice@co.com" }
  },
  {
    "id": "vm-2",
    "tags": ["staging", "eu-west"],
    "owner": { "email": "bob@co.com" }
  }
]

Path	Result
#.id	["vm-1", "vm-2"]
#.owner.email	["alice@co.com", "bob@co.com"]
#.tags	[["prod","us-east"], ["staging","eu-west"]] (array of arrays)
#.tags|@flatten	["prod", "us-east", "staging", "eu-west"] (flattened)

Modifiers

GJSON supports modifiers that transform results using the pipe (|) operator:

Code
#.tags|@flatten       → extract "tags" arrays from each element, then flatten into one array
data|@reverse         → reverse an array

Using the same array above:

Path	Result
#.tags|@flatten	["prod", "us-east", "staging", "eu-west"]
#.id|@reverse	["vm-2", "vm-1"]

Escape Special Characters

Use \ to escape dots in key names:

Code
fav\.movie            → access the key "fav.movie" literally

Given this JSON:

Code
{
  "app.version": "2.1.0",
  "app": { "version": "1.0.0" }
}

Path	Result
app.version	"1.0.0" (traverses into app object)
app\.version	"2.1.0" (accesses the literal key app.version)

Examples

Object Storage Inputs (S3, GCS, Azure Blob, B2, etc.)

For object storage connectors, record_location applies only when the format is JSON.

Flat array at root level — leave empty or use @this:

Code
[
  { "id": 1, "type": "login" },
  { "id": 2, "type": "logout" }
]

Records nested under a key — use the path to the array:

Code
{
  "Records": [{ "eventSource": "s3.amazonaws.com", "eventName": "PutObject" }]
}

Code
record_location = "Records"

Deeply nested records:

Code
{
  "store": {
    "location": {
      "transactions": {
        "daily": [
          { "id": 1, "amount": 100 },
          { "id": 2, "amount": 200 }
        ]
      }
    }
  }
}

Code
record_location = "store.location.transactions.daily"

Extracting a nested field from each element in a root-level array — use #. to iterate:

Code
[
  {
    "apps": [{ "id": "1" }, { "id": "2" }],
    "cve": { "foo": { "id": "uuid-1" } }
  },
  {
    "apps": [{ "id": "3" }, { "id": "4" }],
    "cve": { "foo": { "id": "uuid-2" } }
  }
]

To stream each cve.foo object as an individual record:

Code
record_location = "#.cve.foo"

This produces two records: {"id": "uuid-1"} and {"id": "uuid-2"}.

Flattening nested arrays — use #. with |@flatten:

Using the same data above, to stream each object inside the apps arrays as an individual record:

Code
record_location = "#.apps|@flatten"

Without @flatten, #.apps would return an array of arrays — each element being the full apps array from one parent object. The @flatten modifier merges them into a single flat array, producing four individual records: {"id": "1"}, {"id": "2"}, {"id": "3"}, {"id": "4"}.

Extracting a specific field from nested array elements — combine #. with |@flatten and another #.:

Code
[
  {
    "findings": [
      { "severity": "high", "resource": { "arn": "arn:aws:s3:::bucket1" } },
      { "severity": "low", "resource": { "arn": "arn:aws:s3:::bucket2" } }
    ]
  },
  {
    "findings": [
      { "severity": "medium", "resource": { "arn": "arn:aws:ec2:::instance1" } }
    ]
  }
]

To stream each finding as an individual record:

Code
record_location = "#.findings|@flatten"

This produces three records — one per finding across all parent objects.

GraphQL Inputs

For GraphQL connectors, record_location points to the records array in the GraphQL response. This is typically required since GraphQL responses wrap data under a data key.

Simple response:

Code
{
  "data": {
    "users": [
      { "id": "1", "name": "Alice" },
      { "id": "2", "name": "Bob" }
    ]
  }
}

Code
record_location = "data.users"

Relay-style pagination with edges/nodes:

Code
{
  "data": {
    "organization": {
      "pipelines": {
        "edges": [
          { "node": { "id": "1", "name": "deploy" } },
          { "node": { "id": "2", "name": "test" } }
        ]
      }
    }
  }
}

Code
record_location = "data.organization.pipelines.edges"

Quick Reference

Value	Behavior
Empty string ""	Treats the entire JSON as a single record, or iterates a root-level array
@this	Same as empty — refers to the root of the JSON
@	Same as @this
data.events	Extract the events array from inside data
#.cve.foo	Iterate a root-level array and extract cve.foo from each element
#.apps|@flatten	Iterate a root-level array, extract apps arrays, and flatten into one list
data.#.items|@flatten	Same pattern applied to a nested array under data

Try It Out

You can experiment with GJSON path syntax using the interactive playground at gjson.dev.

For the full GJSON syntax reference, see the GJSON documentation.