Skip to main content

Working with Record Location

The record_location setting specifies where to find records within a JSON object fetched by an input connector. It uses GJSON path syntax to traverse and extract data from nested JSON structures.

When to Use​

Use record_location when your JSON data wraps the actual records inside a nested structure. For example, if your JSON looks like this:

{
  "metadata": { "timestamp": "2024-01-01T10:00:00Z", "version": "1.0" },
  "data": {
    "events": [
      { "id": 1, "type": "login" },
      { "id": 2, "type": "logout" }
    ]
  }
}

Setting record_location to data.events extracts the events array, and each element becomes an individual record in your pipeline.

If your records are already at the root level (either a single JSON object or a root-level array), you can leave record_location empty, or set it to @this.

GJSON Path Syntax​

Record Location uses GJSON path syntax. Here's a quick reference:

Dot Notation​

Use dots to traverse nested objects:

data.events           → access the "events" key inside "data"
store.location.items  → access deeply nested keys

Given this JSON:

{
  "data": {
    "events": [
      { "id": 1, "action": "login" },
      { "id": 2, "action": "logout" }
    ]
  }
}
PathResult
data{"events": [{"id": 1, ...}, {"id": 2, ...}]}
data.events[{"id": 1, "action": "login"}, {"id": 2, "action": "logout"}]

Array Access​

Access array elements by index:

data.events.0         → first element of the events array

Using the same JSON above:

PathResult
data.events.0{"id": 1, "action": "login"}
data.events.1{"id": 2, "action": "logout"}

Wildcards​

Use * and ? for pattern matching on keys:

child*.0              → matches keys starting with "child" (e.g., "children")
c?ildren.0            → matches single-character wildcard

Given this JSON:

{
  "children_us": [{ "name": "Alice" }, { "name": "Bob" }],
  "children_eu": [{ "name": "Clara" }, { "name": "Dan" }],
  "pets": [{ "name": "Rex" }]
}
PathResult
children*.0{"name": "Alice"} (first match for keys starting with "children")
children_??.0{"name": "Alice"} (matches "children_us" — two-char wildcard)

Array Queries​

Query arrays for matching elements

Given this JSON:

{
  "alerts": [
    { "severity": "high", "source": "waf", "message": "SQL injection attempt" },
    { "severity": "low", "source": "ids", "message": "Port scan detected" },
    { "severity": "high", "source": "edr", "message": "Malware detected" },
    { "severity": "medium", "source": "waf", "message": "XSS attempt" }
  ]
}
PathResult
alerts.#(severity=="high"){"severity": "high", "source": "waf", "message": "SQL injection attempt"} (first match)
alerts.#(severity=="high")#[{"severity":"high","source":"waf",...}, {"severity":"high","source":"edr",...}] (all matches)
alerts.#(source=="waf")#.message["SQL injection attempt", "XSS attempt"]

Note: #(...) returns the first match. #(...)# returns all matches.

Array Iterator (#.)​

Use #. to iterate over an array and extract a field from each element:

#.name                → extract "name" from each element in a root-level array
data.items.#.id       → extract "id" from each element in "data.items"

Given this root-level array:

[
  {
    "id": "vm-1",
    "tags": ["prod", "us-east"],
    "owner": { "email": "alice@co.com" }
  },
  {
    "id": "vm-2",
    "tags": ["staging", "eu-west"],
    "owner": { "email": "bob@co.com" }
  }
]
PathResult
#.id["vm-1", "vm-2"]
#.owner.email["alice@co.com", "bob@co.com"]
#.tags[["prod","us-east"], ["staging","eu-west"]] (array of arrays)
#.tags|@flatten["prod", "us-east", "staging", "eu-west"] (flattened)

Modifiers​

GJSON supports modifiers that transform results using the pipe (|) operator:

#.tags|@flatten       → extract "tags" arrays from each element, then flatten into one array
data|@reverse         → reverse an array

Using the same array above:

PathResult
#.tags|@flatten["prod", "us-east", "staging", "eu-west"]
#.id|@reverse["vm-2", "vm-1"]

Escape Special Characters​

Use \ to escape dots in key names:

fav\.movie            → access the key "fav.movie" literally

Given this JSON:

{
  "app.version": "2.1.0",
  "app": { "version": "1.0.0" }
}
PathResult
app.version"1.0.0" (traverses into app object)
app\.version"2.1.0" (accesses the literal key app.version)

Examples​

Object Storage Inputs (S3, GCS, Azure Blob, B2, etc.)​

For object storage connectors, record_location applies only when the format is JSON.

Flat array at root level — leave empty or use @this:

[
  { "id": 1, "type": "login" },
  { "id": 2, "type": "logout" }
]

Records nested under a key — use the path to the array:

{
  "Records": [{ "eventSource": "s3.amazonaws.com", "eventName": "PutObject" }]
}
record_location = "Records"

Deeply nested records:

{
  "store": {
    "location": {
      "transactions": {
        "daily": [
          { "id": 1, "amount": 100 },
          { "id": 2, "amount": 200 }
        ]
      }
    }
  }
}
record_location = "store.location.transactions.daily"

Extracting a nested field from each element in a root-level array — use #. to iterate:

[
  {
    "apps": [{ "id": "1" }, { "id": "2" }],
    "cve": { "foo": { "id": "uuid-1" } }
  },
  {
    "apps": [{ "id": "3" }, { "id": "4" }],
    "cve": { "foo": { "id": "uuid-2" } }
  }
]

To stream each cve.foo object as an individual record:

record_location = "#.cve.foo"

This produces two records: {"id": "uuid-1"} and {"id": "uuid-2"}.

Flattening nested arrays — use #. with |@flatten:

Using the same data above, to stream each object inside the apps arrays as an individual record:

record_location = "#.apps|@flatten"

Without @flatten, #.apps would return an array of arrays — each element being the full apps array from one parent object. The @flatten modifier merges them into a single flat array, producing four individual records: {"id": "1"}, {"id": "2"}, {"id": "3"}, {"id": "4"}.

Extracting a specific field from nested array elements — combine #. with |@flatten and another #.:

[
  {
    "findings": [
      { "severity": "high", "resource": { "arn": "arn:aws:s3:::bucket1" } },
      { "severity": "low", "resource": { "arn": "arn:aws:s3:::bucket2" } }
    ]
  },
  {
    "findings": [
      { "severity": "medium", "resource": { "arn": "arn:aws:ec2:::instance1" } }
    ]
  }
]

To stream each finding as an individual record:

record_location = "#.findings|@flatten"

This produces three records — one per finding across all parent objects.

GraphQL Inputs​

For GraphQL connectors, record_location points to the records array in the GraphQL response. This is typically required since GraphQL responses wrap data under a data key.

Simple response:

{
  "data": {
    "users": [
      { "id": "1", "name": "Alice" },
      { "id": "2", "name": "Bob" }
    ]
  }
}
record_location = "data.users"

Relay-style pagination with edges/nodes:

{
  "data": {
    "organization": {
      "pipelines": {
        "edges": [
          { "node": { "id": "1", "name": "deploy" } },
          { "node": { "id": "2", "name": "test" } }
        ]
      }
    }
  }
}
record_location = "data.organization.pipelines.edges"

Quick Reference​

ValueBehavior
Empty string ""Treats the entire JSON as a single record, or iterates a root-level array
@thisSame as empty — refers to the root of the JSON
@Same as @this
data.eventsExtract the events array from inside data
#.cve.fooIterate a root-level array and extract cve.foo from each element
#.apps|@flattenIterate a root-level array, extract apps arrays, and flatten into one list
data.#.items|@flattenSame pattern applied to a nested array under data

Try It Out​

You can experiment with GJSON path syntax using the interactive playground at gjson.dev.

For the full GJSON syntax reference, see the GJSON documentation.