# Working with Record Location

The `record_location` setting specifies where to find records within a JSON object fetched by an input connector. It uses [GJSON](https://github.com/tidwall/gjson) path syntax to traverse and extract data from nested JSON structures.

## When to Use

Use `record_location` when your JSON data wraps the actual records inside a nested structure. For example, if your JSON looks like this:

```json
{
  "metadata": { "timestamp": "2024-01-01T10:00:00Z", "version": "1.0" },
  "data": {
    "events": [
      { "id": 1, "type": "login" },
      { "id": 2, "type": "logout" }
    ]
  }
}
```

Setting `record_location` to `data.events` extracts the events array, and each element becomes an individual record in your pipeline.

If your records are already at the root level (either a single JSON object or a root-level array), you can leave `record_location` empty, or set it to `@this`.

## GJSON Path Syntax

Record Location uses [GJSON](https://github.com/tidwall/gjson) path syntax. Here's a quick reference:

### Dot Notation

Use dots to traverse nested objects:

```
data.events           → access the "events" key inside "data"
store.location.items  → access deeply nested keys
```

Given this JSON:

```json
{
  "data": {
    "events": [
      { "id": 1, "action": "login" },
      { "id": 2, "action": "logout" }
    ]
  }
}
```

| Path          | Result                                                          |
| ------------- | --------------------------------------------------------------- |
| `data`        | `{"events": [{"id": 1, ...}, {"id": 2, ...}]}`                  |
| `data.events` | `[{"id": 1, "action": "login"}, {"id": 2, "action": "logout"}]` |

### Array Access

Access array elements by index:

```
data.events.0         → first element of the events array
```

Using the same JSON above:

| Path            | Result                          |
| --------------- | ------------------------------- |
| `data.events.0` | `{"id": 1, "action": "login"}`  |
| `data.events.1` | `{"id": 2, "action": "logout"}` |

### Wildcards

Use `*` and `?` for pattern matching on keys:

```
child*.0              → matches keys starting with "child" (e.g., "children")
c?ildren.0            → matches single-character wildcard
```

Given this JSON:

```json
{
  "children_us": [{ "name": "Alice" }, { "name": "Bob" }],
  "children_eu": [{ "name": "Clara" }, { "name": "Dan" }],
  "pets": [{ "name": "Rex" }]
}
```

| Path            | Result                                                              |
| --------------- | ------------------------------------------------------------------- |
| `children*.0`   | `{"name": "Alice"}` (first match for keys starting with "children") |
| `children_??.0` | `{"name": "Alice"}` (matches "children_us" — two-char wildcard)     |

### Array Queries

Query arrays for matching elements

Given this JSON:

```json
{
  "alerts": [
    { "severity": "high", "source": "waf", "message": "SQL injection attempt" },
    { "severity": "low", "source": "ids", "message": "Port scan detected" },
    { "severity": "high", "source": "edr", "message": "Malware detected" },
    { "severity": "medium", "source": "waf", "message": "XSS attempt" }
  ]
}
```

| Path                               | Result                                                                                           |
| ---------------------------------- | ------------------------------------------------------------------------------------------------ |
| `alerts.#(severity=="high")`       | `{"severity": "high", "source": "waf", "message": "SQL injection attempt"}` (first match)        |
| `alerts.#(severity=="high")#`      | `[{"severity":"high","source":"waf",...}, {"severity":"high","source":"edr",...}]` (all matches) |
| `alerts.#(source=="waf")#.message` | `["SQL injection attempt", "XSS attempt"]`                                                       |

Note: `#(...)` returns the first match. `#(...)#` returns all matches.

### Array Iterator (`#.`)

Use `#.` to iterate over an array and extract a field from each element:

```
#.name                → extract "name" from each element in a root-level array
data.items.#.id       → extract "id" from each element in "data.items"
```

Given this root-level array:

```json
[
  {
    "id": "vm-1",
    "tags": ["prod", "us-east"],
    "owner": { "email": "alice@co.com" }
  },
  {
    "id": "vm-2",
    "tags": ["staging", "eu-west"],
    "owner": { "email": "bob@co.com" }
  }
]
```

| Path               | Result                                                          |
| ------------------ | --------------------------------------------------------------- |
| `#.id`             | `["vm-1", "vm-2"]`                                              |
| `#.owner.email`    | `["alice@co.com", "bob@co.com"]`                                |
| `#.tags`           | `[["prod","us-east"], ["staging","eu-west"]]` (array of arrays) |
| `#.tags\|@flatten` | `["prod", "us-east", "staging", "eu-west"]` (flattened)         |

### Modifiers

GJSON supports modifiers that transform results using the pipe (`|`) operator:

```
#.tags|@flatten       → extract "tags" arrays from each element, then flatten into one array
data|@reverse         → reverse an array
```

Using the same array above:

| Path               | Result                                      |
| ------------------ | ------------------------------------------- |
| `#.tags\|@flatten` | `["prod", "us-east", "staging", "eu-west"]` |
| `#.id\|@reverse`   | `["vm-2", "vm-1"]`                          |

### Escape Special Characters

Use `\` to escape dots in key names:

```
fav\.movie            → access the key "fav.movie" literally
```

Given this JSON:

```json
{
  "app.version": "2.1.0",
  "app": { "version": "1.0.0" }
}
```

| Path           | Result                                             |
| -------------- | -------------------------------------------------- |
| `app.version`  | `"1.0.0"` (traverses into `app` object)            |
| `app\.version` | `"2.1.0"` (accesses the literal key `app.version`) |

## Examples

### Object Storage Inputs (S3, GCS, Azure Blob, B2, etc.)

For object storage connectors, `record_location` applies only when the format is JSON.

**Flat array at root level** — leave empty or use `@this`:

```json
[
  { "id": 1, "type": "login" },
  { "id": 2, "type": "logout" }
]
```

**Records nested under a key** — use the path to the array:

```json
{
  "Records": [{ "eventSource": "s3.amazonaws.com", "eventName": "PutObject" }]
}
```

```
record_location = "Records"
```

**Deeply nested records:**

```json
{
  "store": {
    "location": {
      "transactions": {
        "daily": [
          { "id": 1, "amount": 100 },
          { "id": 2, "amount": 200 }
        ]
      }
    }
  }
}
```

```
record_location = "store.location.transactions.daily"
```

**Extracting a nested field from each element in a root-level array** — use `#.` to iterate:

```json
[
  {
    "apps": [{ "id": "1" }, { "id": "2" }],
    "cve": { "foo": { "id": "uuid-1" } }
  },
  {
    "apps": [{ "id": "3" }, { "id": "4" }],
    "cve": { "foo": { "id": "uuid-2" } }
  }
]
```

To stream each `cve.foo` object as an individual record:

```
record_location = "#.cve.foo"
```

This produces two records: `{"id": "uuid-1"}` and `{"id": "uuid-2"}`.

**Flattening nested arrays** — use `#.` with `|@flatten`:

Using the same data above, to stream each object inside the `apps` arrays as an individual record:

```
record_location = "#.apps|@flatten"
```

Without `@flatten`, `#.apps` would return an array of arrays — each element being the full `apps` array from one parent object. The `@flatten` modifier merges them into a single flat array, producing four individual records: `{"id": "1"}`, `{"id": "2"}`, `{"id": "3"}`, `{"id": "4"}`.

**Extracting a specific field from nested array elements** — combine `#.` with `|@flatten` and another `#.`:

```json
[
  {
    "findings": [
      { "severity": "high", "resource": { "arn": "arn:aws:s3:::bucket1" } },
      { "severity": "low", "resource": { "arn": "arn:aws:s3:::bucket2" } }
    ]
  },
  {
    "findings": [
      { "severity": "medium", "resource": { "arn": "arn:aws:ec2:::instance1" } }
    ]
  }
]
```

To stream each finding as an individual record:

```
record_location = "#.findings|@flatten"
```

This produces three records — one per finding across all parent objects.

### GraphQL Inputs

For GraphQL connectors, `record_location` points to the records array in the GraphQL response. This is typically required since GraphQL responses wrap data under a `data` key.

**Simple response:**

```json
{
  "data": {
    "users": [
      { "id": "1", "name": "Alice" },
      { "id": "2", "name": "Bob" }
    ]
  }
}
```

```
record_location = "data.users"
```

**Relay-style pagination with edges/nodes:**

```json
{
  "data": {
    "organization": {
      "pipelines": {
        "edges": [
          { "node": { "id": "1", "name": "deploy" } },
          { "node": { "id": "2", "name": "test" } }
        ]
      }
    }
  }
}
```

```
record_location = "data.organization.pipelines.edges"
```

## Quick Reference

| Value                    | Behavior                                                                     |
| ------------------------ | ---------------------------------------------------------------------------- |
| Empty string `""`        | Treats the entire JSON as a single record, or iterates a root-level array    |
| `@this`                  | Same as empty — refers to the root of the JSON                               |
| `@`                      | Same as `@this`                                                              |
| `data.events`            | Extract the `events` array from inside `data`                                |
| `#.cve.foo`              | Iterate a root-level array and extract `cve.foo` from each element           |
| `#.apps\|@flatten`       | Iterate a root-level array, extract `apps` arrays, and flatten into one list |
| `data.#.items\|@flatten` | Same pattern applied to a nested array under `data`                          |

## Try It Out

You can experiment with GJSON path syntax using the interactive playground at [gjson.dev](https://gjson.dev/).

For the full GJSON syntax reference, see the [GJSON documentation](https://github.com/tidwall/gjson#path-syntax).