# Convert CEF

Convert a CEF (Common Event Format) event from a field into structured output — header fields, extension key-values, column mapping with presets, and overflow collection.

## Overview

The Convert CEF transformation parses a raw CEF line (e.g. `CEF:0|Vendor|Product|1.0|100|Name|5|src=10.0.0.1 act=allow`) held in a record field into a structured record. The seven CEF header slots become named fields, each `key=value` extension pair is mapped to an output column, and anything you haven't mapped is either collected into a single overflow field or emitted as an individual field.

- **Presets** — `common_security_log` (Microsoft Sentinel CommonSecurityLog column names) and `cef_full_names` (ArcSight dictionary names) supply a full column map, header field names, and a default list of numeric columns, so most configurations only need the preset. Your own `columns` entries are laid over the preset key by key.
- **Output modes** — replace the record with the parsed object, merge the parsed fields into the record, or nest the parsed object under a key.
- **Lenient by default** — every record produces output; structural problems (no CEF marker, truncated header, invalid escapes) are handled best-effort and optionally reported as warnings. Set `strict: true` to error instead.

## Dialects

The transform parses two dialects, selected with the `dialect` argument:

- **CEF** (`cef`, default) — spec-faithful parsing. The `CEF:` marker is found anywhere in the line (a syslog prefix before it is skipped, and a space between `CEF:` and the version digit is tolerated). Escaped pipes (`\|`) in header fields do not delimit, and extension values are unescaped (`\\`, `\=`, `\n`, `\r`). Extension keys start with an alphanumeric or `_` and may then contain `_ . , - [ ]`. Trailing whitespace is trimmed from the final extension value only.
- **Legacy key-value** (`legacy_kv`) — compatible with simpler line-oriented CEF parsers. A literal `CEF:` prefix is stripped only at the very start of the line, every pipe delimits a header field (escapes are ignored), no unescaping is performed anywhere, extension keys are strictly alphanumeric (starting with a letter), and the final value keeps its trailing whitespace. Use this dialect when you need output identical to an existing pipeline built on those rules.

In both dialects, a new extension pair starts only at a space immediately followed by a key and `=`. Anything else — unescaped `=` signs inside URLs, tabs, extra spaces — is absorbed into the previous value.

## Presets

- **`common_security_log`** — 132 CEF key → column mappings (`src` → `SourceIP`, `dst` → `DestinationIP`, `dpt` → `DestinationPort`, `act` → `DeviceAction`, `msg` → `Message`, the full custom-string/number/IPv6/flex families → `DeviceCustomString1`…, and so on), Sentinel-style header names (`CefVersion`, `DeviceVendor`, `DeviceProduct`, `DeviceVersion`, `DeviceEventClassID`, `Activity`, `LogSeverity`), 22 numeric columns (ports, byte counts, process IDs, custom numbers), and a default `overflow_field` of `AdditionalExtensions` (applied only if you leave `overflow_field` empty).
  The preset targets the table's current columns: `cn1`-`cn3` map to `FieldDeviceCustomNumber1`-`FieldDeviceCustomNumber3` and `externalId` maps to `ExtID`, not the deprecated `DeviceCustomNumber1`-`3` and `ExternalID` that Microsoft's older CEF mapping lists.
  `ExtID` is a string column, so non-numeric IDs such as `ABC-123` pass through intact.
- **`cef_full_names`** — 117 mappings to ArcSight dictionary names (`src` → `sourceAddress`, `dst` → `destinationAddress`, `spt` → `sourcePort`, `msg` → `message`, `rt` → `deviceReceiptTime`, …), lower-camel header names (`cefVersion`, `deviceVendor`, `deviceProduct`, `deviceVersion`, `deviceEventClassId`, `name`, `severity`), and 17 numeric columns. This preset does not default the overflow field.
- **No preset** (default) — no column map (every extension key is unmapped), no numeric coercion, and the same lower-camel header names as `cef_full_names`.

## Configuration

| Argument | Type | Required | Default | Description |
| -------- | ---- | -------- | ------- | ----------- |
| `key` | string | No | `"message"` | JSONPath to the record field holding the CEF line. A missing or `null` source is parsed as an empty line (see [Notes](#notes)); a non-string value is parsed from its raw JSON text with a `non_string_source` warning. |
| `dialect` | string | No | `"cef"` | Parsing dialect: `"cef"` (spec-faithful) or `"legacy_kv"` (see [Dialects](#dialects)). |
| `output_mode` | string | No | `"replace"` | How the parsed object is written back: `"replace"`, `"merge"`, or `"under_key"` (see [Output Behavior](#output-behavior)). |
| `output_key` | string | Yes (for `under_key`) | - | JSONPath where the parsed object is nested when `output_mode` is `"under_key"`. |
| `preset` | string | No | - | Column-mapping preset: `"common_security_log"` or `"cef_full_names"`. Supplies the column map, default header names, and default numeric columns. |
| `columns` | object | No | - | Map of CEF extension key → output column name. Merged over the preset map key by key, so individual entries override the preset. Keys absent from the merged map go to the overflow field (or become individual output fields when no overflow field is set). |
| `header_fields` | array | No | per preset | Output names for the 7 CEF header slots (version, device vendor, device product, device version, event class ID, name, severity). Must name exactly 7 fields; replaces the preset's names entirely. |
| `envelope` | object | No | - | Map of output field name → JSONPath into the input record. Each value is copied verbatim (any JSON type) from the original record into the output; a missing path yields `null`. Envelope fields are emitted first, in alphabetical order. |
| `overflow_field` | string | No (Yes for `spill` with a marker) | `"AdditionalExtensions"` with the `common_security_log` preset, otherwise - | Field collecting unmapped extension keys as `key=value` entries in encounter order. When configured it is always emitted — `null` if nothing overflowed. When empty, unmapped keys become individual output fields instead. |
| `overflow_separator` | string | No | `";"` | Separator placed between overflow entries. |
| `continuation_marker` | string | No | - | Marker that indicates a continuation inside a value, e.g. `"#012"` (rsyslog) or `"#LF#"`. Empty disables continuation handling. |
| `continuation_mode` | string | No | `"spill"` | What to do at a continuation marker: `"spill"` (keep the value up to the first marker, move the tail to the overflow field), `"newline"` (replace every marker with a newline), or `"keep"` (leave values untouched). `"spill"` with a marker requires `overflow_field`. |
| `strip_quotes` | boolean | No | `false` | Remove one pair of surrounding double quotes from extension values that carry both. Applied before continuation handling and unescaping. Not part of the CEF standard. |
| `empty_to_null` | boolean | No | `false` | Write empty header, extension, and promoted-label values as `null` instead of `""`. For envelope fields, only a value that is exactly the JSON string `""` becomes `null`. |
| `resolve_labels` | boolean | No | `false` | Promote custom-field label pairs: when both a slot (e.g. `cs1`) and its label (`cs1Label`) are present, emit an additional field named by the label text holding the slot value. The raw slot and label columns are always kept. Never overwrites an existing field (skipped with a `label_collision` warning). |
| `numeric_columns` | array | No | preset numeric list | Output column names whose values are coerced to JSON numbers. Leaving it unset uses the preset's numeric list; an explicit `[]` disables coercion entirely; an explicit list replaces the preset list. Header field names are eligible too. Values that do not parse as numbers become `null` (see [Notes](#notes)). |
| `iso_date_columns` | array | No | - | Output column names whose values of the exact shape `YYYY/MM/DD HH:MM:SS` are rewritten to `YYYY-MM-DDTHH:MM:SSZ`. Non-matching values are left for numeric/string handling. |
| `strict` | boolean | No | `false` | Fail on structural problems (missing source, no CEF marker, truncated header, invalid escapes, unmergeable record) instead of best-effort parsing with warnings. |
| `warnings_key` | string | No | - | Output field for parse warnings. When set, it is always emitted as an array of strings — `[]` when the record parsed cleanly. When empty, warnings are dropped. |

## Examples

### Example 1: Sentinel-style parsing with a preset

Parse a firewall CEF line into CommonSecurityLog column names using the `common_security_log` preset. The preset supplies the column map, the header names, the numeric columns, and the `AdditionalExtensions` overflow field.

```json
{
  "operation": "convert_cef",
  "arguments": {
    "preset": "common_security_log"
  }
}
```

Given the input record:

```json
{
  "message": "CEF:0|Palo Alto Networks|PAN-OS|10.1|TRAFFIC|end|3|src=10.0.0.5 dst=203.0.113.7 dpt=443 act=allow PanOSApp=web-browsing"
}
```

The output record will be:

```json
{
  "CefVersion": "0",
  "DeviceVendor": "Palo Alto Networks",
  "DeviceProduct": "PAN-OS",
  "DeviceVersion": "10.1",
  "DeviceEventClassID": "TRAFFIC",
  "Activity": "end",
  "LogSeverity": "3",
  "AdditionalExtensions": "PanOSApp=web-browsing",
  "SourceIP": "10.0.0.5",
  "DestinationIP": "203.0.113.7",
  "DestinationPort": 443,
  "DeviceAction": "allow"
}
```

`src`, `dst`, `dpt`, and `act` are mapped by the preset; `DestinationPort` is on the preset's numeric list, so `443` is emitted as a number. `PanOSApp` is not in the preset map, so it lands in `AdditionalExtensions`. The default `output_mode` is `replace`, so the parsed object becomes the entire output record.

### Example 2: Custom columns with overflow

Map only the keys you care about and collect everything else into a single field.

```json
{
  "operation": "convert_cef",
  "arguments": {
    "columns": {
      "src": "source_ip",
      "act": "action"
    },
    "overflow_field": "extra"
  }
}
```

Given the input record:

```json
{
  "message": "CEF:0|Vectra|X Series|1.0|threat|Port Scan|8|src=10.1.2.3 act=detected dvc=192.0.2.9 out=1024"
}
```

The output record will be:

```json
{
  "cefVersion": "0",
  "deviceVendor": "Vectra",
  "deviceProduct": "X Series",
  "deviceVersion": "1.0",
  "deviceEventClassId": "threat",
  "name": "Port Scan",
  "severity": "8",
  "extra": "dvc=192.0.2.9;out=1024",
  "source_ip": "10.1.2.3",
  "action": "detected"
}
```

With no preset, the header slots use the default names (`cefVersion` through `severity`) and no numeric coercion applies. The unmapped keys `dvc` and `out` are joined into `extra` as `key=value` entries, in the order they appear, separated by the default `";"`.

## Output Behavior

1. **`replace`** (default): the parsed object becomes the entire output record; the original fields do not survive. Use `envelope` to carry specific fields from the original record into the output.
2. **`merge`**: the parsed fields are merged into the original record at the top level; on a name collision the parsed field wins. If the record is not a JSON object, lenient mode falls back to `replace` for that record and adds a `merge_non_object` warning; strict mode returns an error.
3. **`under_key`**: the parsed object is nested at `output_key` and the rest of the record is preserved. If the write is impossible (for example the record is a top-level array), lenient mode falls back to `replace` for that record; strict mode returns an error.

Output fields are emitted in a deterministic order: envelope fields (alphabetical) → the 7 header fields → the overflow field → mapped and individual columns in the order they were first encountered → the warnings array. If a mapped column shares a name with a header, envelope, or overflow field, the mapped column wins and the other slot is skipped.

## Error Handling

With `strict: true`, the transform will fail with an error in these cases (in lenient mode the same conditions produce best-effort output and, where noted, a warning):

1. The source key is missing or `null` (lenient: parsed as an empty line)
2. The source key does not hold a string (lenient: warning `non_string_source`)
3. No CEF marker is found — `cef` dialect only (lenient: warning `no_cef_marker`)
4. The header has fewer than 7 fields — both dialects (lenient: warning `incomplete_header`, `cef` dialect only)
5. An extension value contains an invalid escape sequence, e.g. a Windows path like `C:\Users\bob` — `cef` dialect only (lenient: warning `invalid_escape`)
6. The record cannot be merged or written in `merge`/`under_key` mode (lenient: falls back to `replace`)

A strict error returns the original record unchanged and aborts the whole batch, so keep `strict` off unless a malformed record should stop the pipeline. Lenient mode never errors: every record produces output.

The full set of warnings that can appear under `warnings_key`: `no_cef_marker`, `incomplete_header`, `invalid_escape`, `extension_garbage`, `non_string_source`, `merge_non_object`, `label_collision`, `invalid_utf8`. Warnings are de-duplicated per record.

## Notes

- **A missing source is not a pass-through**: when the source key is missing or `null`, the line is parsed as if it were empty, producing a skeleton output whose first header field is `""` and remaining header fields are `null` (with `empty_to_null`, all seven are `null`) — the original record is not passed through unchanged. Use `strict: true` if you want missing sources to error instead.
- **Numeric coercion is lenient**: values in `numeric_columns` that cannot be parsed become `null` — never an error, and never the original string. Leading zeros (`007` → `7`), a leading `+`, and bare-dot forms (`.5`, `5.`) are accepted; forms like `0x10`, `1_0`, `Infinity`, and `NaN` become `null`. Integers beyond 64-bit range are kept at exact precision.
- **`numeric_columns` unset vs empty**: omitting the argument uses the preset's numeric list; passing `[]` explicitly disables coercion; passing a list uses exactly that list. If a column is in both `iso_date_columns` and `numeric_columns`, the ISO rewrite wins when the value matches the date shape.
- **`iso_date_columns` is textual**: the rewrite is a position check on the exact `YYYY/MM/DD HH:MM:SS` shape with a `Z` appended — no calendar validation and no timezone logic.
- **Duplicate extension keys**: for mapped columns and individual fields, the last occurrence wins. In the overflow field, every occurrence is appended as its own entry.
- **Continuation spill format**: with `continuation_mode: "spill"`, a mapped key's tail goes to the overflow field as `key::tail` (the tail includes the marker); an unmapped key's tail is appended as a separate bare entry with no key.
- **Header slots that are absent** (truncated header) are emitted as `null`. Empty header fields are legal and emitted as `""` (or `null` with `empty_to_null`).
- **Base output names must be unique**: the 7 header names, envelope field names, `overflow_field`, and `warnings_key` must be non-empty and mutually distinct, or the configuration is rejected.
- One useful pattern with `replace` mode: use `envelope` to pull fields such as a timestamp or hostname from the original record into the parsed output, e.g. `"envelope": {"EventTime": "timestamp", "Collector": "structured-data.collector.host"}` — nested JSONPaths are supported and values are copied verbatim.