Copy page

Transforms

Convert CEF

Convert a CEF (Common Event Format) event from a field into structured output — header fields, extension key-values, column mapping with presets, and overflow collection.

Overview

The Convert CEF transformation parses a raw CEF line (e.g. CEF:0|Vendor|Product|1.0|100|Name|5|src=10.0.0.1 act=allow) held in a record field into a structured record. The seven CEF header slots become named fields, each key=value extension pair is mapped to an output column, and anything you haven't mapped is either collected into a single overflow field or emitted as an individual field.

• Presets — common_security_log (Microsoft Sentinel CommonSecurityLog column names) and cef_full_names (ArcSight dictionary names) supply a full column map, header field names, and a default list of numeric columns, so most configurations only need the preset. Your own columns entries are laid over the preset key by key.
• Output modes — replace the record with the parsed object, merge the parsed fields into the record, or nest the parsed object under a key.
• Lenient by default — every record produces output; structural problems (no CEF marker, truncated header, invalid escapes) are handled best-effort and optionally reported as warnings. Set strict: true to error instead.

Dialects

The transform parses two dialects, selected with the dialect argument:

• CEF (cef, default) — spec-faithful parsing. The CEF: marker is found anywhere in the line (a syslog prefix before it is skipped, and a space between CEF: and the version digit is tolerated). Escaped pipes (\|) in header fields do not delimit, and extension values are unescaped (\\, \=, \n, \r). Extension keys start with an alphanumeric or _ and may then contain _ . , - [ ]. Trailing whitespace is trimmed from the final extension value only.
• Legacy key-value (legacy_kv) — compatible with simpler line-oriented CEF parsers. A literal CEF: prefix is stripped only at the very start of the line, every pipe delimits a header field (escapes are ignored), no unescaping is performed anywhere, extension keys are strictly alphanumeric (starting with a letter), and the final value keeps its trailing whitespace. Use this dialect when you need output identical to an existing pipeline built on those rules.

In both dialects, a new extension pair starts only at a space immediately followed by a key and =. Anything else — unescaped = signs inside URLs, tabs, extra spaces — is absorbed into the previous value.

Presets

• common_security_log — 132 CEF key → column mappings (src → SourceIP, dst → DestinationIP, dpt → DestinationPort, act → DeviceAction, msg → Message, the full custom-string/number/IPv6/flex families → DeviceCustomString1…, and so on), Sentinel-style header names (CefVersion, DeviceVendor, DeviceProduct, DeviceVersion, DeviceEventClassID, Activity, LogSeverity), 22 numeric columns (ports, byte counts, process IDs, custom numbers), and a default overflow_field of AdditionalExtensions (applied only if you leave overflow_field empty). The preset targets the table's current columns: cn1-cn3 map to FieldDeviceCustomNumber1-FieldDeviceCustomNumber3 and externalId maps to ExtID, not the deprecated DeviceCustomNumber1-3 and ExternalID that Microsoft's older CEF mapping lists. ExtID is a string column, so non-numeric IDs such as ABC-123 pass through intact.
• cef_full_names — 117 mappings to ArcSight dictionary names (src → sourceAddress, dst → destinationAddress, spt → sourcePort, msg → message, rt → deviceReceiptTime, …), lower-camel header names (cefVersion, deviceVendor, deviceProduct, deviceVersion, deviceEventClassId, name, severity), and 17 numeric columns. This preset does not default the overflow field.
• No preset (default) — no column map (every extension key is unmapped), no numeric coercion, and the same lower-camel header names as cef_full_names.

Configuration

Argument	Type	Required	Default	Description
key	string	No	"message"	JSONPath to the record field holding the CEF line. A missing or null source is parsed as an empty line (see Notes); a non-string value is parsed from its raw JSON text with a non_string_source warning.
dialect	string	No	"cef"	Parsing dialect: "cef" (spec-faithful) or "legacy_kv" (see Dialects).
output_mode	string	No	"replace"	How the parsed object is written back: "replace", "merge", or "under_key" (see Output Behavior).
output_key	string	Yes (for under_key)	-	JSONPath where the parsed object is nested when output_mode is "under_key".
preset	string	No	-	Column-mapping preset: "common_security_log" or "cef_full_names". Supplies the column map, default header names, and default numeric columns.
columns	object	No	-	Map of CEF extension key → output column name. Merged over the preset map key by key, so individual entries override the preset. Keys absent from the merged map go to the overflow field (or become individual output fields when no overflow field is set).
header_fields	array	No	per preset	Output names for the 7 CEF header slots (version, device vendor, device product, device version, event class ID, name, severity). Must name exactly 7 fields; replaces the preset's names entirely.
envelope	object	No	-	Map of output field name → JSONPath into the input record. Each value is copied verbatim (any JSON type) from the original record into the output; a missing path yields null. Envelope fields are emitted first, in alphabetical order.
overflow_field	string	No (Yes for spill with a marker)	"AdditionalExtensions" with the common_security_log preset, otherwise -	Field collecting unmapped extension keys as key=value entries in encounter order. When configured it is always emitted — null if nothing overflowed. When empty, unmapped keys become individual output fields instead.
overflow_separator	string	No	";"	Separator placed between overflow entries.
continuation_marker	string	No	-	Marker that indicates a continuation inside a value, e.g. "#012" (rsyslog) or "#LF#". Empty disables continuation handling.
continuation_mode	string	No	"spill"	What to do at a continuation marker: "spill" (keep the value up to the first marker, move the tail to the overflow field), "newline" (replace every marker with a newline), or "keep" (leave values untouched). "spill" with a marker requires overflow_field.
strip_quotes	boolean	No	false	Remove one pair of surrounding double quotes from extension values that carry both. Applied before continuation handling and unescaping. Not part of the CEF standard.
empty_to_null	boolean	No	false	Write empty header, extension, and promoted-label values as null instead of "". For envelope fields, only a value that is exactly the JSON string "" becomes null.
resolve_labels	boolean	No	false	Promote custom-field label pairs: when both a slot (e.g. cs1) and its label (cs1Label) are present, emit an additional field named by the label text holding the slot value. The raw slot and label columns are always kept. Never overwrites an existing field (skipped with a label_collision warning).
numeric_columns	array	No	preset numeric list	Output column names whose values are coerced to JSON numbers. Leaving it unset uses the preset's numeric list; an explicit [] disables coercion entirely; an explicit list replaces the preset list. Header field names are eligible too. Values that do not parse as numbers become null (see Notes).
iso_date_columns	array	No	-	Output column names whose values of the exact shape YYYY/MM/DD HH:MM:SS are rewritten to YYYY-MM-DDTHH:MM:SSZ. Non-matching values are left for numeric/string handling.
strict	boolean	No	false	Fail on structural problems (missing source, no CEF marker, truncated header, invalid escapes, unmergeable record) instead of best-effort parsing with warnings.
warnings_key	string	No	-	Output field for parse warnings. When set, it is always emitted as an array of strings — [] when the record parsed cleanly. When empty, warnings are dropped.

Examples

Example 1: Sentinel-style parsing with a preset

Parse a firewall CEF line into CommonSecurityLog column names using the common_security_log preset. The preset supplies the column map, the header names, the numeric columns, and the AdditionalExtensions overflow field.

Code
{
  "operation": "convert_cef",
  "arguments": {
    "preset": "common_security_log"
  }
}

Given the input record:

Code
{
  "message": "CEF:0|Palo Alto Networks|PAN-OS|10.1|TRAFFIC|end|3|src=10.0.0.5 dst=203.0.113.7 dpt=443 act=allow PanOSApp=web-browsing"
}

The output record will be:

Code
{
  "CefVersion": "0",
  "DeviceVendor": "Palo Alto Networks",
  "DeviceProduct": "PAN-OS",
  "DeviceVersion": "10.1",
  "DeviceEventClassID": "TRAFFIC",
  "Activity": "end",
  "LogSeverity": "3",
  "AdditionalExtensions": "PanOSApp=web-browsing",
  "SourceIP": "10.0.0.5",
  "DestinationIP": "203.0.113.7",
  "DestinationPort": 443,
  "DeviceAction": "allow"
}

src, dst, dpt, and act are mapped by the preset; DestinationPort is on the preset's numeric list, so 443 is emitted as a number. PanOSApp is not in the preset map, so it lands in AdditionalExtensions. The default output_mode is replace, so the parsed object becomes the entire output record.

Example 2: Custom columns with overflow

Map only the keys you care about and collect everything else into a single field.

Code
{
  "operation": "convert_cef",
  "arguments": {
    "columns": {
      "src": "source_ip",
      "act": "action"
    },
    "overflow_field": "extra"
  }
}

Given the input record:

Code
{
  "message": "CEF:0|Vectra|X Series|1.0|threat|Port Scan|8|src=10.1.2.3 act=detected dvc=192.0.2.9 out=1024"
}

The output record will be:

Code
{
  "cefVersion": "0",
  "deviceVendor": "Vectra",
  "deviceProduct": "X Series",
  "deviceVersion": "1.0",
  "deviceEventClassId": "threat",
  "name": "Port Scan",
  "severity": "8",
  "extra": "dvc=192.0.2.9;out=1024",
  "source_ip": "10.1.2.3",
  "action": "detected"
}

With no preset, the header slots use the default names (cefVersion through severity) and no numeric coercion applies. The unmapped keys dvc and out are joined into extra as key=value entries, in the order they appear, separated by the default ";".

Output Behavior

1. replace (default): the parsed object becomes the entire output record; the original fields do not survive. Use envelope to carry specific fields from the original record into the output.
2. merge: the parsed fields are merged into the original record at the top level; on a name collision the parsed field wins. If the record is not a JSON object, lenient mode falls back to replace for that record and adds a merge_non_object warning; strict mode returns an error.
3. under_key: the parsed object is nested at output_key and the rest of the record is preserved. If the write is impossible (for example the record is a top-level array), lenient mode falls back to replace for that record; strict mode returns an error.

Output fields are emitted in a deterministic order: envelope fields (alphabetical) → the 7 header fields → the overflow field → mapped and individual columns in the order they were first encountered → the warnings array. If a mapped column shares a name with a header, envelope, or overflow field, the mapped column wins and the other slot is skipped.

Error Handling

With strict: true, the transform will fail with an error in these cases (in lenient mode the same conditions produce best-effort output and, where noted, a warning):

1. The source key is missing or null (lenient: parsed as an empty line)
2. The source key does not hold a string (lenient: warning non_string_source)
3. No CEF marker is found — cef dialect only (lenient: warning no_cef_marker)
4. The header has fewer than 7 fields — both dialects (lenient: warning incomplete_header, cef dialect only)
5. An extension value contains an invalid escape sequence, e.g. a Windows path like C:\Users\bob — cef dialect only (lenient: warning invalid_escape)
6. The record cannot be merged or written in merge/under_key mode (lenient: falls back to replace)

A strict error returns the original record unchanged and aborts the whole batch, so keep strict off unless a malformed record should stop the pipeline. Lenient mode never errors: every record produces output.

The full set of warnings that can appear under warnings_key: no_cef_marker, incomplete_header, invalid_escape, extension_garbage, non_string_source, merge_non_object, label_collision, invalid_utf8. Warnings are de-duplicated per record.

Notes

• A missing source is not a pass-through: when the source key is missing or null, the line is parsed as if it were empty, producing a skeleton output whose first header field is "" and remaining header fields are null (with empty_to_null, all seven are null) — the original record is not passed through unchanged. Use strict: true if you want missing sources to error instead.
• Numeric coercion is lenient: values in numeric_columns that cannot be parsed become null — never an error, and never the original string. Leading zeros (007 → 7), a leading +, and bare-dot forms (.5, 5.) are accepted; forms like 0x10, 1_0, Infinity, and NaN become null. Integers beyond 64-bit range are kept at exact precision.
• numeric_columns unset vs empty: omitting the argument uses the preset's numeric list; passing [] explicitly disables coercion; passing a list uses exactly that list. If a column is in both iso_date_columns and numeric_columns, the ISO rewrite wins when the value matches the date shape.
• iso_date_columns is textual: the rewrite is a position check on the exact YYYY/MM/DD HH:MM:SS shape with a Z appended — no calendar validation and no timezone logic.
• Duplicate extension keys: for mapped columns and individual fields, the last occurrence wins. In the overflow field, every occurrence is appended as its own entry.
• Continuation spill format: with continuation_mode: "spill", a mapped key's tail goes to the overflow field as key::tail (the tail includes the marker); an unmapped key's tail is appended as a separate bare entry with no key.
• Header slots that are absent (truncated header) are emitted as null. Empty header fields are legal and emitted as "" (or null with empty_to_null).
• Base output names must be unique: the 7 header names, envelope field names, overflow_field, and warnings_key must be non-empty and mutually distinct, or the configuration is rejected.
• One useful pattern with replace mode: use envelope to pull fields such as a timestamp or hostname from the original record into the parsed output, e.g. "envelope": {"EventTime": "timestamp", "Collector": "structured-data.collector.host"} — nested JSONPaths are supported and values are copied verbatim.