Copy page

Transforms

Deduplicate

Drop duplicate records within a time window, keyed on the whole record or selected fields.

Overview

The Deduplicate transformation removes records it has already seen within a rolling time window, keeping the first occurrence of each key and dropping later repeats. It keys on either the entire record or a chosen subset of fields.

• Whole record (no fields) — the record is canonicalized first (object keys sorted recursively), so records that differ only in key order are treated as identical.
• Selected fields (fields set) — records are considered duplicates when the chosen field values match, regardless of the rest of the record.

Unlike every other transform, Deduplicate compares across records over a short window rather than acting on one record in isolation. Because of that it must be either the only operation in a transform, or the last one — configuring an operation after it is rejected.

Configuration

Argument	Type	Required	Default	Description
window	string	Yes	1m	Deduplication window: 1m, 2m, 3m, 4m, or 5m
fields	array of strings	No	[]	Field paths to key on; empty keys on the whole record

Example

Drop whole-record duplicates seen within one minute:

Code
{
  "operation": "deduplicate",
  "arguments": {
    "window": "1m"
  }
}

Given three records arriving within the window:

Code
{"user": "alice", "action": "login"}
{"user": "alice", "action": "login"}
{"user": "bob", "action": "login"}

The second is dropped as a duplicate; the first and third pass through:

Code
{"user": "alice", "action": "login"}
{"user": "bob", "action": "login"}

To deduplicate on specific fields instead of the whole record — for example, one event per user per window regardless of other fields — set fields:

Code
{
  "operation": "deduplicate",
  "arguments": {
    "window": "5m",
    "fields": ["user"]
  }
}

Notes

1. Deduplicate must be the only operation in a transform, or the last one. An operation placed after it fails validation with ... must be the last operation in the transform.
2. The first record for a key is kept; later matches within the window are dropped. The window is fixed from first sight and does not slide as duplicates arrive.
3. Sizing the window: Deduplicate tracks a bounded number of distinct keys — 5 million by default — across the active window. If your record rate multiplied by the window exceeds that budget, it stops recording new keys until the window drains: known duplicates are still dropped, but new ones may slip through until there is room again. Pick the smallest window that still covers the gap you expect between duplicates so you stay comfortably under the budget at your volume.
4. Whole-record keying is key-order independent (the record is canonicalized first).
5. If a configured key field is wholly absent from a record, that record is passed through rather than dropped, so distinct records are never collapsed onto an empty key. (A field present as null or "" still counts as present.)
6. Deduplication is best-effort and fails open: if the backing store is briefly unavailable, records pass through rather than being dropped, so data is never lost — at most a duplicate slips through.
7. Deduplication is scoped to the node: records are compared against others flowing through the same transform node within the window.