# Deduplicate

Drop duplicate records within a time window, keyed on the whole record or selected fields.

## Overview

The Deduplicate transformation removes records it has already seen within a rolling time window,
keeping the first occurrence of each key and dropping later repeats. It keys on either the entire
record or a chosen subset of fields.

- **Whole record** (no `fields`) — the record is canonicalized first (object keys sorted recursively),
  so records that differ only in key order are treated as identical.
- **Selected fields** (`fields` set) — records are considered duplicates when the chosen field values
  match, regardless of the rest of the record.

Unlike every other transform, Deduplicate compares **across** records over a short window rather than
acting on one record in isolation. Because of that it must be either the **only operation** in a
transform, or the **last** one — configuring an operation after it is rejected.

## Configuration

| Argument | Type | Required | Default | Description |
| -------- | ---- | -------- | ------- | ----------- |
| window | string | Yes | `1m` | Deduplication window: `1m`, `2m`, `3m`, `4m`, or `5m` |
| fields | array of strings | No | `[]` | Field paths to key on; empty keys on the whole record |

## Example

Drop whole-record duplicates seen within one minute:

```json
{
  "operation": "deduplicate",
  "arguments": {
    "window": "1m"
  }
}
```

Given three records arriving within the window:

```json
{"user": "alice", "action": "login"}
{"user": "alice", "action": "login"}
{"user": "bob", "action": "login"}
```

The second is dropped as a duplicate; the first and third pass through:

```json
{"user": "alice", "action": "login"}
{"user": "bob", "action": "login"}
```

To deduplicate on specific fields instead of the whole record — for example, one event per user per
window regardless of other fields — set `fields`:

```json
{
  "operation": "deduplicate",
  "arguments": {
    "window": "5m",
    "fields": ["user"]
  }
}
```

## Notes

1. Deduplicate must be the only operation in a transform, or the last one. An operation placed after
   it fails validation with `... must be the last operation in the transform`.
2. The first record for a key is kept; later matches within the window are dropped. The window is
   fixed from first sight and does not slide as duplicates arrive.
3. **Sizing the window:** Deduplicate tracks a bounded number of distinct keys — 5 million by default
   — across the active window. If your record rate multiplied by the window exceeds that budget, it
   stops recording *new* keys until the window drains: known duplicates are still dropped, but new
   ones may slip through until there is room again. Pick the smallest window that still covers the gap
   you expect between duplicates so you stay comfortably under the budget at your volume.
4. Whole-record keying is key-order independent (the record is canonicalized first).
5. If a configured key field is wholly absent from a record, that record is passed through rather than
   dropped, so distinct records are never collapsed onto an empty key. (A field present as `null` or
   `""` still counts as present.)
6. Deduplication is best-effort and fails open: if the backing store is briefly unavailable, records
   pass through rather than being dropped, so data is never lost — at most a duplicate slips through.
7. Deduplication is scoped to the node: records are compared against others flowing through the same
   transform node within the window.