Skip to main content

Transforms

Data transforms are the core processing components of our system, sitting between input and output connectors. They perform crucial operations on your ingested data, enabling you to clean, normalize, and enrich data before it reaches its final destination.

What Are Data Transforms?​

Data transforms are a series of operations applied to your data as it flows through our pipelines. These operations can include:

  • Data cleaning and normalization
  • Field extraction and parsing
  • Enrichment through conditional addition of key:value pairs to records
  • Filtering and routing based on specific criteria
  • Aggregation and statistical analysis

Our transform engine uses the GJSON and SJSON packages in Golang, providing powerful JSON traversal and manipulation capabilities.

Key Features​

  • Flexible JSON processing using GJSON pathing
  • Support for complex queries and pattern matching
  • Ability to chain multiple transforms for sophisticated data processing
  • Real-time processing with minimal latency

How Transforms Work​

  1. Data ingested through input connectors enters the transform pipeline
  2. Each configured transform applies its operations to the data
  3. Transforms can be chained, with the output of one serving as input to the next
  4. Processed data is then sent to the appropriate output connector

What You'll Find​

In the following pages, you'll find:

  • Detailed explanations of available transform types
  • Configuration options and syntax for each transform
  • Best practices for designing efficient transform pipelines
  • Examples of common transform scenarios

Example:​

Here is an example record. We will try and target some specific fields in the record.

{
  "name": {"first": "Tom", "last": "Anderson"},
  "age":37,
  "children": ["Sara","Alex","Jack"],
  "fav.movie": "Deer Hunter",
  "friends": [
    {"first": "Dale", "last": "Murphy", "age": 44, "nets": ["ig", "fb", "tw"]},
    {"first": "Roger", "last": "Craig", "age": 68, "nets": ["fb", "tw"]},
    {"first": "Jane", "last": "Murphy", "age": 47, "nets": ["ig", "tw"]}
  ]
}
KEYS                 RESULT
name.last        >> "Anderson"
age              >> 37
children         >> ["Sara","Alex","Jack"]
children.#       >> 3
children.1       >> "Alex"
child*.2         >> "Jack"
c?ildren.0       >> "Sara"
fav\.movie       >> "Deer Hunter"
friends.#.first  >> ["Dale","Roger","Jane"]
friends.1.last   >> "Craig"

Query an array for the first match by using #(...), or find all matches with #(...)#. Queries support the ==, !=, <, <=, >, >= comparison operators and the simple pattern matching % operator.

friends.#(last=="Murphy").first   >> "Dale"
friends.#(last=="Murphy")#.first  >> ["Dale","Jane"]
friends.#(age>45)#.last           >> ["Craig","Murphy"]
friends.#(first%"D*").last        >> "Murphy"
friends.#(nets.#(=="fb"))#.first  >> ["Dale","Roger"]

Getting Started​

To begin working with transforms, familiarize yourself with the GJSON pathing syntax and our query capabilities. You can experiment with JSON traversal here using the gjson.dev sandbox.

Explore the sidebar to learn about specific transforms/operations.