Transforms
Data transforms are the core processing components of our system, sitting between input and output connectors. They perform crucial operations on your ingested data, enabling you to clean, normalize, and enrich data before it reaches its final destination.
What Are Data Transforms?
Data transforms are a series of operations applied to your data as it flows through our pipelines. These operations can include:
- Data cleaning and normalization
- Field extraction and parsing
- Enrichment through conditional addition of key
pairs to records - Filtering and routing based on specific criteria
Every operation works on a single record at a time. There are no aggregation, grouping, or windowing operations — a transform cannot combine values across records.
Most operations use the GJSON and SJSON packages in Golang for JSON traversal and manipulation. The JQ operation is the exception: it runs a jq query engine, so GJSON pathing does not apply inside a jq query.
Key Features
- Flexible JSON processing using GJSON pathing
- Support for complex queries and pattern matching
- Ability to chain multiple transforms for sophisticated data processing
- Real-time processing with minimal latency
How Transforms Work
- Data ingested through input connectors enters the transform pipeline
- Each configured transform applies its operations to the data
- Transforms can be chained, with the output of one serving as input to the next
- Processed data is then sent to the appropriate output connector
Limits
A single transform holds at most 20 operations. Configuring more fails with
too many operations, max allowed is 20. Chain additional transforms in the pipeline when you
need more than 20 operations.
What You'll Find
In the following pages, you'll find:
- Detailed explanations of available transform types
- Configuration options and syntax for each transform
- Best practices for designing efficient transform pipelines
- Examples of common transform scenarios
Example:
Here is an example record. We will try and target some specific fields in the record.
Code
Code
Query an array for the first match by using #(...), or find all matches with #(...)#. Queries support the ==, !=, <, <=, >, >= comparison operators and the simple pattern matching % operator.
Code
Getting Started
To begin working with transforms, familiarize yourself with the GJSON pathing syntax and our query capabilities. You can experiment with JSON traversal here using the gjson.dev sandbox.
Explore the sidebar to learn about specific transforms/operations.