Data Routing

Data routing allows you to conditionally send data records to different destinations based on the content of each record. In the pipeline UI, routing is configured through edges - the connections between your inputs and outputs that determine how data flows through your system.

Overview

Each edge in your pipeline represents a routing rule. When you connect an input to an output, you create an edge that can include conditions to control which records pass through. Data flows from inputs through edges to outputs based on the conditions you define.

Edges use logical conditions to evaluate records. For complete documentation on conditionals, operators, and condition types, see Conditionals.

Understanding Edges

In the pipeline interface:

• Edges are the arrow connections between nodes
• Each edge has a name label (e.g., "always", "high-priority")
• Clicking on an edge opens its configuration panel
• You can have multiple edges from a single input to different outputs

Pipeline Structure

Your pipelines follow these important structural rules:

• Single Input: Each component accepts exactly one incoming edge
• Multiple Outputs: Components can have multiple outgoing edges to different destinations
• Tree Structure: Edges cannot create loops - data flows forward only, preventing circular routing

This design ensures predictable data flow and makes it easy to trace how records move through your pipeline.

Configuring an Edge

Accessing Edge Configuration

1. Click on any edge in your pipeline to select it
2. The configuration panel appears showing:
   • Name: The edge identifier
   • Description: Optional description of the edge's purpose
   • Conditions: Current operator and rules summary
3. Click "Edit edge" to modify the configuration

Edit Modes

The edge editor provides two ways to configure routing:

Form View

The visual interface for configuring edges:

• Operator dropdown: Select how conditions are combined
• Rules section: Add and configure individual conditions
• Each rule can be edited or removed independently

Code View

Direct JSON editing for advanced users:

{
  "operator": "and",
  "conditions": [
    {
      "type_id": "equals_any",
      "config": {
        "key": "status",
        "values": ["active"]
      }
    }
  ]
}

Edge Components

Each edge configuration is a logical condition consisting of:

1. Operator - How multiple conditions are combined (always, and, or, nor, xor)
2. Conditions - The leaf conditions or nested logical conditions that records must meet

Operators

Select an operator from the dropdown to determine how rules are evaluated. See Logical Operators for complete details.

Operator	Description	Conditions Required
always	Routes all records without any conditions	0
never	Blocks all records	0
and	All conditions must be true	At least 1
or	At least one condition must be true	At least 2
nor	None of the conditions should be true	At least 1
xor	Exactly one condition must be true	At least 2

Available Conditions

The following condition types can be used in edge routing. See Leaf Condition Types for complete configuration details.

Condition	Description
key_exists	Check if a field exists in the record
equals	Check if a field equals a specific value
equals_any	Check if a field matches any value in a list
contains	Check if a string contains a substring
starts_with	Check if a string starts with a prefix
ends_with	Check if a string ends with a suffix
matches_regex	Check if a string matches a regex pattern
greater_than	Check if a number is greater than a threshold
less_than	Check if a number is less than a threshold
sample	Sample a percentage of records randomly or by key

Creating and Managing Edges

Adding a New Edge

1. In the pipeline view, drag from an input's output port to an output's input port
2. A new edge is created with default settings
3. Click the edge to configure its conditions

Configuring Conditions

1. Click "Edit edge" to open the editor
2. Select your preferred view (Form or Code)
3. In Form view:
   • Choose an operator from the dropdown
   • Click "Add new rule" to add conditions
   • Configure each rule's type and parameters
4. In Code view:
   • Edit the JSON directly
   • Ensure valid JSON syntax
5. Click "Update" to save changes

Multiple Edges from One Input

You can create multiple edges from a single input to different outputs:

• Each edge evaluates independently
• Records can match multiple edges and be sent to multiple destinations
• All edges are evaluated for every record - order doesn't matter

Common Patterns

Simple Pass-Through

{
  "operator": "always",
  "conditions": []
}

All data flows through without filtering.

Multi-Criteria Filtering

{
  "operator": "and",
  "conditions": [
    {
      "type_id": "key_exists",
      "config": {"key": "customer_id"}
    },
    {
      "type_id": "equals_any",
      "config": {
        "key": "status",
        "values": ["active", "pending"]
      }
    }
  ]
}

Records must have a customer_id AND status of active or pending.

Priority Routing

Create multiple edges with different conditions (all evaluate independently):

• Edge 1: "critical" - Routes severity:critical to alerts.urgent
• Edge 2: "standard" - Routes severity:warning to alerts.normal
• Edge 3: "archive" - Routes all records (using "always") to storage.archive

Note: A single record can match multiple edges and be routed to multiple destinations.

Best Practices

1. Name edges descriptively: Use names that describe the routing purpose (e.g., "high-value-customers" not "edge1")

2. Use the Code view for complex logic: While the Form view is user-friendly, Code view is faster for complex configurations and is a great way to copy and paste your edge configurations

3. Test edge conditions: Use test data to verify your routing logic before connecting to production outputs

4. Document with descriptions: Add descriptions to edges explaining their purpose for team members

5. Consider edge interactions: Since all edges are evaluated, ensure your conditions work well together when records may match multiple edges

Visual Pipeline Tips

• Edge colors: Selected edges are highlighted for clarity
• Edge labels: Always visible to understand data flow at a glance
• Hover information: Hover over edges to see quick configuration details
• Remove edges: Use the "Remove" button in the configuration panel to delete unwanted edges

Pipeline Constraints

Understanding these constraints helps you design effective pipelines:

Single Input Rule

Each component can receive data from only one source through a single incoming edge, but can distribute that data to multiple destinations through multiple outgoing edges. This creates a fan-out pattern where data flows from one source to potentially many destinations.

No Circular Routing

Components can have multiple outgoing edges but cannot receive edges that create loops back to themselves. This constraint ensures all pipelines maintain a tree-like structure without cycles, preventing infinite loops and ensuring predictable data flow.

Practical Implications

• Data flow: Always flows forward through the tree structure, never backwards
• Debugging: The tree structure makes it easier to trace data lineage and troubleshoot issues
• Performance: Prevents infinite loops and ensures efficient processing

Validation

The UI validates your configuration:

• JSON syntax in Code view must be valid
• Required fields must be populated
• Operators have minimum condition requirements (see Operators table)

Troubleshooting

Edge not passing data:

• Check if conditions are too restrictive
• Verify field names match your data exactly (see Key Path Syntax)
• Test with a simple "always" operator first

Cannot save edge configuration:

• Ensure JSON is valid in Code view
• Check all required fields are filled
• Verify condition count meets operator requirements

Unexpected routing behavior:

• Remember all edges evaluate independently
• A record matching multiple edges will be sent to all matching destinations
• Use mutually exclusive conditions if you need records to go to only one destination