Microsoft Azure Event Hubs
Streams real-time data from Azure Event Hubs for event ingestion and processing.
Sync Type: Streaming
Overview
The Azure Event Hubs input connector reads event data from Azure Event Hubs. It processes events from all partitions concurrently and keeps its own position for each partition between syncs.
Prerequisites
Before setting up the Azure Event Hubs input, ensure you have:
- A Microsoft Azure account with an active subscription
- An Azure Event Hubs namespace and Event Hub instance
- A registered application in Azure Entra ID with appropriate permissions
- A Microsoft Entra service principal (client ID, client secret and tenant ID) for that application; connection strings and shared access signatures (SAS) are not supported
Required Permissions
The application requires the following permissions:
- Azure Event Hubs Data Receiver role on the Event Hubs namespace or specific Event Hub
- Access to the specified consumer group (default:
$Default)
Setting up API Access
1. Create an Event Hubs Namespace and Event Hub
- Navigate to the Azure Portal
- Create an Event Hubs namespace:
- Go to "Create a resource" → "Integration" → "Event Hubs"
- Configure the namespace settings (name, subscription, resource group, location)
- Choose an appropriate pricing tier
- Create an Event Hub within the namespace:
- Navigate to your Event Hubs namespace
- Click "Event Hubs" → "+ Event Hub"
- Configure the Event Hub (name, partition count, message retention)
2. Register Application in Azure Entra ID
- Open the App Registration page in Azure Entra ID
- Select "New Registration"
- Add a name for your application
- Click "Register"
- Save the
Application (client) IDandDirectory (tenant) ID - Navigate to "Certificates & secrets"
- Click "New client secret"
- Add a description and set expiration
- Save the client secret value immediately (it won't be visible again)
3. Assign Event Hubs Permissions
- Navigate to your Event Hubs namespace in the Azure Portal
- Click "Access control (IAM)" in the left menu
- Select "Add" → "Add role assignment"
- Choose the "Azure Event Hubs Data Receiver" role
- Click "Next"
- Select "User, group, or service principal"
- Click "Select members"
- Search for your registered application name
- Select the application and click "Select"
- Click "Review + assign"
Configuration
Settings
| Setting | Type | Required | Default | Description |
|---|---|---|---|---|
subscription_id | string | Yes | - | The unique identifier of your Azure subscription, used to manage resources and billing in Azure Cloud. |
tenant_id | string | Yes | - | The tenant ID of the Entra ID directory associated with your Azure subscription. |
event_hub_namespace | string | Yes | - | The fully qualified namespace URL (e.g., your-namespace.servicebus.windows.net) |
event_hub_name | string | Yes | - | The name of the specific Event Hub to consume from |
consumer_group | string | Yes | $Default | The consumer group name for reading events (default: $Default) |
lookback_duration | int | Yes | 60 | The duration to look back for events in minutes. Defaults to 60 minutes if not specified. |
record_location | string | No | - | Location of the record in the object. Applies only for JSON objects. Leave empty for the entire record. |
Secrets
| Secret | Type | Required | Description |
|---|---|---|---|
client_id | secret | Yes | The client ID of the Entra ID application |
client_secret | secret | Yes | The client secret of the Entra ID application |
How It Works
Streaming Process
- Connection Initialization: Establishes authenticated connection to Event Hubs using client credentials
- Partition Discovery: Automatically discovers all partitions within the specified Event Hub
- Checkpoint Loading: Loads previous consumption state from persistent storage
- Parallel Processing: Creates separate consumers for each partition to process events concurrently
- Event Reading: Each sync reads every partition from where it left off up to the newest event at that moment, then stops. The next sync starts about 10 seconds later, unless a cron schedule is set on the pipeline.
- State Management: Updates checkpoints after processing each batch to track progress
Checkpoint Management
Monad stores its own position for each partition; no Azure checkpoint store or partition ownership is used.
On the first run, each partition starts at the current time minus Lookback Duration (60 minutes by default).
Events older than that, or past the Event Hub's retention period, are not read.
The consumer group must already exist in the Event Hub (default $Default).
Delivery is at-least-once, so an event can occasionally be delivered more than once.
Performance Considerations
- Partition Count: Higher partition counts enable greater parallelism and throughput
- Consumer Group: Use dedicated consumer groups to avoid conflicts with other applications
- Batch Size: Processes up to 100 events per batch for optimal performance
Using this input in more than one pipeline
Each pipeline that uses this input reads every partition and keeps its own checkpoints, so every pipeline receives every event, even when they share a consumer group. Azure allows at most five concurrent readers per partition in one consumer group, and each pipeline opens one reader per partition. Give each pipeline its own consumer group so additional pipelines, or other applications reading the same group, cannot push a partition over that limit.
A pipeline added later starts from the current time minus Lookback Duration, so it does not receive older events that an earlier pipeline already read. See Using One Input in Multiple Pipelines.
Troubleshooting
Common Issues
-
Authentication Errors:
- Verify client ID and secret are correct
- Ensure application has "Azure Event Hubs Data Receiver" role
- Check tenant ID matches your Azure subscription
-
Connection Failures:
- Verify Event Hub namespace URL format
- Ensure Event Hub name exists in the namespace
- Check network connectivity and firewall rules
-
No Events Received:
- Verify events are being sent to the Event Hub
- Check consumer group configuration
- Ensure partition has available events
Related Articles
Sync frequency
This input consumes from the source queue or subscription, ingesting records at whatever rate the source delivers them rather than on a fixed poll. See Input Sync Frequency for details.