Microsoft Azure Queue Storage
Consumes messages from Microsoft Azure Queue Storage queues for event ingestion and processing.
Sync Type: Streaming
Overview
The Azure Queue Storage input connector reads message data from an Azure Queue Storage queue. Each sync dequeues messages in batches of 32 until the queue is empty, then stops; the next sync starts about 10 seconds later. Each message is emitted and then deleted from the queue. Message text is decoded from base64 only when Decode Base64 Messages is enabled.
One queue per pipeline
Do not point two pipelines at the same queue, whether they share this input or each have their own input configured with the same queue. The pipelines compete for the queue's messages: each message is delivered to only one of them and then deleted, so each pipeline silently receives only part of the data. To send this data to several destinations, add more outputs to a single pipeline, or give each pipeline its own queue. See Using One Input in Multiple Pipelines.
Prerequisites
Before setting up the Azure Queue Storage input, ensure you have:
- A Microsoft Azure account with an active subscription
- An Azure Storage account with Queue Storage enabled
- One or more message queues created in the storage account
- A registered application in Azure Entra ID with appropriate permissions; only a Microsoft Entra service principal (client ID, client secret and tenant ID) is accepted
Required Permissions
The application requires the following permissions:
- Storage Queue Data Message Processor role on the storage account
- This role provides permissions to read (dequeue), process, and delete messages from queues
Setting up API Access
1. Create a Storage Account and Queue
- Navigate to the Azure Portal
- Create a Storage Account:
- Go to "Create a resource" → "Storage" → "Storage account"
- Configure the storage account settings (name, subscription, resource group, location)
- Choose an appropriate redundancy level and performance tier
- Create a queue within the storage account:
- Navigate to your Storage account
- Click "Queues" in the left sidebar under "Data storage"
- Click "+ Queue"
- Enter a queue name (e.g.,
myqueue,notifications,events)
2. Register Application in Azure Entra ID
- Open the App Registration page in Azure Entra ID
- Select "New Registration"
- Add a name for your application (e.g., "Monad-Queue-Consumer")
- Click "Register"
- Save the
Application (client) IDandDirectory (tenant) IDfrom the Overview page - Navigate to "Certificates & secrets"
- Click "New client secret"
- Add a description and set an expiration (recommend 12 months)
- Copy the client secret value immediately (it will not be visible again)
3. Assign Storage Queue Permissions
- Navigate to your Storage account in the Azure Portal
- Click "Access control (IAM)" in the left menu
- Select "Add" → "Add role assignment"
- Search for and choose the "Storage Queue Data Message Processor" role
- Click "Next"
- Select "User, group, or service principal"
- Click "Select members"
- Search for your registered application name
- Select the application and click "Select"
- Click "Review + assign"
Network access (restricted storage accounts)
If your Azure Storage account restricts public network access with a firewall (Networking → Firewalls and virtual networks), Monad's cluster must be explicitly allowed to reach it.
Why IP allowlisting fails for same-region clusters
If your Monad cluster runs in the same Azure region as your Azure Storage account, adding Monad's public IP to the allowlist will not work. Azure routes same-region requests over its internal network, so the storage account sees a private source IP that a public-IP allowlist rule can never match. This is documented Azure platform behavior:
"You can't use IP network rules to restrict access to clients in the same Azure region as the storage account. IP network rules have no effect on requests that originate from the same Azure region. Use Virtual network rules to allow same-region requests."
Resolution: allow Monad's subnet with a virtual-network rule
Because same-region traffic arrives with a private IP, you must allow Monad by subnet identity rather than by IP, using a VNet Service Endpoint:
- Monad provisions a VNet Service Endpoint for
Microsoft.Storageon the subnet of the Monad cluster running your connector. Contact Monad Customer Support to have this enabled and to obtain the subnet resource ID (the subnet lives in a Monad-managed VNet). - Add that subnet to your Storage Account's allowed virtual networks: Storage Account → Networking → Firewalls and virtual networks → Add existing virtual network. With the Service Endpoint in place, requests from Monad are recognized as coming from an allowed virtual network instead of an unmatchable private IP.
- Save. Access from Monad should succeed once the rule propagates.
If the portal won't let you add the subnet
The Azure portal only lets you add a virtual network/subnet you have read access to. Because the Monad cluster's VNet is Monad-managed, you likely won't see it in the portal picker. Add the rule with the Azure CLI instead:
Code
Monad Customer Support will provide the <monad-cluster-subnet-resource-id>.
This affects any Monad deployment running on Azure (AKS) — shared-tenancy SaaS as well as dedicated clusters — whenever your Azure Storage account is in the same Azure region as the AKS cluster running the connector. Deployments where Monad and the storage account are in different regions are unaffected (public-IP allowlisting works normally there).
Configuration
Settings
| Setting | Type | Required | Default | Description |
|---|---|---|---|---|
account_name | string | Yes | - | The name of the Azure Storage account (e.g., mystorageaccount). |
queue_name | string | Yes | - | Queue name to consume messages from. |
tenant_id | string | Yes | - | The tenant ID of the Entra ID directory associated with your Azure subscription. |
decode_base64_messages | bool | No | - | Decode message text from base64 before storing. Enable if your queue messages are base64-encoded (e.g., sent via Azure SDK defaults). |
visibility_timeout_seconds | int | No | 30 | How long a dequeued message stays invisible to other consumers while being processed. If not deleted within this window, the message reappears in the queue. Min: 1, Max: 43200 (12 hours). Defaults to 30 if 0. |
use_synthetic_data | bool | No | false | Generate synthetic data for testing, instead of connecting to a real data source. Defaults to an hourly cron schedule for cron-based inputs. |
Secrets
| Secret | Type | Required | Description |
|---|---|---|---|
client_id | secret | Yes | The client ID of the Entra ID application |
client_secret | secret | Yes | The client secret of the Entra ID application |
How It Works
Message Processing Details
- Base64 Decoding: When Decode Base64 Messages is enabled, message text is base64-decoded, with fallback to raw text if decoding fails. When it is disabled, message text is kept as received.
- Message Deletion: A message is deleted as soon as Monad has read it. If processing a batch takes longer than the visibility timeout, messages can reappear in the queue and be collected twice.
- Queue Draining: When the queue is empty, the connector creates a checkpoint and waits for the next sync
Record format
Each record has the following fields:
message_idinsertion_timemessage_text(the message body as a string)dequeue_countexpiration_timenext_visible_timepop_receipt
Performance Considerations
- Polling Interval: Connector polls about every 10 seconds for new messages
- Batch Size: Dequeues up to 32 messages per API request
- Queue Visibility: Dequeued messages are invisible to other consumers for a visibility timeout period (default: 30 seconds)
- Scaling: Process multiple queues by creating separate connectors for each queue name
Related Articles
- Azure Queue Storage Documentation
- Queue Storage API Reference
- Azure RBAC Roles for Storage
- Storage Queue Data Message Processor Role
Sync frequency
This input consumes from the source queue or subscription, ingesting records at whatever rate the source delivers them rather than on a fixed poll. See Input Sync Frequency for details.