Copy page

Microsoft

Azure Queue Storage

Consumes messages from Microsoft Azure Queue Storage queues for event ingestion and processing.

Sync Type: Incremental

Overview

The Azure Queue Storage input connector establishes a connection to Azure Queue Storage to ingest message data. It continuously polls queues for messages, dequeues them, emits the message data, and deletes successfully processed messages from the queue. Messages are automatically decoded from base64 format when applicable.

Prerequisites

Before setting up the Azure Queue Storage input, ensure you have:

1. A Microsoft Azure account with an active subscription
2. An Azure Storage account with Queue Storage enabled
3. One or more message queues created in the storage account
4. A registered application in Azure Entra ID with appropriate permissions
5. Access to the storage account access key or connection string

Required Permissions

The application requires the following permissions:

• Storage Queue Data Message Processor role on the storage account
• This role provides permissions to read (dequeue), process, and delete messages from queues

Setting up API Access

1. Create a Storage Account and Queue

1. Navigate to the Azure Portal
2. Create a Storage Account:
   • Go to "Create a resource" → "Storage" → "Storage account"
   • Configure the storage account settings (name, subscription, resource group, location)
   • Choose an appropriate redundancy level and performance tier
3. Create a queue within the storage account:
   • Navigate to your Storage account
   • Click "Queues" in the left sidebar under "Data storage"
   • Click "+ Queue"
   • Enter a queue name (e.g., myqueue, notifications, events)

2. Register Application in Azure Entra ID

1. Open the App Registration page in Azure Entra ID
2. Select "New Registration"
3. Add a name for your application (e.g., "Monad-Queue-Consumer")
4. Click "Register"
5. Save the Application (client) ID and Directory (tenant) ID from the Overview page
6. Navigate to "Certificates & secrets"
7. Click "New client secret"
8. Add a description and set an expiration (recommend 12 months)
9. Copy the client secret value immediately (it will not be visible again)

3. Assign Storage Queue Permissions

1. Navigate to your Storage account in the Azure Portal
2. Click "Access control (IAM)" in the left menu
3. Select "Add" → "Add role assignment"
4. Search for and choose the "Storage Queue Data Message Processor" role
5. Click "Next"
6. Select "User, group, or service principal"
7. Click "Select members"
8. Search for your registered application name
9. Select the application and click "Select"
10. Click "Review + assign"

Network access (restricted storage accounts)

If your Azure Storage account restricts public network access with a firewall (Networking → Firewalls and virtual networks), Monad's cluster must be explicitly allowed to reach it.

Why IP allowlisting fails for same-region clusters

If your Monad cluster runs in the same Azure region as your Azure Storage account, adding Monad's public IP to the allowlist will not work. Azure routes same-region requests over its internal network, so the storage account sees a private source IP that a public-IP allowlist rule can never match. This is documented Azure platform behavior:

"You can't use IP network rules to restrict access to clients in the same Azure region as the storage account. IP network rules have no effect on requests that originate from the same Azure region. Use Virtual network rules to allow same-region requests."

— Azure Storage network security limitations

Resolution: allow Monad's subnet with a virtual-network rule

Because same-region traffic arrives with a private IP, you must allow Monad by subnet identity rather than by IP, using a VNet Service Endpoint:

1. Monad provisions a VNet Service Endpoint for Microsoft.Storage on the subnet of the Monad cluster running your connector. Contact Monad Customer Support to have this enabled and to obtain the subnet resource ID (the subnet lives in a Monad-managed VNet).
2. Add that subnet to your Storage Account's allowed virtual networks: Storage Account → Networking → Firewalls and virtual networks → Add existing virtual network. With the Service Endpoint in place, requests from Monad are recognized as coming from an allowed virtual network instead of an unmatchable private IP.
3. Save. Access from Monad should succeed once the rule propagates.

If the portal won't let you add the subnet

The Azure portal only lets you add a virtual network/subnet you have read access to. Because the Monad cluster's VNet is Monad-managed, you likely won't see it in the portal picker. Add the rule with the Azure CLI instead:

Code
az storage account network-rule add \
  --resource-group <your-storage-account-resource-group> \
  --account-name <your-storage-account-name> \
  --subnet <monad-cluster-subnet-resource-id>

Monad Customer Support will provide the <monad-cluster-subnet-resource-id>.

Configuration

Settings

Setting	Type	Required	Description
Account Name	string	Yes	The name of the Azure Storage account (e.g., mystorageaccount). This is the first part of your storage account URL before .queue.core.windows.net
Queue Name	string	Yes	The name of the specific queue to consume messages from. Separate multiple queues by creating separate connectors with different queue names
Tenant ID	string	Yes	The Azure Entra ID tenant (directory) ID where your application is registered
Decode Base64 Messages	boolean	No	Decode message text from base64 before storing. Enable if your queue messages are base64-encoded (e.g., sent via Azure SDK defaults).
Visibility Timeout (seconds)	Integer	No	How long a dequeued message stays invisible to other consumers while being processed. If not deleted within this window, the message reappears in the queue. Min: 1, Max: 43200 (12 hours). Defaults to 30 if 0.
Use Synthetic Data	boolean	No	Generate synthetic demo data instead of connecting to the real storage account

Secrets

Secret	Type	Required	Description
Client ID	string	Yes	The application (client) ID from your Azure Entra ID app registration
Client Secret	string	Yes	The client secret value from your Azure Entra ID app registration

How It Works

Message Processing Details

• DeQueueCount Tracking: Messages dequeued more than once are skipped to prevent reprocessing of problematic messages
• Base64 Decoding: Message text is automatically base64-decoded when possible, with fallback to raw text if decoding fails
• Message Lifetime: Messages remain in the queue until successfully processed and deleted
• Queue Draining: When the queue is empty, the connector creates a checkpoint and waits for the next polling interval

Performance Considerations

• Polling Interval: Connector polls every 30 seconds for new messages
• Batch Size: Dequeues up to 32 messages per API request
• Queue Visibility: Dequeued messages are invisible to other consumers for a visibility timeout period (default: 300 seconds)
• Scaling: Process multiple queues by creating separate connectors for each queue name

Related Articles

• Azure Queue Storage Documentation
• Queue Storage API Reference
• Azure RBAC Roles for Storage
• Storage Queue Data Message Processor Role

Sync frequency

This input consumes from the source queue or subscription, ingesting records at whatever rate the source delivers them rather than on a fixed poll. See Input Sync Frequency for details.