Skip to main content

Azure Queue Storage

Consumes messages from Microsoft Azure Queue Storage queues for event ingestion and processing.

Sync Type: Incremental

Overview​

The Azure Queue Storage input connector establishes a connection to Azure Queue Storage to ingest message data. It continuously polls queues for messages, dequeues them, emits the message data, and deletes successfully processed messages from the queue. Messages are automatically decoded from base64 format when applicable.

Prerequisites​

Before setting up the Azure Queue Storage input, ensure you have:

  1. A Microsoft Azure account with an active subscription
  2. An Azure Storage account with Queue Storage enabled
  3. One or more message queues created in the storage account
  4. A registered application in Azure Entra ID with appropriate permissions
  5. Access to the storage account access key or connection string

Required Permissions​

The application requires the following permissions:

  • Storage Queue Data Message Processor role on the storage account
  • This role provides permissions to read (dequeue), process, and delete messages from queues

Setting up API Access​

1. Create a Storage Account and Queue​

  1. Navigate to the Azure Portal
  2. Create a Storage Account:
    • Go to "Create a resource" → "Storage" → "Storage account"
    • Configure the storage account settings (name, subscription, resource group, location)
    • Choose an appropriate redundancy level and performance tier
  3. Create a queue within the storage account:
    • Navigate to your Storage account
    • Click "Queues" in the left sidebar under "Data storage"
    • Click "+ Queue"
    • Enter a queue name (e.g., myqueue, notifications, events)

2. Register Application in Azure Entra ID​

  1. Open the App Registration page in Azure Entra ID
  2. Select "New Registration"
  3. Add a name for your application (e.g., "Monad-Queue-Consumer")
  4. Click "Register"
  5. Save the Application (client) ID and Directory (tenant) ID from the Overview page
  6. Navigate to "Certificates & secrets"
  7. Click "New client secret"
  8. Add a description and set an expiration (recommend 12 months)
  9. Copy the client secret value immediately (it will not be visible again)

3. Assign Storage Queue Permissions​

  1. Navigate to your Storage account in the Azure Portal
  2. Click "Access control (IAM)" in the left menu
  3. Select "Add" → "Add role assignment"
  4. Search for and choose the "Storage Queue Data Message Processor" role
  5. Click "Next"
  6. Select "User, group, or service principal"
  7. Click "Select members"
  8. Search for your registered application name
  9. Select the application and click "Select"
  10. Click "Review + assign"

Configuration​

Settings​

SettingTypeRequiredDescription
Account NamestringYesThe name of the Azure Storage account (e.g., mystorageaccount). This is the first part of your storage account URL before .queue.core.windows.net
Queue NamestringYesThe name of the specific queue to consume messages from. Separate multiple queues by creating separate connectors with different queue names
Tenant IDstringYesThe Azure Entra ID tenant (directory) ID where your application is registered
Decode Base64 MessagesbooleanNoDecode message text from base64 before storing. Enable if your queue messages are base64-encoded (e.g., sent via Azure SDK defaults).
Visibility Timeout (seconds)IntegerNoHow long a dequeued message stays invisible to other consumers while being processed. If not deleted within this window, the message reappears in the queue. Min: 1, Max: 43200 (12 hours). Defaults to 30 if 0.
Use Synthetic DatabooleanNoGenerate synthetic demo data instead of connecting to the real storage account

Secrets​

SecretTypeRequiredDescription
Client IDstringYesThe application (client) ID from your Azure Entra ID app registration
Client SecretstringYesThe client secret value from your Azure Entra ID app registration

How It Works​

Message Processing Details​

  • DeQueueCount Tracking: Messages dequeued more than once are skipped to prevent reprocessing of problematic messages
  • Base64 Decoding: Message text is automatically base64-decoded when possible, with fallback to raw text if decoding fails
  • Message Lifetime: Messages remain in the queue until successfully processed and deleted
  • Queue Draining: When the queue is empty, the connector creates a checkpoint and waits for the next polling interval

Performance Considerations​

  • Polling Interval: Connector polls every 30 seconds for new messages
  • Batch Size: Dequeues up to 32 messages per API request
  • Queue Visibility: Dequeued messages are invisible to other consumers for a visibility timeout period (default: 300 seconds)
  • Scaling: Process multiple queues by creating separate connectors for each queue name