Azure Blob Storage Queue
Enables ingestion of Azure Blob Storage objects via Event Grid BlobCreated events delivered to an Azure Storage Queue.
Sync Type: Incremental
Details
The Azure Blob Storage Queue connector is the Azure equivalent of the AWS SQS S3 input. Instead of polling a container and listing its blobs, it consumes a queue that is fed by Event Grid: whenever a blob is created in your container, Event Grid publishes a Microsoft.Storage.BlobCreated event to an Azure Storage Queue, which this connector monitors. The connector then downloads the referenced blob, processes it according to your configured format and compression settings, and extracts records for your pipeline.
This queue-fronted approach is the pattern Microsoft recommends for higher-volume ingestion, since it avoids repeatedly listing the container contents.
How It Works
- Blob Event Generation: When a blob is created (committed) in your container, Azure generates a
Microsoft.Storage.BlobCreatedevent. - Event Routing: An Event Grid subscription routes that event to your Azure Storage Queue as a (base64-encoded) message.
- Event Processing: The connector dequeues messages, base64-decodes them, and parses the Event Grid (or CloudEvents) envelope.
- Commit Filtering: Only
BlobCreatedevents whosedata.apiis a fully-committed block-blob write (PutBlob,PutBlockList,CopyBlob) are acted on, so half-written blobs are never fetched. - Blob Download: The connector parses the account, container, and blob name from the event's
data.urland downloads the blob. - Data Processing: Downloaded blobs are decompressed (if needed) and parsed according to your format settings.
- Record Extraction: Individual records are extracted and sent to your data pipeline.
- Cleanup: Successfully processed queue messages are deleted from the queue.
Prerequisites
Before setting up the Azure Blob Storage Queue input, ensure you have:
- A Microsoft Azure account with an active subscription.
- A StorageV2 (general purpose v2), BlockBlobStorage, or BlobStorage account. Storage (general purpose v1) accounts cannot integrate with Event Grid and are not supported.
- A blob container that receives your source data.
- An Azure Storage Queue to receive blob events.
- An Event Grid subscription routing
BlobCreatedevents from the container to the queue. - A registered application in Azure Entra ID with the permissions below.
Required Permissions
The registered application requires two role assignments:
- Storage Queue Data Message Processor on the queue — read (dequeue), process, and delete messages. (Read-only "Storage Queue Data Reader" is not sufficient, because the connector acks by deleting.)
- Storage Blob Data Reader on the container — download blob contents. If your blobs span multiple storage accounts, grant this on each account.
Setup Instructions
Step 1: Create a Storage Account (if needed)
- Navigate to the Azure Portal.
- Go to Create a resource → Storage → Storage account.
- On the Basics tab, set Performance to Standard (this creates a StorageV2 account — the "Kind" is not a separate field and new accounts are never v1).
- Choose a subscription, resource group, name, and region, then create the account.
Use a standard blob account (leave Enable hierarchical namespace off). ADLS Gen2 (hierarchical namespace) is not supported by this connector: ADLS files are written with append-then-flush, and each flush would re-read the whole file and re-emit every record, duplicating data without bound for a continuously-appended file. Point this connector at write-once block blobs. (Block-blob writes on an HNS account still work; only the ADLS
dfs/FlushWithClosewrite path is unsupported.)
Step 2: Create a Container and a Queue
- In your storage account, open Containers (under Data storage), click + Container, and name it (e.g.,
security-logs). - Open Queues (under Data storage), click + Queue, and name it (e.g.,
blob-events). - Note the queue's full URL from the queue's Properties — it has the form
https://<account>.queue.core.windows.net/<queue>. You will enter it as Queue URL.
Step 3: Route Blob Events to the Queue (Event Grid)
- Open the storage account that holds the container → Events → + Event Subscription.
- Under Basics:
- Name: e.g.,
blobcreated-to-queue. - Event Schema: choose either Event Grid Schema or Cloud Event Schema v1.0 — the connector auto-detects and parses both.
- Filter to Event Types: select only
Blob Created. - Endpoint Type: Storage Queues → select your storage account and the queue from Step 2.
- Name: e.g.,
- Under Filters (recommended):
- Enable subject filtering to scope to one container: Subject Begins With =
/blobServices/default/containers/security-logs/, and optionally Subject Ends With =.jsonl(or another suffix). - Add an advanced filter on
data.apiis inPutBlob, PutBlockList, CopyBlobso only fully-committed block blobs produce events.
- Enable subject filtering to scope to one container: Subject Begins With =
- Click Create.
Step 4: Register an Application in Azure Entra ID
- Open the App Registration page in Azure Entra ID.
- Select New registration, name it (e.g.,
Monad-Blob-Queue), and click Register. - From the Overview page, save the
Application (client) IDandDirectory (tenant) ID. - Navigate to Certificates & secrets → New client secret, add a description and expiration, and copy the client secret value immediately (it is not shown again).
Step 5: Assign Permissions
Assign both roles to the registered application (at the account level, or narrowly on the queue and container):
- For the queue: open the storage account → Access control (IAM) → Add → Add role assignment → choose Storage Queue Data Message Processor → assign to your application.
- For the container: Add role assignment → choose Storage Blob Data Reader → assign to the same application.
RBAC assignments can take a few minutes to propagate.
Network access (restricted storage accounts)
If your Azure Storage account restricts public network access with a firewall (Networking → Firewalls and virtual networks), Monad's cluster must be explicitly allowed to reach it.
Why IP allowlisting fails for same-region clusters
If your Monad cluster runs in the same Azure region as your Azure Storage account, adding Monad's public IP to the allowlist will not work. Azure routes same-region requests over its internal network, so the storage account sees a private source IP that a public-IP allowlist rule can never match. This is documented Azure platform behavior:
"You can't use IP network rules to restrict access to clients in the same Azure region as the storage account. IP network rules have no effect on requests that originate from the same Azure region. Use Virtual network rules to allow same-region requests."
Resolution: allow Monad's subnet with a virtual-network rule
Because same-region traffic arrives with a private IP, you must allow Monad by subnet identity rather than by IP, using a VNet Service Endpoint:
- Monad provisions a VNet Service Endpoint for
Microsoft.Storageon the subnet of the Monad cluster running your connector. Contact Monad Customer Support to have this enabled and to obtain the subnet resource ID (the subnet lives in a Monad-managed VNet). - Add that subnet to your Storage Account's allowed virtual networks: Storage Account → Networking → Firewalls and virtual networks → Add existing virtual network. With the Service Endpoint in place, requests from Monad are recognized as coming from an allowed virtual network instead of an unmatchable private IP.
- Save. Access from Monad should succeed once the rule propagates.
If the portal won't let you add the subnet
The Azure portal only lets you add a virtual network/subnet you have read access to. Because the Monad cluster's VNet is Monad-managed, you likely won't see it in the portal picker. Add the rule with the Azure CLI instead:
Code
Monad Customer Support will provide the <monad-cluster-subnet-resource-id>.
This affects any Monad deployment running on Azure (AKS) — shared-tenancy SaaS as well as dedicated clusters — whenever your Azure Storage account is in the same Azure region as the AKS cluster running the connector. Deployments where Monad and the storage account are in different regions are unaffected (public-IP allowlisting works normally there).
Configuration
Settings
| Setting | Type | Required | Description |
|---|---|---|---|
| Queue URL | string | Yes | Full URL of the Storage Queue that Event Grid delivers BlobCreated events to (e.g., https://mystorageaccount.queue.core.windows.net/blob-events). |
| Tenant ID | string | Yes | The Azure Entra ID tenant (directory) ID where your application is registered. |
| Format | string | Yes | File format of the referenced blobs. Supported values: json, jsonl, csv, delimited, wsv (default: json). |
| Compression | string | Yes | Compression format of the referenced blobs. Supported values: auto, gzip, none. |
| Record Location | string | No | Location of the record in the JSON blob. JSON only; leave empty for the entire record. See Record Location for syntax and examples. |
| Key Filter | object | No | Optional filter applied to the blob name parsed from each event, before fetch. See Key Filter below. |
| With Metadata | boolean | No | When enabled, annotates each record with its source account, container, and blob name under _monad_metadata. Default: false. |
| Use Synthetic Data | boolean | No | Generate synthetic demo data instead of connecting to the real storage account. |
Secrets
| Secret | Type | Required | Description |
|---|---|---|---|
| Client ID | string | Yes | The application (client) ID from your Azure Entra ID app registration. |
| Client Secret | string | Yes | The client secret value from your Azure Entra ID app registration. |
Key Filter
Filters the blob name (parsed from the event) before the connector fetches it. Useful when the queue receives events for a wider set of blobs than this connector should process. Matching is case-insensitive on both the blob name and the configured value.
Key Filter is a discriminated union on the type field — the empty case is {"type": "none"} rather than an empty {mode, operator, value} object, so the per-field rules only apply when filtering is actually requested:
| Field | Type | Required | Description |
|---|---|---|---|
type | string | Yes | none passes every blob through; filter enables the rules in filter below. |
filter | object | When type is filter | The match parameters described below. Omit when type is none. |
When type is filter, the nested filter object takes:
| Field | Type | Required | Description |
|---|---|---|---|
mode | string | Yes | include processes only matching blobs; exclude skips matching blobs. |
operator | string | Yes | One of contains, ends_with, starts_with. |
value | string | Yes | The substring (or suffix/prefix) to match against the blob name. |
Examples:
- No filtering —
{ "type": "none" } - Process only gzipped JSON files —
{ "type": "filter", "filter": { "mode": "include", "operator": "ends_with", "value": ".json.gz" } } - Skip a noisy prefix —
{ "type": "filter", "filter": { "mode": "exclude", "operator": "starts_with", "value": "internal/" } }
When the field is omitted entirely (or set to {"type": "none"}), every blob referenced by the queue is processed.
Example Blob Event Message
Event Grid delivers each event to the queue base64-encoded; after decoding, the message body is a single-element array. The connector accepts both the Event Grid and CloudEvents schemas.
Event Grid schema:
Code
CloudEvents 1.0 schema (same data, different envelope — type/time/source/specversion):
Code
The connector parses these events and downloads the referenced blob for processing.
Important Notes
- Event Filtering: Only
Microsoft.Storage.BlobCreatedevents are processed;BlobDeleted,BlobRenamed,BlobTierChanged, and directory events are ignored. - Commit Filtering: Only fully-committed block-blob writes (
data.apiofPutBlob,PutBlockList, orCopyBlob) are fetched, so partial or in-progress uploads are never processed. ADLS Gen2 (hierarchical namespace) is not supported — itsFlushWithClosecommit is intentionally excluded, because append-then-flush would re-read and re-emit the whole file on every flush (unbounded duplication for continuously-appended files). Use write-once block blobs. - At-least-once Delivery: Event Grid guarantees at-least-once delivery and can deliver out of order, so duplicate events are possible. A message that fails repeatedly is never dropped — it stays in the queue and is retried indefinitely; the connector logs a warning once a message has been retried many times so a stuck message stays visible.
- Multiple Containers/Accounts: The account, container, and blob are read from each event's
data.url, so a single connector can serve many containers — even across storage accounts — as long as the application has Storage Blob Data Reader on each. - Message Deletion: Queue messages are deleted only after the referenced blob is successfully processed. On failure, the message remains in the queue and is retried.
Related Articles
- Reacting to Azure Blob storage events
- Azure Blob Storage as an Event Grid source (event schema)
- Route Event Grid events to Azure Queue Storage
- Azure RBAC Roles for Storage
Sync frequency
This input consumes from the source queue or subscription, ingesting records at whatever rate the source delivers them rather than on a fixed poll. See Input Sync Frequency for details.