# Azure Blob Storage Queue Enables ingestion of Azure Blob Storage objects via Event Grid `BlobCreated` events delivered to an Azure Storage Queue. **Sync Type: Incremental** ## Details The Azure Blob Storage Queue connector is the Azure equivalent of the AWS [SQS S3](../aws/aws-sqs-s3) input. Instead of polling a container and listing its blobs, it consumes a queue that is fed by Event Grid: whenever a blob is created in your container, Event Grid publishes a `Microsoft.Storage.BlobCreated` event to an Azure Storage Queue, which this connector monitors. The connector then downloads the referenced blob, processes it according to your configured format and compression settings, and extracts records for your pipeline. This queue-fronted approach is the pattern Microsoft recommends for higher-volume ingestion, since it avoids repeatedly listing the container contents. ## How It Works 1. **Blob Event Generation**: When a blob is created (committed) in your container, Azure generates a `Microsoft.Storage.BlobCreated` event. 2. **Event Routing**: An Event Grid subscription routes that event to your Azure Storage Queue as a (base64-encoded) message. 3. **Event Processing**: The connector dequeues messages, base64-decodes them, and parses the Event Grid (or CloudEvents) envelope. 4. **Commit Filtering**: Only `BlobCreated` events whose `data.api` is a fully-committed block-blob write (`PutBlob`, `PutBlockList`, `CopyBlob`) are acted on, so half-written blobs are never fetched. 5. **Blob Download**: The connector parses the account, container, and blob name from the event's `data.url` and downloads the blob. 6. **Data Processing**: Downloaded blobs are decompressed (if needed) and parsed according to your format settings. 7. **Record Extraction**: Individual records are extracted and sent to your data pipeline. 8. **Cleanup**: Successfully processed queue messages are deleted from the queue. ## Prerequisites Before setting up the Azure Blob Storage Queue input, ensure you have: 1. A Microsoft Azure account with an active subscription. 2. A **StorageV2 (general purpose v2)**, **BlockBlobStorage**, or **BlobStorage** account. Storage (general purpose v1) accounts **cannot** integrate with Event Grid and are not supported. 3. A blob container that receives your source data. 4. An Azure Storage Queue to receive blob events. 5. An Event Grid subscription routing `BlobCreated` events from the container to the queue. 6. A registered application in Azure Entra ID with the permissions below. ## Required Permissions The registered application requires two role assignments: - **Storage Queue Data Message Processor** on the queue — read (dequeue), process, and **delete** messages. (Read-only "Storage Queue Data Reader" is not sufficient, because the connector acks by deleting.) - **Storage Blob Data Reader** on the container — download blob contents. If your blobs span multiple storage accounts, grant this on each account. ## Setup Instructions ### Step 1: Create a Storage Account (if needed) 1. Navigate to the [Azure Portal](https://portal.azure.com). 2. Go to **Create a resource** → **Storage** → **Storage account**. 3. On the **Basics** tab, set **Performance** to **Standard** (this creates a StorageV2 account — the "Kind" is not a separate field and new accounts are never v1). 4. Choose a subscription, resource group, name, and region, then create the account. > Use a **standard** blob account (leave **Enable hierarchical namespace** off). **ADLS Gen2 (hierarchical namespace) is not supported** by this connector: ADLS files are written with append-then-flush, and each flush would re-read the whole file and re-emit every record, duplicating data without bound for a continuously-appended file. Point this connector at write-once block blobs. (Block-blob writes on an HNS account still work; only the ADLS `dfs`/`FlushWithClose` write path is unsupported.) ### Step 2: Create a Container and a Queue 1. In your storage account, open **Containers** (under **Data storage**), click **+ Container**, and name it (e.g., `security-logs`). 2. Open **Queues** (under **Data storage**), click **+ Queue**, and name it (e.g., `blob-events`). 3. Note the queue's full URL from the queue's **Properties** — it has the form `https://.queue.core.windows.net/`. You will enter it as **Queue URL**. ### Step 3: Route Blob Events to the Queue (Event Grid) 1. Open the storage account that holds the **container** → **Events** → **+ Event Subscription**. 2. Under **Basics**: - **Name**: e.g., `blobcreated-to-queue`. - **Event Schema**: choose either **Event Grid Schema** or **Cloud Event Schema v1.0** — the connector auto-detects and parses both. - **Filter to Event Types**: select **only** `Blob Created`. - **Endpoint Type**: **Storage Queues** → select your storage account and the queue from Step 2. 3. Under **Filters** (recommended): - Enable subject filtering to scope to one container: **Subject Begins With** = `/blobServices/default/containers/security-logs/`, and optionally **Subject Ends With** = `.jsonl` (or another suffix). - Add an advanced filter on `data.api` **is in** `PutBlob, PutBlockList, CopyBlob` so only fully-committed block blobs produce events. 4. Click **Create**. ### Step 4: Register an Application in Azure Entra ID 1. Open the [App Registration](https://entra.microsoft.com/#view/Microsoft_AAD_RegisteredApps/ApplicationsListBlade) page in Azure Entra ID. 2. Select **New registration**, name it (e.g., `Monad-Blob-Queue`), and click **Register**. 3. From the **Overview** page, save the `Application (client) ID` and `Directory (tenant) ID`. 4. Navigate to **Certificates & secrets** → **New client secret**, add a description and expiration, and **copy the client secret value immediately** (it is not shown again). ### Step 5: Assign Permissions Assign both roles to the registered application (at the account level, or narrowly on the queue and container): 1. For the **queue**: open the storage account → **Access control (IAM)** → **Add** → **Add role assignment** → choose **Storage Queue Data Message Processor** → assign to your application. 2. For the **container**: **Add role assignment** → choose **Storage Blob Data Reader** → assign to the same application. RBAC assignments can take a few minutes to propagate. ## Network access (restricted storage accounts) If your Azure Storage account restricts public network access with a firewall (**Networking → Firewalls and virtual networks**), Monad's cluster must be explicitly allowed to reach it. ### Why IP allowlisting fails for same-region clusters If your Monad cluster runs in the same Azure region as your Azure Storage account, adding Monad's public IP to the allowlist will **not** work. Azure routes same-region requests over its internal network, so the storage account sees a private source IP that a public-IP allowlist rule can never match. This is documented Azure platform behavior: > "You can't use IP network rules to restrict access to clients in the same Azure region as the storage account. IP network rules have no effect on requests that originate from the same Azure region. Use Virtual network rules to allow same-region requests." > > — [Azure Storage network security limitations](https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security-limitations#restrictions-for-ip-network-rules) ### Resolution: allow Monad's subnet with a virtual-network rule Because same-region traffic arrives with a private IP, you must allow Monad by **subnet identity** rather than by IP, using a VNet Service Endpoint: 1. Monad provisions a **VNet Service Endpoint for `Microsoft.Storage`** on the subnet of the Monad cluster running your connector. Contact Monad Customer Support to have this enabled and to obtain the **subnet resource ID** (the subnet lives in a Monad-managed VNet). 2. Add that subnet to your Storage Account's allowed virtual networks: **Storage Account → Networking → Firewalls and virtual networks → Add existing virtual network**. With the Service Endpoint in place, requests from Monad are recognized as coming from an allowed virtual network instead of an unmatchable private IP. 3. Save. Access from Monad should succeed once the rule propagates. ### If the portal won't let you add the subnet The Azure portal only lets you add a virtual network/subnet you have read access to. Because the Monad cluster's VNet is Monad-managed, you likely won't see it in the portal picker. Add the rule with the Azure CLI instead: ```bash az storage account network-rule add \ --resource-group \ --account-name \ --subnet ``` Monad Customer Support will provide the ``. :::note This affects any Monad deployment running on Azure (AKS) — shared-tenancy SaaS as well as dedicated clusters — whenever your Azure Storage account is in the same Azure region as the AKS cluster running the connector. Deployments where Monad and the storage account are in different regions are unaffected (public-IP allowlisting works normally there). ::: ## Configuration ### Settings | Setting | Type | Required | Description | | ------- | ---- | -------- | ----------- | | Queue URL | string | Yes | Full URL of the Storage Queue that Event Grid delivers `BlobCreated` events to (e.g., `https://mystorageaccount.queue.core.windows.net/blob-events`). | | Tenant ID | string | Yes | The Azure Entra ID tenant (directory) ID where your application is registered. | | Format | string | Yes | File format of the referenced blobs. Supported values: `json`, `jsonl`, `csv`, `delimited`, `wsv` (default: `json`). | | Compression | string | Yes | Compression format of the referenced blobs. Supported values: `auto`, `gzip`, `none`. | | Record Location | string | No | Location of the record in the JSON blob. JSON only; leave empty for the entire record. See [Record Location](../../guides/record-location) for syntax and examples. | | Key Filter | object | No | Optional filter applied to the blob name parsed from each event, before fetch. See [Key Filter](#key-filter) below. | | With Metadata | boolean | No | When enabled, annotates each record with its source account, container, and blob name under `_monad_metadata`. Default: `false`. | | Use Synthetic Data | boolean | No | Generate synthetic demo data instead of connecting to the real storage account. | ### Secrets | Secret | Type | Required | Description | | ------ | ---- | -------- | ----------- | | Client ID | string | Yes | The application (client) ID from your Azure Entra ID app registration. | | Client Secret | string | Yes | The client secret value from your Azure Entra ID app registration. | ### Key Filter Filters the blob name (parsed from the event) before the connector fetches it. Useful when the queue receives events for a wider set of blobs than this connector should process. Matching is case-insensitive on both the blob name and the configured value. Key Filter is a discriminated union on the `type` field — the empty case is `{"type": "none"}` rather than an empty `{mode, operator, value}` object, so the per-field rules only apply when filtering is actually requested: | Field | Type | Required | Description | | ----- | ---- | -------- | ----------- | | `type` | string | Yes | `none` passes every blob through; `filter` enables the rules in `filter` below. | | `filter` | object | When `type` is `filter` | The match parameters described below. Omit when `type` is `none`. | When `type` is `filter`, the nested `filter` object takes: | Field | Type | Required | Description | | ----- | ---- | -------- | ----------- | | `mode` | string | Yes | `include` processes only matching blobs; `exclude` skips matching blobs. | | `operator` | string | Yes | One of `contains`, `ends_with`, `starts_with`. | | `value` | string | Yes | The substring (or suffix/prefix) to match against the blob name. | Examples: - No filtering — `{ "type": "none" }` - Process only gzipped JSON files — `{ "type": "filter", "filter": { "mode": "include", "operator": "ends_with", "value": ".json.gz" } }` - Skip a noisy prefix — `{ "type": "filter", "filter": { "mode": "exclude", "operator": "starts_with", "value": "internal/" } }` When the field is omitted entirely (or set to `{"type": "none"}`), every blob referenced by the queue is processed. ## Example Blob Event Message Event Grid delivers each event to the queue base64-encoded; after decoding, the message body is a single-element array. The connector accepts both the Event Grid and CloudEvents schemas. Event Grid schema: ```json [{ "topic": "/subscriptions/{subscription-id}/resourceGroups/Storage/providers/Microsoft.Storage/storageAccounts/my-storage-account", "subject": "/blobServices/default/containers/security-logs/blobs/2026/10/08/app.jsonl", "eventType": "Microsoft.Storage.BlobCreated", "eventTime": "2026-10-08T18:41:00.9584103Z", "id": "831e1650-001e-001b-66ab-eeb76e069631", "data": { "api": "PutBlob", "contentLength": 524288, "blobType": "BlockBlob", "url": "https://my-storage-account.blob.core.windows.net/security-logs/2026/10/08/app.jsonl", "sequencer": "00000000000004420000000000028963" }, "dataVersion": "", "metadataVersion": "1" }] ``` CloudEvents 1.0 schema (same `data`, different envelope — `type`/`time`/`source`/`specversion`): ```json [{ "source": "/subscriptions/{subscription-id}/resourceGroups/Storage/providers/Microsoft.Storage/storageAccounts/my-storage-account", "subject": "/blobServices/default/containers/security-logs/blobs/2026/10/08/app.jsonl", "type": "Microsoft.Storage.BlobCreated", "time": "2026-10-08T18:41:00.9584103Z", "id": "831e1650-001e-001b-66ab-eeb76e069631", "data": { "api": "PutBlob", "contentLength": 524288, "blobType": "BlockBlob", "url": "https://my-storage-account.blob.core.windows.net/security-logs/2026/10/08/app.jsonl" }, "specversion": "1.0" }] ``` The connector parses these events and downloads the referenced blob for processing. ## Important Notes - **Event Filtering**: Only `Microsoft.Storage.BlobCreated` events are processed; `BlobDeleted`, `BlobRenamed`, `BlobTierChanged`, and directory events are ignored. - **Commit Filtering**: Only fully-committed block-blob writes (`data.api` of `PutBlob`, `PutBlockList`, or `CopyBlob`) are fetched, so partial or in-progress uploads are never processed. **ADLS Gen2 (hierarchical namespace) is not supported** — its `FlushWithClose` commit is intentionally excluded, because append-then-flush would re-read and re-emit the whole file on every flush (unbounded duplication for continuously-appended files). Use write-once block blobs. - **At-least-once Delivery**: Event Grid guarantees at-least-once delivery and can deliver out of order, so duplicate events are possible. A message that fails repeatedly is **never dropped** — it stays in the queue and is retried indefinitely; the connector logs a warning once a message has been retried many times so a stuck message stays visible. - **Multiple Containers/Accounts**: The account, container, and blob are read from each event's `data.url`, so a single connector can serve many containers — even across storage accounts — as long as the application has **Storage Blob Data Reader** on each. - **Message Deletion**: Queue messages are deleted only after the referenced blob is successfully processed. On failure, the message remains in the queue and is retried. ## Related Articles - [Reacting to Azure Blob storage events](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-event-overview) - [Azure Blob Storage as an Event Grid source (event schema)](https://learn.microsoft.com/en-us/azure/event-grid/event-schema-blob-storage) - [Route Event Grid events to Azure Queue Storage](https://learn.microsoft.com/en-us/azure/event-grid/custom-event-to-queue-storage) - [Azure RBAC Roles for Storage](https://learn.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-portal) ## Sync frequency This input consumes from the source queue or subscription, ingesting records at whatever rate the source delivers them rather than on a fixed poll. See [Input Sync Frequency](/docs/guides/sync-frequency) for details.