Skip to main content

Azure Virtual Network Flow Logs

Ingests Azure Virtual Network Flow Logs from Azure Blob Storage to analyze network traffic patterns and security events.

Sync Type: Incremental

Requirements​

Before setting up the Azure Virtual Network Flow Logs input, you need to:

  1. Have a Microsoft Account with an active Azure Subscription
  2. Create an Azure Entra ID application with appropriate permissions
  3. Configure Azure Virtual Network Flow Logs to be stored in an Azure Storage Account
  4. Grant the application access to the Azure Storage Account containing flow logs

Details​

The Azure Virtual Network Flow Logs input allows you to collect and ingest network flow logs generated by Azure Virtual Networks. These logs provide detailed information about network traffic flowing through your Azure virtual networks, including source and destination IP addresses, ports, protocols, and traffic decisions (allow/deny).

Flow logs are stored in Azure Blob Storage in a structured format and are automatically partitioned by date and time. The input processes these logs incrementally, ensuring that only new data is ingested since the last successful run.

Note: This input fetches logs from t-24h on the first sync. Subsequent syncs follow the pattern above.

How It Works​

When the input is run for the first time, it will process flow logs from the current day. After every successful sync, a timestamp of the last successful fetch is saved. This timestamp is used to incrementally pull only new data, ensuring efficient processing and avoiding duplicates.

The input:

  • Connects to the specified Azure Storage Account
  • Reads flow log files from the insights-logs-flowlogflowevent container
  • Processes JSON-formatted flow log records
  • Maintains state to track progress and enable incremental syncing
  • Includes a 75-minute data availability delay to account for Azure's log processing time

Setting up Azure Virtual Network Flow Logs​

Step 1: Enable Flow Logs on Your Virtual Network​

  1. Navigate to the Azure portal
  2. Go to Network Watcher service
  3. In the left sidebar, select Flow logs under Logs
  4. Click + Add to create a new flow log
  5. Configure the flow log:
    • Target resource: Select your Virtual Network or specific subnets
    • Storage account: Choose or create a storage account for storing logs
    • Retention (days): Set retention period as needed
    • Flow logs version: Use Version 2 for enhanced features
    • Enable Traffic Analytics: Optional, for additional insights
  6. Click Create to enable flow logging

Step 2: Create Azure Entra ID Application​

  1. Navigate to App Registrations in the Azure portal
  2. Click New Registration
  3. Provide a name for the application (e.g., "Monad Flow Logs Reader")
  4. Select appropriate account types (typically "Accounts in this organizational directory only")
  5. Click Register
  6. Save the Application (client) ID and Directory (tenant) ID from the overview page
  7. Go to Certificates & secrets in the left sidebar
  8. Under Client secrets, click New client secret
  9. Add a description and set expiration period
  10. Click Add and save the Value (this is your client secret)

Step 3: Grant Storage Account Access​

The application needs permission to read from the storage account containing flow logs:

  1. Navigate to your Storage Account in the Azure portal
  2. Click Access Control (IAM) in the left sidebar
  3. Click + Add > Add role assignment
  4. Select the Storage Blob Data Reader role
  5. Click Next
  6. Under Assign access to, select User, group, or service principal
  7. Click + Select members
  8. Search for your application name and select it
  9. Click Select, then Review + assign
  10. Click Review + assign again to confirm

Step 4: Verify Flow Log Storage Structure​

Flow logs are automatically stored in your storage account with the following structure:

https://{storageAccountName}.blob.core.windows.net/insights-logs-flowlogflowevent/flowLogResourceID=/{subscriptionID}_NETWORKWATCHERRG/NETWORKWATCHER_{Region}_{ResourceName}-{ResourceGroupName}-FLOWLOGS/y={year}/m={month}/d={day}/h={hour}/m=00/macAddress={macAddress}/PT1H.json

Configuration​

Settings​

SettingTypeRequiredDescription
Subscription IDstringYesThe Azure subscription ID where the virtual network and storage account are located
Tenant IDstringYesThe Azure Entra ID tenant (directory) ID
Storage Account URLstringYesThe Azure storage account URL where flow logs are stored (format: https://accountname.blob.core.windows.net)
RegionstringYesThe Azure region where the virtual network is located
Virtual Network NamestringYesThe name of the virtual network for which flow logs are being collected
Resource Group NamestringYesThe name of the resource group containing the virtual network

Secrets​

SettingTypeRequiredDescription
Client IDstringYesThe application (client) ID registered in Azure Entra ID
Client SecretstringYesThe client secret associated with the registered application in Entra ID

Troubleshooting​

Common Issues​

  1. Connection Failed: Verify that the storage account URL is correct and the application has proper permissions
  2. No Data Found: Ensure that flow logs are enabled on your virtual network and data is being written to the storage account
  3. Authentication Errors: Check that the Tenant ID, Client ID, and Client Secret are correct and the application exists in the specified tenant
  4. Permission Denied: Verify that the application has "Storage Blob Data Reader" role on the storage account

Data Availability​

  • Input Connector Delay: This input connector includes a 75-minute delay before processing flow logs. While Azure flow logs are typically available in the storage account within a few minutes, this connector waits 75 minutes to ensure all data for a given time period has been fully written and is available for processing.
  • The delay helps ensure data completeness and consistency during ingestion

Sample Record​

{
  "category": "FlowLogFlowEvent",
  "flowLogGUID": "e4c5243c-1499-9049-49b6-c37e475461ba",
  "flowLogResourceID": "/SUBSCRIPTIONS/B1R5T891-1234-4QAE-A218-3CFEE98B2F49/RESOURCEGROUPS/NETFLOW-LOGS/PROVIDERS/MICROSOFT.NETWORK/NETWORKWATCHERS/NETWORKWATCHER_SOUTHEASTASIA/FLOWLOGS/SECURITY-FLOWLOG",
  "flowLogVersion": 3,
  "flowRecords": {
    "flows": [
      {
        "aclID": "11111111-1111-1111-1111-111111111111",
        "flowGroups": [
          {
            "flowTuples": [
              "1750561744331,52.168.1.15,10.1.0.5,47650,41856,6,I,E,NX,1,13,4,851"
            ],
            "rule": "PlatformRule"
          }
        ]
      },
      {
        "aclID": "/subscriptions/a2d4e6f8-1234-5678-9abc-def012345678/resourceGroups/security-logs/providers/Microsoft.Network/networkSecurityGroups/web-server-nsg",
        "flowGroups": [
          {
            "flowTuples": [
              "1758478816929,172.16.0.10,57.150.149.97,32852,443,6,O,E,NX,19,4143,11,54390",
              "1754034146064,172.16.0.10,13.83.125.0,56497,8080,6,O,B,D,3,5094,3,2393",
              "1756586830496,10.0.0.4,20.189.173.5,43852,443,6,O,B,NX,19,3443,25,2688"
            ],
            "rule": "DefaultRule_AllowInternetOutBound"
          }
        ]
      },
      {
        "aclID": "/subscriptions/b131b891-1234-4aaa-b018-3cfee98b2f49/resourceGroups/netflow-logs/providers/Microsoft.Network/networkSecurityGroups/database-nsg",
        "flowGroups": [
          {
            "flowTuples": [
              "1752977686288,152.58.35.136,10.1.0.5,63848,22,6,I,B,D,46,19412,194,3199",
              "1759841483868,203.0.113.5,172.16.0.10,63378,22,6,I,B,A,79,29237,159,10015"
            ],
            "rule": "UserRule_SSH"
          },
          {
            "flowTuples": [
              "1755669293961,198.51.100.42,172.16.0.10,45874,8000,6,I,B,D,7,142,7,557",
              "1757608612538,152.58.35.136,172.16.0.10,39685,8000,6,I,E,NX,0,1650,7,280",
              "1757338739829,198.51.100.42,172.16.0.10,35100,80,6,I,E,D,8,1925,4,197"
            ],
            "rule": "UserRule_AllowHTTP"
          },
          {
            "flowTuples": [
              "1759870061365,89.248.163.133,10.1.0.5,64868,57054,6,I,D,A,0,0,0,0",
              "1754063532239,147.185.132.89,172.16.0.10,44856,36810,17,I,D,NX,0,0,0,0",
              "1758439621020,148.113.210.254,10.0.0.4,52950,30420,17,I,D,D,0,0,0,0"
            ],
            "rule": "DefaultRule_BlockMalicious"
          }
        ]
      }
    ]
  },
  "macAddress": "112233445566",
  "operationName": "FlowLogFlowEvent",
  "targetResourceID": "/subscriptions/b1d33822-1234-4aeb-b498-3dfe198b2f49/resourceGroups/netflow-logs/providers/Microsoft.Network/virtualNetworks/SecurityVNet",
  "time": "2025-08-11T23:46:43.288886Z"
}