Collecting Azure Resource Logs Without Creating Noise

Thu, 24 Sep 2026 00:00:00 +1000

Once the Azure control plane is visible, the next question is which resource-level signals deserve to be collected. This is where many telemetry programs become expensive and disappointing. Teams enable every diagnostic category they can find, send it all to one workspace, and then discover that most of the data is not tied to a detection, investigation, or assurance use case.

A good Azure telemetry design is selective by intent, not selective by accident.

Diagnostic settings are the bridge between Azure resources and operational destinations such as Log Analytics, Event Hubs, and storage accounts. They are powerful, but they need architecture. Collection should be driven by risk, investigation paths, and response ownership.

The Mental Model

Think of resource logs as evidence from the services that carry business risk. The control plane tells you that a resource changed. Resource logs tell you how that resource behaved. For security telemetry, the question is not “can this service emit logs?” The question is “which emitted events would change a detection, investigation, or governance decision?”

This shifts the conversation from log categories to decisions. For a key vault, useful questions include who accessed secrets, whether access was denied, whether purge protection matters, and whether administrative changes align with policy. For a firewall or application gateway, useful questions include what was allowed, what was denied, which rules matched, and whether traffic patterns changed. For storage, useful questions include access paths, authentication methods, network source, and data access patterns.

The mental model is signal portfolio management. Some logs are high-value and low-volume. Some are high-value but expensive. Some are operationally useful but rarely security relevant. Some only matter during a major incident. Treating them all equally creates cost pressure and weakens analyst focus.

An architecture-first approach classifies resource logs into tiers: mandatory security evidence, conditional investigation support, operational-only telemetry, and excluded noise. That classification should be visible in platform standards, not hidden in individual diagnostic settings.

How It Really Works

Diagnostic settings define which log categories and metrics are collected from a resource and where they are sent. A single resource can have diagnostic settings that route data to destinations such as a Log Analytics workspace, an Event Hubs namespace, or a storage account. Each destination supports a different operating model.

flowchart TD A[Azure Resource] --> B[Diagnostic Setting] B --> C[Log Analytics Workspace] B --> D[Event Hubs] B --> E[Storage Account] C --> F[Sentinel Detections] D --> G[SIEM or Data Pipeline] E --> H[Archive and Retention]

Log Analytics is commonly used when teams need query, detection, workbook, and Microsoft Sentinel integration. Event Hubs is useful when telemetry must be streamed to another platform, a downstream security data lake, or a central enterprise pipeline. Storage can support archive and compliance retention patterns, although it is less convenient for active detection.

The critical design choice is not only destination. It is category selection. Azure services expose different diagnostic categories, and not all categories have equal security value. A storage account, key vault, firewall, application gateway, Kubernetes service, or database service will each require a different collection decision. Copying the same diagnostic template everywhere is rarely correct.

At scale, deployment matters. Azure Policy can deploy diagnostic settings for supported resources and can audit resources that are missing expected settings. Infrastructure as code can also define diagnostic settings alongside the resource. The most mature pattern is to combine both: templates create the expected configuration at deployment time, and policy detects drift or resources created outside the normal path.

Cost management must be part of the design. Ingestion volume, retention, table plan choices, and query behavior all affect the operating cost. Security teams need enough telemetry to investigate and detect, while platform teams need predictable cost boundaries. That trade-off should be explicit.

Real-World Impact

Diagnostic setting design directly affects incident response. During a suspected data exposure, the team may need to know whether storage data access logs were collected, whether key vault access was visible, whether firewall traffic was retained, and whether the data is queryable in the same workspace as related identity and control-plane events.

If the logs are missing, the investigation becomes a reconstruction exercise. If the logs exist but are split across disconnected destinations, response slows down. If the logs exist but are buried under unclassified noise, analysts spend time filtering rather than deciding. A deliberate collection design improves all three outcomes.

There is also a governance impact. When diagnostic settings are standardized, security and platform leaders can measure telemetry coverage. They can report which subscriptions and resource classes meet the logging baseline, where exceptions exist, and whether those exceptions have owners. That is much stronger than a general statement that “logging is enabled.”

For detection engineering, resource logs unlock higher-fidelity analytics. A network rule change from Activity Log is useful. A rule change followed by unusual allowed traffic from a firewall is more useful. A key vault configuration change is useful. The same change followed by denied secret reads, successful reads from a new identity, or access from an unexpected network is more useful again.

The impact is not just better alerting. It is better prioritization. Resource logs help teams decide which control-plane changes matter most because they show whether the affected service behaved differently afterwards.

Gotchas and Edge Cases

Diagnostic settings can fail as an operating model when ownership is unclear. Security may define the requirement, platform may own policy, application teams may own the resource, and operations may own the workspace. Without clear responsibility, gaps persist because everyone assumes another team is handling them.

Another gotcha is over-collection. Sending every metric and log category to a central workspace can create cost and performance issues without improving detection. More data can actually reduce security value if analysts cannot tell which tables matter or if high-value events are buried under routine operational records.

Service differences are also important. Azure services do not all expose the same diagnostic categories, and category names can evolve. A policy that works for one resource type may not apply to another. Treat resource classes individually and review high-risk services more often.

Log size and limits matter. Some destinations and services impose practical limits, and large records or high-volume categories can behave differently than expected. Defender for Cloud continuous export, for example, notes that Log Analytics supports records up to a specific size limit. That kind of detail matters when designing downstream pipelines.

Finally, do not forget the security of the telemetry path itself. A workspace, event hub, or storage account receiving security logs is a high-value target. If an attacker can modify diagnostic settings, delete data, or change routing, they can damage your evidence trail. Protect the collection pipeline with the same seriousness as production infrastructure.

Best Practices

Start by defining a resource telemetry baseline for high-risk Azure services. Key Vault, Storage, networking services, identity-adjacent services, Kubernetes, databases, and internet-facing services usually deserve early attention. For each resource class, document the mandatory categories, optional categories, destination, retention, and reason for collection.

Use policy to audit and deploy diagnostic settings where supported. Pair that with infrastructure as code so new resources are born with the expected telemetry configuration. Review policy exemptions as security exceptions, not administrative clutter.

Route telemetry based on use case. Send detection and investigation data to Log Analytics and Sentinel. Stream enterprise integration data to Event Hubs where needed. Use archive destinations for long-term retention requirements. Avoid one-size-fits-all routing.

Create a cost feedback loop. Track ingestion by resource type, category, subscription, and owner. If a category is expensive, either prove its security value or redesign the route. Cost control should not mean blind deletion; it should mean evidence-based collection.

Test the baseline with incident scenarios. Pick scenarios such as secret access, public exposure, firewall rule changes, and suspicious administrative updates. Confirm that the necessary logs are present, queryable, retained, and understandable by the teams who will use them.

Practical tiered diagnostic-settings baseline

The implementation pattern I prefer is a resource-class baseline with two tiers: mandatory evidence that is always collected, and investigative telemetry that is enabled only where the service, risk, and cost model justify it. This Key Vault example uses the Microsoft.Insights/diagnosticSettings@2021-05-01-preview Bicep resource, the Microsoft-documented Audit category group for mandatory evidence, and allLogs for the deeper tier.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


param keyVaultName string
param logAnalyticsWorkspaceId string

@allowed([
 'mandatoryEvidence'
 'investigative'
])
param telemetryTier string = 'mandatoryEvidence'

resource keyVault 'Microsoft.KeyVault/vaults@2023-07-01' existing = {
 name: keyVaultName
}

resource diagnosticSetting 'Microsoft.Insights/diagnosticSettings@2021-05-01-preview' = {
 name: 'set-by-platform-security-baseline'
 scope: keyVault
 properties: {
 workspaceId: logAnalyticsWorkspaceId
 logAnalyticsDestinationType: 'Dedicated'
 logs: telemetryTier == 'mandatoryEvidence' ? [
 {
 categoryGroup: 'Audit'
 enabled: true
 }
 ] : [
 {
 categoryGroup: 'allLogs'
 enabled: true
 }
 ]
 metrics: [
 {
 category: 'AllMetrics'
 enabled: false
 }
 ]
 }
}

At scale, wrap this decision in Azure Policy DeployIfNotExists assignments and remediation tasks, but do not pretend one diagnostic setting fits every resource provider. Category groups are not universal, category names can differ by service, and allLogs can change cost materially. The baseline should say which resource classes require audit evidence, which classes justify investigative categories, and which teams own exceptions.

🍺

Brewed Insight: Diagnostic settings aren’t a checkbox to tick during deployment. They’re the contract between Azure services and the security decisions your organization is going to need to make later - usually at 2am, during an incident, when it’s too late to go back and turn on the log you needed.

Log Analytics on Brewed in the Cloud by Chris Hailes