Tools: Solved: What’s the most unexpectedly expensive thing in your Azure bill lately?

🚀 Executive Summary ## 🎯 Key Takeaways ## That Time Our Monitoring Bill Was Bigger Than Our Compute Bill ## Why Your Bill is Exploding: The Silent Data Tsunami ## Taming the Beast: Three Ways to Fix Your Log Analytics Bill ## 1. The Quick Fix: Stop the Bleeding, Now! ## 2. The Permanent Fix: From Fire Hose to Scalpel ## 3. The ‘Nuclear’ Option: Re-architect for Cost ## Final Thoughts from the Trenches TL;DR: Azure Log Analytics can unexpectedly become a major cloud expense due to default diagnostic settings ingesting excessive data. This guide provides three strategies, from quick fixes like setting daily caps to architectural changes using hot/cold data paths, to effectively manage and reduce data ingestion costs. Azure’s Log Analytics can silently become your biggest cloud expense if not properly managed. This guide offers three practical strategies, from quick fixes to permanent architectural changes, to control runaway data ingestion costs and get your bill back in line. I remember it like it was yesterday. We’d just rolled out a slick new microservice, ‘auth-svc-prod’. One of our sharp junior engineers, let’s call him Ben, had done a textbook job setting up monitoring. He wired up App Insights, enabled diagnostic settings on every related resource, and pointed it all to a central Log Analytics Workspace. Pat on the back, job done. Or so we thought. A month later, I get a Slack message from finance with a screenshot of our Azure bill and one simple question: “What is ‘Log Ingestion’ and why does it cost more than the VMs for our entire production environment?” Ben hadn’t done anything wrong. In fact, he’d followed the documentation perfectly. And that, right there, is the trap. Welcome to the silent killer of Azure budgets: default diagnostic logging. When you’re standing up a new environment, it’s tempting to just click the “Send to Log Analytics” button and enable all categories. Azure makes it easy. The problem is that many services are incredibly chatty by default. Your App Service is logging every single successful health probe ping. Your firewall is logging every single packet it allows. Your storage account is logging every successful read operation. Individually, these are tiny drops of data. But when you have millions of transactions an hour, those drops become a fire hose aimed directly at your Log Analytics Workspace, and you pay for every single gigabyte that comes through the door. The root cause isn’t a bug; it’s a misalignment of priorities. The default settings are optimized for maximum visibility, not for cost. It’s on us, the engineers in the trenches, to tune that fire hose down to a manageable—and valuable—stream of data. Panicking won’t help, but quick action will. Here are the three levels of response I use when I see that dreaded cost alert. Your first job is to triage the wound. We’re not doing surgery yet; we’re applying a tourniquet. The goal here is to cap the cost immediately so you have time to think. Warning: The daily cap is a blunt instrument. It will stop all data ingestion once the limit is reached, which means you could lose critical security or error logs. Use this as a temporary measure to give yourself breathing room, not a long-term solution. Once the immediate bleeding has stopped, it’s time for proper surgery. This is about being intentional with the data you collect. You need to go back to the source—the Diagnostic Settings for each resource. Instead of checking the “AllMetrics” and “AllLogs” boxes, you need to go through the list and select only the categories that provide real business or operational value. Do you really need to log every successful request to your prod-web-app-01? Probably not. But you absolutely need the AppServiceHTTPLogs for 4xx and 5xx errors. A good approach looks like this: Pro Tip: Look into using Data Collection Rules (DCRs). They are the future of Azure Monitor. DCRs let you apply a KQL transformation to your data *before* it gets ingested and billed. You can use them to drop noisy columns or filter out entire log entries (like 200 OK health checks) at the source. This is the most powerful tool in your cost-optimization arsenal. Sometimes, even with filtering, the volume of data you’re required to keep for compliance is just too high to be cost-effective in a “hot” Log Analytics Workspace. This is when you have to rethink the architecture. The strategy is to split your data streams based on their purpose: This is more work to set up, but for large-scale environments, it’s the only sustainable way to keep a lid on logging costs without sacrificing compliance or visibility. That surprise bill was a painful but valuable lesson for our team. It forced us to stop treating logging as an afterthought and start treating it as a critical piece of our application architecture. Don’t wait for the finance team to come knocking. Be proactive. Go look at your Usage table today. You might be surprised by what you find. 👉 Read the original article on TechResolve.blog If this article helped you, you can buy me a coffee: 👉 https://buymeacoffee.com/darianvance Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse COMMAND_BLOCK: // Run this in your Log Analytics Workspace to find the biggest data hogs Usage | where TimeGenerated > ago(30d) | where IsBillable == true | summarize BillableDataGB = sum(Quantity) / 1000 by DataType | sort by BillableDataGB desc Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: // Run this in your Log Analytics Workspace to find the biggest data hogs Usage | where TimeGenerated > ago(30d) | where IsBillable == true | summarize BillableDataGB = sum(Quantity) / 1000 by DataType | sort by BillableDataGB desc COMMAND_BLOCK: // Run this in your Log Analytics Workspace to find the biggest data hogs Usage | where TimeGenerated > ago(30d) | where IsBillable == true | summarize BillableDataGB = sum(Quantity) / 1000 by DataType | sort by BillableDataGB desc - Default diagnostic logging in Azure services often leads to unexpectedly high Log Analytics ingestion costs by sending non-critical data. - The ‘Quick Fix’ involves identifying noisy data types using KQL queries and setting a temporary daily ingestion cap in Log Analytics Workspace settings. - The ‘Permanent Fix’ requires granular control over Diagnostic Settings, selecting only valuable log categories, and leveraging Data Collection Rules (DCRs) for pre-ingestion data transformation and filtering. - The ‘Nuclear Option’ involves re-architecting by splitting data streams into a ‘Hot Path’ (lean Log Analytics for real-time needs) and a ‘Cold Path’ (Azure Storage Account for cost-effective long-term retention). - Step 1: Find the Noisiest Tables. Run a KQL query in your workspace to see which data types are costing you the most money. It’s almost always one or two offenders. - Step 2: Set a Daily Cap. In the Log Analytics Workspace settings, under “Usage and estimated costs”, you can set a daily ingestion cap. When the cap is hit, Azure stops ingesting data for the rest of the day. - Hot Path (Expensive): A lean Log Analytics Workspace that only ingests high-priority data needed for real-time alerting and interactive dashboards. Think application errors, security alerts, and key performance indicators. Retention might be 30-90 days. - Cold Path (Cheap): For everything else—verbose application logs, network flow logs, compliance data—that you need to keep but rarely need to query. Instead of sending this to Log Analytics, you send it directly to an Azure Storage Account. It’s orders of magnitude cheaper. If you ever need to analyze it, you can query it in-place with Azure Data Explorer or rehydrate it on-demand.