Case Study 1: Thirty-Six Percent of the Bill Belonged to Nobody

"Finance asked a reasonable question — which team spent this? — and it took us three weeks to discover that the honest answer for a third of it was 'none of them.'"

Executive Summary

Kestrel's monthly cloud bill was $30,511.36. Finance asked for it split by team, for a budgeting exercise. The data platform team could split it by service and could not split it by team.

Three weeks of tagging and reconstruction produced the answer, and the answer was not the one anyone expected:

data-platform          $14,050.66    46.0%
analytics               $4,435.20    14.5%
data-science              $922.80     3.0%
** UNATTRIBUTED **     $11,102.70    36.4%

$11,102.70 a month had no owner, and every dollar of it turned out to be waste — a dev cluster that never stopped, a warehouse created for a finished migration, two storage prefixes nobody had read in over a year.

The finding was not the total and it was not the largest line. It was that unattributed spend is enriched in waste, because a resource is unowned for the same reason it is wasted: nobody is watching it.

Two quarters later the bill was $17,286.23 and the unattributed share was 4.1%.

Skills applied: cost attribution and tagging (§33.5); the seven wastes (§33.7); ranking by payoff per week (§33.8); and the organizational move that made the number actually fall.

Background

The request was routine and the platform team could not answer it.

What existed: a cloud provider console showing spend by service, a Snowflake usage view showing credits by warehouse, and no connection between either and a team.

Why the split was impossible, and none of the three reasons is unusual:

Shared resources. One Snowflake account, four warehouses, used by everyone. One S3 bucket with prefixes that mostly but not entirely map to teams.

Untagged resources. A tagging policy existed, was applied to about 60% of resources, and had been introduced eighteen months after the platform.

Resources created by tools. Airflow launches ephemeral EMR clusters; the launcher did not propagate tags. Every cluster the orchestrator created was untagged, which was most of the compute.

⚠️ Failure Mode — a tagging policy fixes the future and nothing else

Kestrel had a tagging policy. It was well-written, it was enforced in Terraform, and it explained only 60% of the bill.

Three gaps, and they are the same three everywhere:

• Resources predating the policy. Never retro-tagged, because retro-tagging is nobody's project.
• Resources created outside Terraform. Console changes during incidents (Chapter 28 §28.10's drift, arriving on the cost side), and every one made at 3am by someone with a more urgent problem.
• Resources created by tools. The largest gap by far. The orchestrator, the ML platform, and the CI system all created compute, and none propagated tags, because tag propagation is a feature nobody asks for until they try to split a bill.

The important consequence: the untagged resources are not a random sample. They are systematically the old ones, the ones created under pressure, and the ones created by automation nobody reviews. Which is exactly the population where waste accumulates, and it is why §33.5 claims the unattributed share is a waste-finding metric rather than an accounting one.

A policy that will explain 100% of next year's bill explains 60% of this one, and this one is the bill you were asked about.

The Problem

Three weeks of work produced the attribution, mostly by hand: reading Terraform, matching Airflow DAGs to clusters, mapping bucket prefixes to teams, and asking people.

And it produced the 36.4%.

What was in it:

dev cluster (always on)                $5,184.00    3 nodes, 24x7
REPORTING_WH (auto-suspend off)        $5,760.00    Medium, never suspends
backups/2023/                            $110.40    4,800 GB, last read: never
_scratch/                                 $48.30    2,100 GB, 91 prefixes
                                     ───────────
                                      $11,102.70

Each has a story, and the stories rhyme.

The dev cluster was created for a two-week Spark evaluation nineteen months earlier. It was used — occasionally, by about six people, for genuinely useful things — which is precisely why nobody suggested deleting it and nobody claimed it either.

REPORTING_WH was created during the Chapter 21 migration, with auto-suspend disabled because the cold-start latency was annoying during a period of constant iteration. The migration ended. The warehouse did not.

backups/2023/ was a manual backup taken before a schema change, by an engineer who has since changed teams. Never read. Never deleted.

_scratch/ was 91 prefixes from 91 different pieces of ad-hoc work.

🔎 Read the Plan — the common structure of all four

Every one was created deliberately, for a good reason, by a competent person — and then the reason ended and nothing happened.

The asymmetry that produces this is the same one Chapter 30 §30.5 found in permissions and Chapter 25 Case Study 2 found in alerting:

text creating it: has a clear benefit, a person who wants it, and a deadline removing it: has a diffuse benefit, no requester, and a small risk that somebody was using it

Removal is a task with negative expected value to the individual doing it — it takes their time, saves money that appears on nobody's evaluation, and carries a small chance of breaking something and being blamed. So it does not happen, and this is not a culture problem or a discipline problem. It is an incentive that is pointing the wrong way.

Which is why the fix in this case study is organizational rather than technical. Kestrel's team could have written a script to find idle resources in an afternoon — and did, eventually, and it is waste.py. But the script had been possible for nineteen months and nobody had written it, because nobody would have been the one to act on its output.

The change that mattered was making the unattributed bucket somebody's budget line. Not the script.

The Analysis

With the attribution in hand, waste.py sized seven patterns (§33.7):

pattern                                $/month        $/year   weeks
─────────────────────────────────────────────────────────────────────
warehouse auto-suspend disabled       $5,040.00    $60,480.00    0.02
dev cluster never stopped             $3,916.80    $47,001.60    0.20
over-frequent dashboard refresh       $2,059.20    $24,710.40    0.10
cross-region egress                   $1,656.00    $19,872.00    1.00
raw JSON retained past need             $216.83     $2,601.99    0.50
small-file GET amplification            $177.60     $2,131.20    2.00
orphaned storage                        $158.70     $1,904.40    1.00
─────────────────────────────────────────────────────────────────────
TOTAL                                $13,225.13   $158,701.56

$13,225.13 a month — 43.3% of the bill — and 83.3% of it was idle rather than inefficient.

The two findings that changed the plan:

Storage was 2.1% of the bill. The team's existing cost backlog — written before any of this — had four storage items and one compute item. "Compress the clickstream," "lifecycle old Parquet," "delete duplicate bronze," "evaluate Glacier." Together they were worth under $300 a month against a $30,511 bill, and they were the backlog because storage is easy to reason about and upsets nobody (§33.4).

And the ranking by payoff inverted the order.

📐 Design Decision — the two-day sprint that was worth $132,192 a year

Ranked by size, the plan was: fix the auto-suspend, then the dev cluster, then the dashboard. Ranked by annual saving per week of work, the middle two swap — and more importantly, the top three separate from everything else by an enormous margin:

text auto-suspend $60,480/yr 0.02 weeks $3,024,000 per engineer-week dashboard refresh $24,710/yr 0.10 weeks $247,104 dev cluster $47,002/yr 0.20 weeks $235,008 ──────────────────────────────────────────────────────── cross-region $19,872/yr 1.00 weeks $19,872 <- 12x lower

The top three are $132,192 a year for under two days of work combined. The team did all three in one afternoon and one morning.

What the ranking made visible that the size ordering hid: small-file GET amplification is $1,065.60 per engineer-week — a factor of 2,838 below the top item, and only four rows away on a size-ranked list. It had been near the top of the backlog for a year because "we have a small file problem" is a recognizable engineering statement and "the warehouse doesn't suspend" is not.

Compaction was still done, six weeks later — and justified on query latency and on Chapter 31's deletion story, which are its real benefits. The $2,131 of GET savings was removed from the proposal, because a weak cost argument attached to work with a strong non-cost argument invites the rejection of both.

The Decision

Five changes, and only the second one is the reason the number moved.

One: the two-day sprint. Auto-suspend on REPORTING_WH (a single setting), a schedule on the dev cluster (stop at 19:00, start at 08:00 on weekdays), and the dashboard refresh interval.

Two: the unattributed bucket is charged to the platform team.

🧱 Kestrel Platform — the one change that made the metric fall

Before: the unattributed share was a number in a monthly report. It was 36.4%, it was interesting, and it stayed at 36.4% for two months while everyone agreed it was too high.

After: the unattributed spend is added to the data platform team's budget line, in full, every month.

It went from 36.4% to 4.1% in two months, and the mechanism is worth being precise about because it is not really about money:

• It gave the number an owner — the one thing §30.4 says every artifact needs, applied to a liability rather than a table.
• It made investigation cheaper than absorption. A platform team carrying $11,102.70 a month of somebody else's spend will spend a week finding out whose.
• It created the right incentive for other teams too. Once the platform team started asking, other teams tagged their resources — because an untagged resource now had a person who wanted it claimed.

And the detail that surprised everyone: almost all of the reduction was deletion, not tagging.

text the 36.4%, resolved: deleted or stopped 87% tagged to a team 13%

When somebody finally has to claim a resource, most of the time nobody wants it. That is the whole case study in one line, and it is why the attribution exercise is worth doing even if you never produce a per-team budget.

Three: tag propagation in the orchestrator. The Airflow EMR launcher now stamps team, pipeline, and dag_id on every cluster it creates. This closed the largest structural gap.

Four: waste.py runs weekly, reporting the seven patterns with current numbers.

Five: a monthly report with two numbers at the top — total spend and unattributed share.

What Happened

The bill fell 43.3%, and the overwhelming majority of it came from turning things off. No query was rewritten in the first quarter. No architecture changed.

Two second-order effects the team did not anticipate:

Other teams started asking for cost reviews. Once the monthly report existed and was not accusatory (§33.12), the analytics team requested a review of their own warehouses and found $600 a month. A finding a team brings to itself lands differently from one delivered to it.

The compaction project got approved. It had been rejected twice on cost grounds. After $132,192 a year had been delivered, the same team's proposal for six weeks of compaction work — justified on latency and deletion rather than on GET requests — was approved in one meeting. Credibility bought by cheap wins funded the expensive one.

Lessons

1. Splitting a bill by team is much harder than splitting it by service, and almost no organization can do it on request.

2. ⚠️ A tagging policy fixes the future and nothing else. Kestrel's explained 60% of the bill, and the missing 40% was systematically old resources, incident-created resources, and tool-created resources — the population where waste accumulates.

3. Tag propagation in orchestrators is the largest structural gap, because it is a feature nobody asks for until they try to split a bill.

4. Unattributed spend is enriched in waste. All $11,102.70 of Kestrel's was waste, because a resource is unowned for the same reason it is wasted: nobody is watching it.

5. 🔎 Creating a resource has a requester and a deadline; removing one has neither. Removal has negative expected value to the individual doing it, which is an incentive problem rather than a discipline problem.

6. 🧱 Charging the unattributed bucket to a team is what makes the number fall — 36.4% → 4.1% in two months. Not the script, which had been writable for nineteen months.

7. 87% of the unattributed spend was resolved by deletion, 13% by tagging. When somebody finally has to claim a resource, most of the time nobody wants it.

8. The existing cost backlog was four storage items worth under $300 a month, against a $30,511 bill — because storage is easy to reason about and upsets nobody.

9. 📐 Ranking by payoff per week put $132,192 a year into two days, and revealed that small-file compaction was 2,838× worse per engineer-week than the top item while sitting four rows away on a size-ranked list.

10. Compaction was still worth doing — on latency and deletion, not on cost. A weak cost argument attached to work with a strong non-cost argument invites the rejection of both.

11. Cheap wins buy credibility for expensive ones. A twice-rejected project was approved after $132,192 had been delivered.

12. A cost review a team requests for itself lands differently from one delivered to it.

Questions for Discussion

1. The script that found the waste could have been written in an afternoon, nineteen months earlier. Why wasn't it — and what would have caused it to be?

2. Charging unattributed spend to the platform team is arguably unfair: they did not create most of it. Defend the decision, then argue against it.

3. 87% of unattributed spend was deleted rather than claimed. What does that suggest about how you should interpret a low unattributed share?

4. The existing backlog was four storage items. What made storage feel like the place to look, and what would redirect that instinct in general?

5. The dev cluster was genuinely used by six people, occasionally. Scheduling it saved $3,916.80/month and made it unavailable at 22:00. Was that the right call, and how would you find out?

6. Compaction's cost argument was removed from its proposal. Is that honest, tactical, or both?

7. The bill fell 43.3% with no query rewritten and no architecture changed. What does that say about where cost optimization effort usually goes?