Chapter 33: Cloud Cost Optimization

"The finance team asked which team spent the $30,000. We could tell them which service. We could not tell them which team, and it turned out that a third of it belonged to nobody at all."

Overview

This is the chapter the book has been building toward since page one. Chapter 1 opened with a nightly Spark job costing $3,840.00 because of one CAST in a WHERE clause, fixed to $74.88 by deleting four characters. Every chapter since has had a cost consequence, and this one collects them.

The argument of the chapter is a single claim: cloud cost is an engineering property, decided in code, in seconds, usually without noticing. Not a procurement outcome, not a vendor negotiation, and not something a finance team can fix. The people who determine the bill are the people writing queries, and most of them have never seen the bill.

Three things this chapter does that a cost dashboard does not:

It computes cost before the work runs (§33.6), which is the only point at which the information changes a decision.

It attributes the bill, and measures the share nobody claims — 36.4% at Kestrel, which is the finding rather than the total (§33.5).

It ranks fixes by payoff per week of work rather than by size (§33.8), and the two orderings disagree immediately — which matters, because everybody ranks by size.

One warning up front. Every price here is the book's frozen list price (Appendix D). Your rate card is not this one, list prices are not what large customers pay, and the numbers will have moved by the time you read this. The arithmetic is the transferable part; the constants are not.

33.1 Cost Is Decided in Code

The bill is the sum of a very large number of small engineering decisions, and almost none of them felt like a cost decision at the time.

Chapter 1's job is the canonical example. Someone wrote:

WHERE CAST(event_ts AS DATE) = '2026-03-14'

against a table partitioned by event_date. The cast defeated partition pruning, turning a 34 GB scan into 4.2 TB. 160 executors for 10.0 hours instead of 24 for 1.3.

before:  160 × 10.0 × $2.400 = $3,840.00 / night  →  $1,401,600 / year
after:    24 ×  1.3 × $2.400 =    $74.88 / night  →     $27,331 / year
                                                        ─────────
                                          saving      $1,374,269 / year

A 51.3× reduction from deleting four characters, and the person who wrote the cast was not being careless — they were being defensive about a type mismatch, which is ordinarily good practice.

Three properties of cost that follow from this, and they shape everything else in the chapter:

Cost defects are silent. No error, no alert, no failed test. The job succeeded every night for months.

Cost defects are cheap to create and expensive to find. Four characters, four months, $458,000.

Cost is not visible where it is created. The engineer writing the query does not see a number. The bill arrives thirty days later, aggregated by service, to somebody else.

🎓 Interview Angle — "how would you reduce our data infrastructure costs?"

The answer that fails is a list of tactics: reserved instances, spot, compression, lifecycle policies. Every one is real and every one is a guess about a system you have not seen.

The answer that works asks for the shape of the bill first, and it demonstrates that you know costs are not uniform:

"First I'd want the bill split by service and by team, and I'd expect a large share to be unattributable — that's usually where the easy money is. Then I'd want to know which meter dominates: per-byte-scanned, per-warehouse-second, or per-node-hour, because they reward completely different optimizations.

Before touching anything, I'd look for idle spend — a warehouse that never suspends, a dev cluster that never stops, storage nobody reads. That's usually the cheapest money in the building, and it needs no engineering judgment at all.

Then I'd rank the rest by annual saving per week of work, not by size, because the biggest line is usually not the best first move."

If you want one memorable sentence: "the cheapest optimization is deleting something nobody uses, and every organization I've seen has more of that than it expects."

33.2 Know Your Meter

Three pricing models dominate data infrastructure, and they reward different behaviour.

The most consequential difference is what happens when your query is fast.

On a per-byte meter, a fast query and a slow query that read the same data cost the same. Optimizing CPU buys nothing; optimizing the scan is everything.

On a per-second meter, finishing faster saves money directly — and so does suspending, which is why §33.7's largest single waste is a warehouse that never suspends.

On a per-node-hour meter you pay for the cluster whether or not it is doing anything, which means idle time is indistinguishable from work and the optimization is as much about scheduling as about the query.

⚠️ Failure Mode — optimizing for the wrong meter

A team spends three weeks tuning a query's CPU efficiency, on BigQuery. The query goes from 90 seconds to 22 seconds and the cost does not change by a cent, because it scans the same bytes.

The reverse also happens, and is more expensive: a team on a per-node-hour meter carefully reduces bytes scanned by a factor of ten, and the bill barely moves — because the cluster is provisioned for the peak and runs for the same wall-clock hour regardless.

This is not a hypothetical. §33.3 measures Chapter 1's fix on all three meters and the ratios are 126.5×, 128.6×, and 51.3× for exactly this reason.

The check takes five minutes and almost nobody does it: open the bill, find the line your work affects, and read the unit. Per TiB? Per credit-second? Per instance-hour? You cannot optimize a number you have not read the units of, and the three optimizations are close to disjoint.

33.3 The Same Workload, Three Meters

code/cost_model.py --estimate prices five real Kestrel queries on all three meters:

query                                    GB scan    per-TiB    per-sec per-node-h
─────────────────────────────────────────────────────────────────────────────────
Ch1 sessionization, BROKEN  (full scan)   4,300.0     $26.25     $80.00  $3,840.00
Ch1 sessionization, FIXED   (pruned)         34.0      $0.21      $0.62     $74.88
daily revenue rollup                         82.0      $0.50      $0.42      $7.68
a dashboard tile                              1.4      $0.01      $0.01      $0.05
an analyst's SELECT *                       890.0      $5.43      $2.44     $21.12

And the Chapter 1 fix, priced three ways:

per TiB scanned            $26.25 -> $0.21       126.5x
per warehouse-second       $80.00 -> $0.62       128.6x
per node-hour           $3,840.00 -> $74.88       51.3x

📐 Design Decision — the ratio and the dollars rank differently, and people optimize ratios

Read those three rows carefully, because they disagree in an instructive way.

By ratio, the per-node-hour meter shows the smallest win — 51.3× against 126.5× and 128.6×. By dollars, it shows by far the largest: $3,765.12 saved a night, against $26.04 and $79.38.

Why the ratio is worse on the node-hour meter: the fixed job still provisions 24 nodes and still runs for 1.3 hours. You pay for a machine that exists, not for the work it does, so a perfectly efficient query on an over-provisioned cluster still costs the cluster. The other two meters bill closer to the work.

Which is the more useful number? The dollars, always — and yet the ratio is what gets reported, because "126× faster" is a better sentence than "$26 a night." Kestrel's own postmortem for Chapter 1's job led with 51.3× and buried $1,374,269, and the reason the fix was prioritized at all was that somebody put the annual figure in the subject line.

The general rule: report the dollars per year, and use the ratio as supporting detail. A 200× improvement on a $4/month line item is a fact about a query. $1.37 million a year is a fact about the business, and the two get funded differently.

33.4 Where the Money Actually Is

--bill produces a month of Kestrel's costs, and the category split is the first surprise:

BY CATEGORY
    compute                $18,873.60    61.9%
    warehouse               $8,928.00    29.3%
    egress                  $1,713.60     5.6%
    storage                   $644.76     2.1%
    requests                  $351.40     1.2%
                          ───────────
    TOTAL                  $30,511.36

Storage is 2.1% of the bill. Kestrel holds roughly 27.4 TiB — three years of raw clickstream JSON, the Parquet derived from it, bronze orders, silver, gold, ML snapshots — and the whole of it costs $644.76 a month.

💸 Cost Check — storage is 2.1% of the bill and receives most of the attention

Compute plus warehouse is $27,801.60 — 91.1%. Storage is $644.76 — 2.1%. They are separated by a factor of more than 43, and the effort spent on each is very close to inverted.

Storage optimization is popular for three bad reasons:

• It is easy to reason about. Gigabytes times a price. No query plans, no concurrency.
• It produces satisfying percentages. "We cut storage by 60%!" — of 2.1%, which is 1.3% of the bill.
• It is nobody's fault. Deleting old files upsets no one, whereas telling a team its dashboard costs $2,000 a month is a conversation.

The arithmetic that ends the debate: deleting every byte Kestrel stores saves $644.76 a month. Turning off one warehouse that nobody queries saves $5,040.00 a month — nearly eight times as much — and takes about a minute.

Two honest qualifications, because the rule is not universal:

• Storage drives compute. More bytes stored is more bytes scanned, and the scanning is where the money is. Compaction and partitioning are storage changes with compute payoffs, and they are the exception (§33.9).
• At extreme scale the ratio inverts. A petabyte-scale archive with light query traffic is a storage bill. Check yours before assuming Kestrel's shape.

But the default assumption should be that your bill is a compute bill, and the fastest way to confirm it is the five-line category split above.

33.5 Attribution, and the Share Nobody Claims

The category split says which service. The useful question is which team, and which pipeline.

BY OWNER
    ** UNATTRIBUTED **     $11,102.70    36.4%
    data-platform          $14,050.66    46.0%
    analytics               $4,435.20    14.5%
    data-science              $922.80     3.0%

$11,102.70 of $30,511.36 — 36.4% — has no owner, and that is the finding. Not the total, and not the biggest line.

What is in it:

dev cluster (always on)                $5,184.00
REPORTING_WH (auto-suspend off)        $5,760.00
backups/2023/                            $110.40
_scratch/                                 $48.30

Every one of these is unowned because it is waste. Nobody claims the dev cluster because everybody uses it occasionally; nobody claims REPORTING_WH because it was created for a migration that finished. The correlation is not a coincidence and it is the reason attribution is worth the effort: unattributed spend is enriched in waste, so the attribution exercise is itself a waste-finding exercise.

🔎 Read the Plan — tagging is the whole mechanism, and it fails in one specific way

Attribution is a tag on every resource — a bucket prefix, a cluster, a warehouse, a query — naming a team and a pipeline. The mechanism is trivial and the failure is not.

Tags are applied at creation and never at deletion. So a resource created before the tagging policy, or created by hand during an incident, or created by a tool that does not propagate tags, is untagged forever — and untagged resources are exactly the ones nobody is watching.

Three things that make it work, learned in order at Kestrel:

• Refuse to create untagged resources. A Terraform policy (Chapter 28) that fails the plan. This fixes the future and nothing else.
• Charge the untagged bucket to somebody. Kestrel assigns unattributed spend to the platform team's budget. This is the change that made the number move — a 36.4% unattributed share is an abstraction until it is on someone's budget line, at which point it is investigated within a week.
• Report the unattributed share as a metric, monthly, next to the total. It went from 36.4% to 4.1% in two months, and almost all of the reduction was deletion rather than tagging.

That last detail is the one worth carrying: most unattributed spend, once someone has to claim it, turns out not to be wanted at all.

🧭 Version Note — showback before chargeback

Showback tells each team what it spent. Chargeback moves the money to their budget.

Start with showback and stay there longer than you think. Chargeback creates immediate incentives to game the attribution rather than reduce the cost — teams argue about whether a shared pipeline belongs to them, and the arguing costs more engineering time than the spend under discussion.

Kestrel's sequence, which took nine months: publish a monthly per-team number → charge the unattributed share to the platform team → let teams opt into chargeback once they trust the number. Only the second step was mandatory, and it was the one that did the work.

33.6 The Pre-Flight Estimate

Cost information changes a decision only before the work runs. A dashboard showing last month's spend is a historical record.

Three places to put the estimate, in increasing order of value:

In the engineer's hands, on demand. EXPLAIN with bytes, or a dry run. BigQuery's dry run returns bytes scanned without executing; Snowflake's EXPLAIN gives partition counts; Spark's plan shows the scan node's size. Chapter 1's 🔎 callout is exactly this, and reading a plan remains the highest return-on-time skill in this book.

In code review. A bot that comments the estimated cost of a changed query on the pull request (Chapter 27). This is where it changes the most decisions, because the author is already thinking about the query and the reviewer has a number to react to.

In CI, as a gate. A query whose estimate exceeds a threshold fails the build until someone acknowledges it.

🧪 Try It — estimate before you run, three times

bash cd part-06-advanced-topics/chapter-33-cloud-cost-optimization/code python cost_model.py --estimate

Then, for the next three queries you write against a real system:

1. Write down what you think it will cost, before running it.
2. Run the dry run or EXPLAIN and get the estimate.
3. Run it, and read the actual from the query history.

Most people are wrong by more than 10× on the first one and within 2× by the third. The skill is almost entirely calibration, and it takes about twenty minutes to acquire.

Then do it once for a query you have already shipped. That one is more uncomfortable and more useful.

🏭 From the Pipeline — the estimate that stopped a query rather than optimizing one

An analyst asked for a table joining every clickstream event to every order, to answer "which page did people view before buying?"

The dry run said 4,290 GB — 4.19 TiB — scanned per run, at daily refresh. On the per-TiB meter that is $26.19 per run, $9,558 a year — for a question that turned out to be asked twice a quarter.

The conversation that followed took ten minutes and produced a different artifact: a query the analyst runs on demand, over 30 days instead of three years, at $0.21 a run.

Nothing was optimized. The original query was not made faster, cheaper, or better. It was not built, and the alternative cost 0.02% as much because it answered the question that was actually being asked.

This is the highest-value use of a pre-flight estimate and it is the one nobody counts, because a saving from work not done leaves no trace in any dashboard. Kestrel's team keeps a manual tally of these — nine in a year, an estimated $71,000 avoided — precisely because no automated system will ever attribute it to them.

33.7 Seven Wastes, Sized

--waste sizes seven patterns, each with the arithmetic behind it:

pattern                                $/month        $/year   weeks
─────────────────────────────────────────────────────────────────────
warehouse auto-suspend disabled       $5,040.00    $60,480.00    0.02
dev cluster never stopped             $3,916.80    $47,001.60    0.20
over-frequent dashboard refresh       $2,059.20    $24,710.40    0.10
cross-region egress                   $1,656.00    $19,872.00    1.00
raw JSON retained past need             $216.83     $2,601.99    0.50
small-file GET amplification            $177.60     $2,131.20    2.00
orphaned storage                        $158.70     $1,904.40    1.00
─────────────────────────────────────────────────────────────────────
TOTAL                                $13,225.13   $158,701.56

$13,225.13 a month is 43.3% of the bill.

The arithmetic, one line each:

Auto-suspend disabled. REPORTING_WH is a Medium warehouse, 4 credits/hour at $2.00, running 24 hours a day and queried about 3. 4 × 21 × 30 × $2.00 = $5,040.00 of paying for idle.

Dev cluster never stopped. 3 nodes billed 24 × 30 and used about 8 × 22. 3 × (720 − 176) × $2.400 = $3,916.80.

Over-frequent dashboard refresh. 18 tiles refreshing every 5 minutes — 8,640 refreshes a month — against a dashboard viewed twice a day, needing 60.

Cross-region egress. 18.4 TB/month at $0.09/GB, from a reader in the wrong region.

Raw JSON past need. 75% of the 12.57 TB of raw JSON is older than the 400 days it is needed for.

Small-file GET amplification. 486M GET requests a month against 42M after compaction.

Orphaned storage. _scratch/ and backups/2023/ — 6,900 GB, no owner, no reads in 400 days.

📏 Scale Note — the first three are idle, not inefficiency

$11,016.00 of the $13,225.13 — 83.3% — is paying for resources that are doing nothing.

This is the shape almost everywhere and it is worth internalizing, because it contradicts the instinct that cost optimization is about making things faster:

• Idle spend requires no engineering judgment to remove. Nobody has to understand the query.
• It has no risk profile. Suspending an unused warehouse cannot produce a wrong answer.
• It is invisible in every performance metric. A cluster doing nothing has excellent latency.

And it accumulates by exactly the mechanism Chapter 30 §30.5 described for grants: someone disables auto-suspend during a migration because the cold-start latency is annoying, the migration ends, and re-enabling it has a cost (someone might notice a slow first query) and no visible benefit.

So the first cost review of any platform should not look at queries at all. It should ask, for every compute resource: what fraction of the time it is billed is it doing work? Kestrel's answer for REPORTING_WH was 12.5%, and for the dev cluster 24.4%.

33.8 Rank by Payoff, Not by Size

Everybody ranks the list by size. --rank divides annual saving by weeks of work, and the orderings disagree immediately:

ranked by SIZE                         ranked by PAYOFF PER WEEK
1. auto-suspend      $60,480/yr        1. auto-suspend      $60,480/yr in 0.02w
2. dev cluster       $47,002/yr        2. dashboard refresh $24,710/yr in 0.10w
3. dashboard refresh $24,710/yr        3. dev cluster       $47,002/yr in 0.20w
...                                    ...
7. orphaned storage   $1,904/yr        7. small-file GETs    $2,131/yr in 2.00w

They agree on first place and disagree from second onward, and the last place swaps entirely: small-file compaction is the worst payoff on the list despite not being the smallest saving, because two weeks of work buys $2,131 a year.

📐 Design Decision — the number that should be on the ranking is dollars per week

warehouse auto-suspend disabled is $60,480.00 a year for 48 minutes of work — a rate of $3,024,000 per engineer-week.

small-file GET amplification is $2,131.20 a year for two weeks — $1,065.60 per engineer-week.

The two differ by a factor of 2,838, and a list ranked by size puts them four rows apart.

Three practical consequences:

• Do every sub-day item before any multi-week item, regardless of size. The auto-suspend fix, the dashboard refresh interval, and the dev cluster schedule are $132,192 a year for under two days of work combined.
• Multi-week optimizations need a different justification than cost. Compaction is worth doing at Kestrel — for query latency and for Chapter 31's deletion story — and its $2,131 of GET savings is a rounding error that should not be the argument. Using a weak cost argument for work that has a strong non-cost argument is how good projects get rejected.
• Effort estimates are the weak input, and they are yours to make honestly. A "one week" that is really six weeks inverts the ranking, so estimate the fix you will actually ship rather than the ideal one.

And the sequencing benefit that is easy to miss: doing the cheap items first buys credibility for the expensive ones. Kestrel's compaction project was approved on the back of $132,192 already delivered, which is a better argument than any projection.

33.9 The Optimizations That Actually Matter

Five, in the order they pay off for a data platform.

1. Partition pruning. Chapter 9. The single highest-value optimization in this book — Chapter 1's job is one instance, and the failure mode is always the same: a function applied to the partition column in a WHERE clause. Check the plan, not the query.

2. File size. Chapter 9 §9.6. Too small and you pay per-request and per-task overhead; too large and you lose parallelism. 128 MB–1 GB is the durable range, and the cost of getting it wrong is mostly in compute rather than in requests.

3. Column pruning and projection. SELECT * on a wide columnar table reads every column. On a per-byte meter this is a direct multiplier, and the analyst's SELECT * in §33.3 scans 890 GB where the four columns needed scan 31.

4. Incremental instead of full. Chapter 20. A model rebuilding three years nightly to add one day is paying 1,095× for the same result — and this is the most common expensive mistake in a dbt project, because full refresh is the default and it works.

5. Suspend, stop, and expire. §33.7. Not an optimization at all, which is why it is fifth on a list of optimizations and first on the list of things to do.

💸 Cost Check — compression, and why it is not on the list

Compression is the optimization everyone reaches for and it is usually already done.

Kestrel's clickstream: 4.19 TB/year of JSON → 341 GB/year of Parquet, a 12.3× reduction. That conversion is enormous and it happened in Chapter 8, as a format decision rather than a cost project.

What remains is tuning the codec — Snappy versus Zstd versus gzip — and the numbers are much smaller than people expect:

text Snappy (default) 341 GB/yr $7.84/month storage Zstd level 3 268 GB/yr $6.16/month saving $1.68/month gzip 241 GB/yr $5.54/month saving $2.30/month, slower reads

Moving from Snappy to Zstd saves $20.16 a year in storage and costs some CPU on write. It is not a project.

The exception, and it is a real one: compression affects bytes scanned, so on a per-byte meter a better codec is a compute saving rather than a storage saving — and there the arithmetic can be worth doing. Check which meter you are on before deciding whether this paragraph applies to you, which is §33.2's point arriving in a specific case.

33.10 Reserved Capacity, Spot, and the Commitment Trap

Discounts are real, substantial, and the place where cost optimization most often goes wrong.

Three mechanisms:

Reserved instances / savings plans. Commit to a spend level for one or three years for roughly 30–60% off. The saving is genuine and the commitment is the risk.

Spot / preemptible instances. 60–90% off, and the machine can be taken away with two minutes' notice. Excellent for a re-runnable batch job (Chapter 24's retries) and unusable for anything with a deadline it cannot miss twice.

Committed-use storage and volume tiers. Smaller discounts, almost no downside.

⚠️ Failure Mode — the commitment that outlived the workload

The trap is not that you commit. It is that you commit to the shape of a workload you are about to change.

Kestrel came close: a three-year reservation sized against the current Spark footprint, priced at a 42% saving. In the same quarter, the Chapter 21 migration reduced that footprint by 61%.

Had the reservation been signed, Kestrel would have been paying for capacity it had just eliminated — for three years — and the reservation would have made the migration look like it saved nothing. That second effect is the more insidious one: a commitment converts a future efficiency into a sunk cost, and teams stop pursuing efficiencies they cannot benefit from.

Three rules that survive contact:

• Commit to the floor, not the current level. Reserve the capacity you are confident you will still need in two years — usually 50–70% of current — and pay on demand for the rest. The blended rate is close to optimal and the downside is bounded.
• Never commit in the six months before a planned migration. Obvious, and it happens constantly, because the commitment is negotiated by people who are not in the migration's planning meetings.
• Do the engineering first. A 42% discount on a workload you can reduce by 61% is the wrong order. Optimize, let it settle for a quarter, then commit to what remains.

The finance team will push the other way, and reasonably — the discount is certain and your efficiency projection is not. The honest answer is to commit to the floor, which is a position both sides can defend.

33.11 Unit Economics: Cost Per Thing

A total is not a number you can reason about. "Our data platform costs $30,511 a month" answers no question anyone has. Is that a lot? Compared to what? What happens if we double?

A unit cost answers all three, and for a data platform there are usually two worth publishing: cost per business event, and cost as a share of the thing the business measures.

For Kestrel, at $30,511.36 a month — $366,136.32 a year:

2,400,000 orders/year          ->  $0.1526 per order
5,110,000,000 events/year      ->  $0.0717 per 1,000 events
6,480,000 order lines/year     ->  $0.0565 per order line
$182,000,000 GMV               ->  0.201% of GMV

After the optimizations in this chapter, at $17,286.23 a month:

                                   $0.0864 per order
                                   0.114% of GMV

Three things a unit cost does that a total does not:

It survives a growth conversation. "If we double orders, does the bill double?" is unanswerable from a total and straightforward from a decomposition. The answer at Kestrel is no, and knowing why is the useful part (below).

It makes a comparison possible. 0.201% of GMV is a number another company's data leader can react to; $30,511 is not. It is also the only form in which cost work is legible to a finance team, which matters when you are asking for engineering time.

It reframes an increase. A bill that rises 30% while volume rises 45% is a 10% improvement in unit cost, and reporting only the total makes a success look like a failure. This is the single most common way cost work goes unrewarded.

📏 Scale Note — which lines scale with volume, and which do not

The question that makes a unit cost predictive rather than descriptive, and it takes an afternoon:

Line	Scales with	Doubling orders
S3 storage	cumulative volume	+100%, eventually
Spark sessionization	events scanned	+100%
S3 GET/PUT	files touched	+100%
Egress	rows exported	+100%
dbt transform	rows transformed	+~60% — fixed overhead per run
Kafka brokers	peak throughput	+0% until a threshold, then a step
Airflow	number of tasks	+0%
Dev cluster, idle warehouse	nothing	+0%

Kestrel's decomposition came out at roughly 64% variable and 36% fixed, so doubling orders raises the bill by about 64%, not 100% — and unit cost per order falls by 18%.

Two consequences worth planning around:

• Growth improves your unit economics automatically, which means a rising total bill during growth is expected and is not evidence of a problem. Say so before the growth happens, because afterwards it sounds like an excuse.
• The step functions are the risk. Kafka is flat until it needs a fourth broker, at which point it jumps by $1,728 a month in one day. Know where your steps are, because a step arriving unannounced during a growth quarter is the cost conversation nobody wants to have.

33.12 Telling Another Team What They Cost

A substantial share of the money is spent by people who do not report to you, and the conversation that follows a cost finding determines whether anything changes.

Kestrel's first attempt went badly. An engineer posted in a shared channel: "the marketing dashboard costs $2,160 a month, can we turn down the refresh?" It was true, it was polite, and it produced three weeks of defensiveness — because it arrived in public, as a number about somebody's work, from a team with no stake in what the dashboard was for.

🔎 Read the Plan — four things that made the second attempt work

The same finding, delivered differently, was agreed in one twenty-minute meeting.

Bring the alternative, not the problem. Not "this costs $2,160." Instead: "this refreshes every five minutes and is viewed twice a day; at hourly refresh it costs $180 and nobody would notice. Would that break anything?" The question at the end is not decoration — it is the part that makes it a conversation rather than a verdict.

Ask what it is for, first and genuinely. Kestrel's dashboard turned out to have one tile that did need five-minute refresh — a live campaign spend monitor — and seventeen that did not. The engineer who brought the finding did not know that, and the finding as originally stated was wrong for one-eighteenth of the dashboard.

Give the number in the units the other team thinks in. Marketing does not have an intuition for $2,160 of warehouse credits. $2,160 a month is roughly one junior contractor-week, or 0.5% of the campaign budget it monitors — and the second framing is the one that made the decision obvious to them.

Never in public first. A number about somebody's work, posted where their manager can see it, is received as a criticism regardless of intent. Kestrel's rule now is that a cost finding goes to the owning team privately, and only the aggregate goes in the monthly report.

The outcome: one tile at five minutes, seventeen at hourly, $2,059.20 a month saved, and the marketing team subsequently asked for two more cost reviews of their own accord — which is the real measure of whether the conversation went well.

🔁 Idempotency Check — a retry is a cost, and a retry storm is a bill

Every retry in this book costs money, and the cost is invisible because it is attributed to the successful run.

text mechanism what a retry costs ───────────────────────────────────────────────────────────────────────── a Spark task retry one task's compute, and if the stage fails four times, four attempts at the whole stage a warehouse query retry the full query again, on a per-second meter an API retry with backoff requests against a quota; and a 429 loop is free in dollars and expensive in access a DAG-level retry the WHOLE task, including everything it did before it failed an idempotent write nothing extra -- the second write is a no-op at the target, but the COMPUTE that produced it was paid for again

The last row is the one worth internalising: idempotency makes retries correct, not free. A nightly job that fails at 90% and retries from the beginning pays 190% of its cost, and the bill shows one successful run.

Three things that follow:

Chunk long jobs so a retry is small (§7.7, §13.1). A 90-minute job that retries from scratch costs 90 minutes; the same work in six 15-minute chunks costs 15.

Cap retries, and alert on the count rather than only on the final failure. A task that succeeds on its third attempt every night is a nightly 3× cost and a green dashboard.

And put retry attempts in the run record (Exercise 25.23a). attempt_number is one column and it turns "why did last month cost 20% more" into a query — which is otherwise one of the harder cost questions to answer, because nothing in the bill knows about attempts.

🔐 Privacy & Governance — deleting data for cost reasons is a governance decision

The cheapest way to reduce storage cost is to delete things, and the moment somebody proposes it, a cost conversation has become a retention conversation.

text the proposal the question it actually raises ───────────────────────────────────────────────────────────────────── "lifecycle old bronze to Glacier" does our stated retention permit the data to be hard to reach? (Compliance often requires RETRIEVABILITY, not just retention.) "delete raw clickstream after 90 days" our policy says three years. Which one is true? (Ch 30's finding.) "drop the orphaned scratch schemas" who confirms they are orphaned, and is anything in there subject to a legal hold? "shorten time-travel retention" it shortens our own deletion latency, which is GOOD -- and it removes the month-end investigation window

The fourth row is the one that goes the right way, and it is worth naming because it is the exception: a shorter retention is usually a privacy improvement and a cost improvement at once (Chapter 10's 🔐).

The other three are the same trap in different clothes: a cost optimisation that changes what the organisation can do or must do. And they are proposed by engineers, approved in a cost review, and never seen by anyone who knows the retention policy.

Two lines that prevent it, both cheap:

Any lifecycle or deletion change goes through the same review as a schema change. It is a definitional change to the platform's obligations, and §27.7's shapes apply.

And the cost model records a retention_source per dataset — policy, legal hold, engineering judgment, or "nobody decided." The last value is the most common one, and finding out is the point.

The stronger version of the argument, which is worth making to a finance partner: storage is 2.1% of the bill (§33.4), so there is almost never a cost reason to take a retention risk. The compute lines are 91% and none of them raises a governance question.

🏭 From the Pipeline — the $4 saving that took two engineer-weeks

A team spent two weeks compacting small files across the lake. The work was correct: 340,000 files became 1,712, listing times improved, and a query that had taken 214 seconds took 19.

The saving reported to finance was the storage and request cost: $177.60 a month.

Finance asked how much the two weeks had cost. Roughly $12,000 fully loaded, against $2,131 a year — a payback period of five and a half years, and the finding was written up as a cautionary tale about engineering priorities.

It was the wrong write-up, and the mistake is instructive.

The $177.60 was the cost saving. The 214-to-19-second improvement was the actual return, and nobody had priced it:

text the query ran 22 times a day, across 8 analysts 195 seconds saved x 22 x 250 working days = 1,072,500 seconds = 298 analyst-hours / year at ~$70/h fully loaded = ~$20,860 / year

Payback in about seven months, on a number nobody had computed.

Three lessons, and the third is the one Chapter 33 is actually about.

The cost model measures the bill and the bill is not the whole cost. Analyst time, engineer time waiting on CI (Chapter 27's 📏), and margin to a deadline (Chapter 25 §25.7) are all real and none of them is on an invoice.

Which means "rank by $/year ÷ weeks" (§33.7) is a heuristic with a blind spot, and small-file compaction is precisely the row where it misleads — §33.9 says the cost argument for compaction is weak and the latency argument is strong, and this is what that looks like in a review.

And the write-up mattered more than the work. The team did a good piece of engineering and reported the wrong number about it, which is Chapter 33 §33.3's design-decision callout in reverse: they reported the dollars and should have reported both.

🔎 Read the Plan — the three numbers on a bill that answer everything

A cloud bill is a large document with three useful numbers in it, and finding them takes about ten minutes on any provider.

```text 1. THE SPLIT BY SERVICE. compute / warehouse / storage / network / requests -> which meter are you actually on? (§33.2) -> and is storage 2% or 40%? Kestrel's shape is not universal.

2. THE UNATTRIBUTED SHARE. group by your cost-allocation tag; the untagged remainder is the number. -> 36.4% at Kestrel, and unattributed spend is enriched in waste

3. IDLE AGAINST BUSY, on the largest line. for a warehouse: hours UP against hours QUERIED for a cluster: node-hours BILLED against node-hours DOING WORK -> 83.3% of Kestrel's waste was idle rather than inefficient ```

Number 2 is the one to compute first, because it bounds how much the other two can tell you. A bill that is 40% unattributed cannot be analysed by team, by job, or by pipeline — and the fix is a tag, applied at resource creation, which is a Chapter 28 concern.

Number 3 is the one that produces the largest single finding, and it is not on any invoice. It requires joining the bill to the query history or the job history, which is why Exercise 25.22's query tag matters and why the run-record table (Exercise 25.23a) keeps paying for itself.

And the discipline that makes any of it repeatable: write the three numbers down monthly, in the same place. A trend in the unattributed share is a governance signal; a trend in the idle share is an operational one; a trend in the split tells you when your platform's shape has changed enough that the earlier analysis no longer applies.

33.13 Making It a Habit

Everything above is a project. The durable version is a habit, and it is a small one.

Four practices, in the order they stick:

Put the cost in the pull request. Chapter 27. The single highest-leverage change, because it puts a number where a decision is being made.

Publish per-team spend monthly, with the unattributed share. §33.5. The unattributed share is the metric that moves things, and it went 36.4% → 4.1% at Kestrel in two months.

Add cost to the incident review. Chapter 26's postmortems already ask what broke. Add: what did it cost? — a runaway query, a retry loop, a backfill re-run. It builds the intuition faster than any dashboard.

Estimate before shipping. §33.6. Three queries is enough to calibrate.

🧱 Kestrel Platform — what cost work is on the platform now

text platform/cost/ estimate.py # dry run -> dollars, on this rate card pr_cost_bot.py # comments the delta on every changed query (Ch 27) attribute.py # bill -> team + pipeline, from tags waste.py # the seven patterns, run weekly rates.yml # the rate card, versioned, ONE place

rates.yml is the detail worth copying. Before it, five scripts each had prices hard-coded, three of them stale, and two different answers to "what does a node-hour cost" were in circulation — which made every cost estimate arguable and therefore ignorable.

One versioned file, one source of truth, and it is Chapter 30 §30.8's certified-definition problem in a new place. A cost estimate nobody trusts changes no decisions, and trust here came from a boring file rather than from better arithmetic.

Results after two quarters:

	Before	After
Monthly bill	$30,511.36	$17,286.23
Unattributed share	36.4%	4.1%
Idle spend	$11,016.00/mo	$412.80/mo
Cost estimate in PR	none	every changed query
Queries not built after an estimate	untracked	9 in a year, ~$71,000 avoided

The bill fell 43.3%, and the great majority of it came from turning things off.

33.14 Summary

Cloud cost is an engineering property, decided in code, in seconds, usually without noticing. Chapter 1's CAST cost $3,840.00 a night instead of $74.88 — $1,374,269 a year from four characters — and the job succeeded every night for months.

⚠️ Know your meter before optimizing. Per byte scanned, per warehouse-second, per node-hour reward close to disjoint work. Three weeks of CPU tuning on a per-byte meter changes nothing.

📐 Ratios and dollars rank differently, and people report ratios. Chapter 1's fix is 126.5× / 128.6× / 51.3× on the three meters and saves the most dollars on the meter with the worst ratio. Report the annual dollars; use the ratio as detail.

💸 Storage is 2.1% of Kestrel's bill; compute and warehouse are 91.1%. Deleting every byte saves $644.76/month; turning off one unused warehouse saves $5,040.00 and takes a minute. Storage optimization is popular because it is easy, satisfying, and upsets nobody. Check your own split before assuming — but expect a compute bill.

🔎 36.4% of the bill had no owner, and unattributed spend is enriched in waste — because a resource is unowned for the same reason it is wasted. Charging the unattributed bucket to somebody is the change that moves the number: 36.4% → 4.1% in two months, almost all of it by deletion rather than tagging.

Showback before chargeback. Chargeback creates incentives to argue about attribution rather than reduce cost.

📏 83.3% of Kestrel's waste was idle, not inefficient. A warehouse doing work 12.5% of the time it is billed; a dev cluster 24.4%. Idle spend needs no engineering judgment, carries no correctness risk, and is invisible in every performance metric. The first cost review should ask, of every compute resource: what fraction of its billed time is it working?

📐 Rank by annual saving per week of work, not by size. Auto-suspend is $3,024,000 per engineer-week; small-file compaction is $1,065.60 — a factor of 2,838, four rows apart on a size-ranked list. Do every sub-day item before any multi-week item: $132,192 a year for under two days of work.

And give multi-week work its real justification. Compaction is worth doing for latency and for Chapter 31's deletion story — using a weak cost argument for work with a strong non-cost argument is how good projects get rejected.

🏭 The best pre-flight estimate stops a query rather than optimizing one. A 4.19 TiB daily join, $9,558 a year, replaced by an on-demand query at $0.21 a run — for a question asked twice a quarter. Nine of these in a year, ~$71,000 avoided, and no dashboard will ever attribute it to you.

⚠️ Do the engineering before you commit. A 42% reservation against a footprint a migration was about to cut by 61% — and worse, a commitment converts a future efficiency into a sunk cost, so teams stop pursuing savings they cannot benefit from. Commit to the floor, not the current level.

🧱 One versioned rate card. Kestrel had five scripts with hard-coded prices and two circulating answers to "what does a node-hour cost," which made every estimate arguable and therefore ignorable. A cost estimate nobody trusts changes no decisions.

📏 Publish a unit cost, not a total. $0.1526 per order, 0.201% of GMV — and after this chapter's work, $0.0864 and 0.114%. A total answers no question anyone has. And decompose it: Kestrel is ~64% variable, ~36% fixed, so doubling orders raises the bill by about 64% and lowers unit cost by 18%. Say that before the growth happens, because afterwards it sounds like an excuse.

🔎 A cost finding about somebody else's work is received as a criticism unless you bring the alternative, ask what it is for, use their units, and go to them privately first. Kestrel's first attempt — true, polite, and posted in a shared channel — produced three weeks of defensiveness. The second took twenty minutes, and one tile of eighteen genuinely did need the five-minute refresh, which the finding as first stated had got wrong.

The habit, in four parts: cost in the pull request · monthly per-team spend with the unattributed share · cost in the incident review · estimate before shipping.

Part VI ends here

Six chapters that share one property: none of them is about moving data, and all of them determine whether a platform that moves data correctly is one anybody can use.

Chapter 29 asked whether you need streaming and mostly answered no. Chapter 30 made the platform findable. Chapter 31 made it lawful. Chapter 32 made it serviceable to models. Chapter 33 made it affordable — and the recurring finding across all five, in five different departments, is the one this book keeps arriving at: a control that exists and does not operate is not a control, and the only way to tell the difference is to measure it.

Part VII is architecture patterns — medallion, mesh, event-driven, and the migration off a legacy stack. It is the part where the book is most willing to say that a popular idea is usually implemented badly, and Chapter 35 is direct about which one.

Key terms: unit economics · cost attribution · showback · chargeback · tagging · on-demand pricing · reserved instances · spot instances · auto-suspend · partition pruning · file compaction · egress · lifecycle policy · credit · cost per query