Key Takeaways: Cloud Cost Optimization

The one thing

Cloud cost is an engineering property, decided in code, in seconds, usually without noticing. Not a procurement outcome, not a vendor negotiation, and not something a finance team can fix. The people who determine the bill are the people writing queries, and most of them have never seen it.

Why it stays broken

Chapter 1's CAST: $3,840.00 a night instead of $74.88 — $1,374,269 a year from four characters, and the job succeeded every night for months.

Three properties of cost defects:

• They are silent. No error, no alert, no failed test.
• They are cheap to create and expensive to find. Four characters, four months, $458,000.
• They are invisible where they are created. The engineer sees no number; the bill arrives thirty days later, aggregated by service, to somebody else.

Know your meter

⚠️ Three weeks of CPU tuning on a per-byte meter changes nothing. And reducing bytes tenfold on a node-hour meter can barely move the bill.

The five-minute check nobody does: open the bill, find the line your work affects, and read the unit.

📐 Ratios and dollars rank differently, and people report ratios. Chapter 1's fix is 126.5× / 128.6× / 51.3× on the three meters and saves the most dollars on the meter with the worst ratio — because a node-hour meter bills a floor the others do not. Report annual dollars; use the ratio as detail.

Where the money is

💸 Storage is 2.1% of Kestrel's bill; compute plus warehouse is 91.1% — a factor of more than 43, with effort spent very nearly inverted.

Deleting every byte saves $644.76/month. Suspending one unused warehouse saves $5,040.00 and takes a minute.

Storage optimization is popular because it is easy to reason about, produces satisfying percentages, and upsets nobody.

Two honest exceptions: storage drives compute, so compaction and partitioning are storage changes with compute payoffs; and at petabyte scale with light query traffic the ratio inverts. Check yours — but expect a compute bill.

Attribution

🔎 36.4% of Kestrel's bill had no owner, and every dollar of it was waste — because a resource is unowned for the same reason it is wasted: nobody is watching it.

A tagging policy fixes the future and nothing else. Kestrel's explained 60%, and the missing 40% was systematically pre-policy resources, incident-created resources, and tool-created resources — the population where waste accumulates. Tag propagation in orchestrators is the largest structural gap.

🧱 Charging the unattributed bucket to a team is what makes the number fall. 36.4% → 4.1% in two months. Not the script, which had been writable for nineteen months and which nobody would have acted on.

87% of it resolved by deletion, 13% by tagging. When somebody finally has to claim a resource, most of the time nobody wants it.

Showback before chargeback. Chargeback creates incentives to argue about attribution rather than reduce cost.

Waste

📏 83.3% of Kestrel's waste was idle, not inefficient. A warehouse working 12.5% of its billed time; a dev cluster 24.4%.

Idle spend needs no engineering judgment, carries no correctness risk, and is invisible in every performance metric — a cluster doing nothing has excellent latency.

It accumulates by the same asymmetry as Chapter 30's grants and Chapter 25's alerts:

creating it:  a clear benefit, a person who wants it, a deadline
removing it:  a diffuse benefit, no requester, and a small risk of blame

Removal has negative expected value to the individual doing it. That is an incentive problem, not a discipline problem, which is why the fix is organizational.

The first cost review should not look at queries. It should ask, of every compute resource: what fraction of its billed time is it working?

Ranking

📐 Rank by annual saving per week of work, not by size.

auto-suspend        $60,480/yr   0.02 weeks   $3,024,000 per engineer-week
dashboard refresh   $24,710/yr   0.10 weeks     $247,104
dev cluster         $47,002/yr   0.20 weeks     $235,008
─────────────────────────────────────────────────────────
small-file GETs      $2,131/yr   2.00 weeks       $1,065.60

A factor of 2,838 between top and bottom, four rows apart on a size-ranked list.

Do every sub-day item before any multi-week item — Kestrel's top three were $132,192 a year for under two days of work.

Give multi-week work its real justification. Compaction is worth doing for latency and for Chapter 31's deletion story. A weak cost argument attached to work with a strong non-cost argument invites the rejection of both.

Cheap wins buy credibility for expensive ones. A twice-rejected compaction project was approved after $132,192 had been delivered.

Estimating

Cost information changes a decision only before the work runs. Three places, in increasing value: in the engineer's hands (dry run, EXPLAIN) · in code review · in CI as a gate.

🏭 The best estimate stops a query rather than optimizing one. A 4.19 TiB daily join, $9,558 a year, replaced by an on-demand query at $0.21 — for a question asked twice a quarter. Nine of these in a year, ~$71,000 avoided, and no dashboard will ever attribute it to you.

📐 A soft gate beats a hard one. Some changes should cost ten times more; a gate that blocks legitimate work is removed within a quarter. The label forces exactly one thing: a human has seen the number.

📐 An unexpected cost number is usually a signal that somebody built the wrong thing. The analyst wanted a ten-day rolling average of a daily metric — ten days of output, not ten days of input — and the correct implementation cost nothing extra.

What monitoring cannot see

⚠️ Every monitor in this book measures time, correctness, or volume. None measures money. Seven monitoring systems green for thirty consecutive nights while a job cost 10× too much.

The output was byte-for-byte identical, so every data-shaped assertion was blind by construction.

🔎 Autoscaling decoupled cost from latency, and latency was the only accidental cost signal. Without it the job would have run 13 hours, missed the 6am SLA by eight hours, and been caught on night one. A reliability feature, working as designed, removed the detection mechanism.

Every autoscaled workload needs a work-volume or cost signal, because it has traded its latency signal away.

Node-hours track work, not machines. 10× the input is 10× the cost whether 24 machines do it slowly or 64 do it quickly.

An SLA met by 8 minutes and one met by 3 hours 42 minutes report the same green. Alert on remaining slack, not on breach.

Set duration thresholds relative to the job's own history. An 8-hour alarm cannot see a 3.75× regression on a 1.3-hour job.

Assert on input volume, not only output volume.

Unit economics

📏 Publish a unit cost, not a total. $0.1526 per order and 0.201% of GMV — after this chapter's work, $0.0864 and 0.114%.

A total answers no question anyone has. A unit cost survives a growth conversation, makes comparison possible, and reframes an increase: a bill rising 30% while volume rises 45% is a 10% improvement, and reporting only the total makes a success look like a failure.

Decompose into fixed and variable. Kestrel is ~64% variable, so doubling orders raises the bill by about 64% and lowers unit cost by 18%. Say that before the growth happens, because afterwards it sounds like an excuse.

Know your step functions. Kafka is flat until it needs a fourth broker, then jumps $1,728 a month in one day.

Commitments

⚠️ Do the engineering before you commit. A 42% discount against a footprint a migration was about to cut by 61%.

And the insidious second effect: a commitment converts a future efficiency into a sunk cost, so teams stop pursuing savings they cannot benefit from.

Three rules: commit to the floor (50–70% of current), not the current level · never commit in the six months before a planned migration · optimize, let it settle a quarter, then commit to what remains.

Telling other people

🔎 A cost finding about somebody else's work is received as a criticism unless you:

• Bring the alternative, not the problem — and end with a question.
• Ask what it is for, genuinely. Kestrel's finding was wrong for one tile of eighteen.
• Use their units. $2,160/month is 0.5% of the campaign budget it monitors.
• Never in public first.

Kestrel's first attempt — true, polite, posted in a shared channel — produced three weeks of defensiveness. The second took twenty minutes, and the marketing team then asked for two more reviews of their own accord.

The habit

Cost in the pull request · monthly per-team spend with the unattributed share · cost in the incident review · estimate before shipping.

🧱 One versioned rate card. Kestrel had five scripts with hard-coded prices and two circulating answers to "what does a node-hour cost." A cost estimate nobody trusts changes no decisions, and the trust came from a boring file rather than better arithmetic.

Result: $30,511.36 → $17,286.23 a month, 36.4% → 4.1% unattributed, with no query rewritten and no architecture changed in the first quarter.

The code

code/cost_model.py — a pre-flight estimator on three meters, an attributed bill with the unattributed share, seven sized wastes, and a payoff-per-week ranking. Forty-eight self-checks, all deriving from one frozen rate card. --demo runs all four.