Case Study 2: Eight Minutes Inside the SLA, $20,217 Over Budget

"Every monitor was green for a month. The invoice was the alert."

Executive Summary

A one-line change to Kestrel's nightly sessionization widened a window from one day to ten, and the job's cost went from $74.88 to $748.80 a night — exactly 10×, because node-hours track work.

Nothing detected it for a full billing month.

The job did not fail. It did not breach the 6am SLA. It did not trip a duration alarm, a freshness check, or any of Chapter 23's twenty-two assertions. The autoscaler absorbed the extra work by adding machines, so the finish time moved from 02:18 to 05:52 — eight minutes inside the deadline.

                          nodes    hours    finishes    cost/night
before                       24     1.30       02:18        $74.88
after, autoscaled            64     4.88       05:52       $748.80
after, WITHOUT autoscaling   24    13.00       14:00       $748.80

Read the third row. Without autoscaling, the job would have missed the 6am SLA by eight hours and been caught on the first night. Autoscaling — a reliability feature, working exactly as designed — converted a one-night detection into a month-long one.

Total excess over the month: $20,217.60, or 66.3% of a normal monthly bill. Annualized, had it not been found: $245,980.80.

Skills applied: the invisibility of cost regressions (§33.1); what latency monitoring cannot see; cost estimates in code review (§33.6); and building a cost signal that is not the invoice.

Background

The change was small, reviewed, and correct in intent.

An analyst needed a rolling ten-day session metric. The engineer widened the window in the sessionization job:

-  .filter(F.col("event_date") == run_date)
+  .filter(F.col("event_date").between(run_date - timedelta(days=9), run_date))

Two lines, one pull request, one approving review. The reviewer checked that the window arithmetic was right — it was — and that the downstream aggregation still grouped correctly. It did.

Nobody computed what it would cost, because there was nowhere in the process that asked.

What the platform monitored at the time, and this is a well-monitored platform by the standards of the preceding chapters:

Chapter 23   22 data quality assertions        all passed
Chapter 25   freshness: gold by 06:00          met, every night
Chapter 25   volume: row counts within band    passed (10x events, 1x sessions out)
Chapter 25   DAG duration                      alarm at 8h; peak was 4.88h
Chapter 26   6am SLA                           met, 30/30 nights
Chapter 27   CI: unit tests, dbt tests         passed
Chapter 28   Terraform drift                   none

Seven monitoring systems, all green, for thirty consecutive nights.

⚠️ Failure Mode — every monitor in this book measures time, correctness, or volume. None measures money.

Go down the list above and notice what each one is actually watching:

• Freshness and the SLA watch when data arrives.
• Duration watches how long a job takes.
• Volume and quality assertions watch what the output contains.
• CI and drift watch what changed in the code and the infrastructure.

The output of this job was byte-for-byte identical. The same sessions, the same row counts, the same distributions — because the extra nine days of input were re-processed and then discarded by the same daily aggregation. There was nothing wrong with the data, and every data-shaped monitor is therefore blind by construction.

The duration alarm was the closest call and it was set on the wrong scale. It was configured at 8 hours — chosen to catch a hung job — and the run peaked at 4.88 hours against a 1.30-hour baseline. A 3.75× regression in duration passed a monitor that exists to detect duration problems, because the threshold was absolute rather than relative to the job's own history.

The general finding: a platform can be comprehensively monitored and have no cost signal at all, and this is the normal state. Cost is the one platform property whose feedback loop is an invoice.

The Problem

The invoice arrived and compute was up $20,217.60.

Finding the cause took two hours, which is worth stating because it is the one part of this case study that went well: the attribution work from Case Study 1 meant the increase was immediately localized to one DAG.

compute, by pipeline          previous month      this month     delta
sessionization                    $2,246.40      $22,464.00   +$20,217.60
hourly clickstream ingest         $2,419.20       $2,419.20        $0.00
kafka brokers                     $5,184.00       $5,184.00        $0.00
airflow                           $3,456.00       $3,456.00        $0.00
ml training                         $384.00         $384.00        $0.00
dev cluster                       $5,184.00       $5,184.00        $0.00

Without per-pipeline attribution this would have been "compute is up 69%, investigate" — and the investigation would have started with the biggest lines rather than the changed one.

The Analysis

The mechanism is arithmetic and it is worth doing slowly, because the conclusion is not obvious.

Node-hours track work, not machines.

before:   24 nodes x 1.30 h  =   31.2 node-hours  x $2.400 =   $74.88
after:    64 nodes x 4.88 h  =  312.0 node-hours  x $2.400 =  $748.80
                                 ──────
                                 exactly 10x, the window multiplier

Autoscaling did not make the job more expensive. It made it finish sooner, at the same cost. The ten-fold increase in work costs ten-fold whether it is done by 24 machines slowly or 64 machines quickly.

🔎 Read the Plan — the reliability feature that removed the only cost signal

This is the finding, and it generalizes well beyond this incident.

Before autoscaling, a cost regression and a latency regression were the same event. Ten times the work on a fixed cluster is ten times the wall clock, which breaches an SLA, which pages somebody, on the first night. The 6am deadline was, accidentally, a cost control.

Autoscaling decouples them. It exists precisely to keep wall-clock stable as work varies — which is a good property, and which means wall clock is no longer a proxy for work, and therefore no longer a proxy for cost.

text without autoscaling: work ↑ 10x -> duration ↑ 10x -> SLA breach -> page with autoscaling: work ↑ 10x -> duration ↑ 3.75x -> no breach -> silence machines ↑ 2.7x -> cost ↑ 10x

The eight minutes of remaining slack are the detail worth sitting with. The job finished at 05:52 against a 06:00 deadline for thirty consecutive nights. It was not comfortably fine; it was one bad night from a major incident, and nobody knew, because a met SLA reports the same green as a comfortably-met one.

Two consequences:

• Every autoscaled workload needs a cost or work-volume signal, because it has traded its latency signal away. Bytes scanned, node-hours, or dollars — any of the three works, and none of them is duration.
• An SLA met by 8 minutes and an SLA met by 3 hours 42 minutes should not look the same. Kestrel now alerts on remaining slack falling below 45 minutes, which would have caught this on night one as a reliability finding even without any cost monitoring at all.

Then the second question: why did the volume assertion pass?

Because it was on the output. Sessions produced per night were unchanged — the job read ten days and emitted one day's sessions. The input volume changed by 10× and nothing measured input volume, which is a gap Chapter 23's register had never considered because every assertion there is about correctness.

The Decision

Four changes, in the order they were shipped.

One: cost estimates in the pull request. pr_cost_bot.py (Chapter 27) computes the estimated node-hours or bytes scanned for any changed query and comments the delta.

🤖 cost estimate for this change

   sessionization.py
     bytes scanned    34 GB  ->  340 GB      (+900%)
     node-hours        31.2  ->  312.0       (+900%)
     cost/night      $74.88  ->  $748.80     (+$673.92)
     annualized     $27,331  ->  $273,312    (+$245,981)

   ⚠️  This change increases estimated cost by more than 50%.
       Add the `cost-reviewed` label to proceed.

📐 Design Decision — a soft gate, and why it is not a hard one

The bot blocks the merge until someone adds a label. It does not block the change.

The reasoning, and the team argued about it for a week:

A hard threshold would be wrong. Some changes genuinely should cost ten times more — a new pipeline, a widened backfill, a model that needs more history. A gate that says "no" to legitimate work gets removed within a quarter, and then you have nothing.

The label is the whole mechanism. It forces exactly one thing: a human has seen the number. The engineer who widened the window would have seen +$245,981 annualized and, on the evidence of what happened next, would have asked the analyst whether ten days was really needed.

It was not. The analyst wanted a ten-day rolling average of a daily metric, which is computed from ten days of output, not ten days of input — a WINDOW over the existing daily sessions table. The correct implementation costs nothing extra.

So the incident was not a cost problem at all. It was a requirement misunderstood by one step, and the cost estimate is valuable here as a prompt to re-read the requirement rather than as a budget control. That is the general case: an unexpected cost number is usually a signal that somebody has built the wrong thing, and treating it purely as a spending question misses the cheaper fix.

Two: alert on remaining SLA slack, not on breach. Below 45 minutes of slack pages. This is a reliability improvement that happens to be a cost signal.

Three: a daily per-pipeline cost anomaly check. Day-over-day cost per DAG; alert above 40%. This is the direct fix and it is deliberately third, because the two above catch the problem earlier and for better reasons.

Four: a volume assertion on inputs, not only outputs. Chapter 23's register gained one row: rows read by each job, within 3× of its trailing median.

What Happened

The daily anomaly check has fired four times in the following year:

• Twice for legitimate changes — a new pipeline and a widened backfill. Both were acknowledged in under an hour, which is the correct outcome and is not a false positive.
• Once for a Snowflake warehouse resized from Medium to Large by someone debugging a slow query and not resized back — 4 extra credits an hour, around the clock. Caught the next day; left alone it would have been $1,920 over ten days.
• Once for a genuine retry storm — a task failing and relaunching two hours into a 16-node run, 40 times overnight. $3,072.00 in one night (40 × 16 × 2.0 h × $2.400), and it also broke nothing that any other monitor could see, for exactly the reasons in the ⚠️ callout.

The pull-request bot's effect is harder to measure and the team believes it is larger. In its first year it commented on 63 changes, of which 11 exceeded the 50% threshold. Of those eleven:

proceeded as written, label added        6
changed after seeing the number          4
abandoned                                1

Five of eleven changed because someone saw a number at the moment they could still act on it — and one of the four changed ones was the ten-day window, re-implemented correctly as a window function over the daily table.

Lessons

1. Every monitor in this book measures time, correctness, or volume. None measures money. Seven monitoring systems were green for thirty consecutive nights.

2. The output was byte-for-byte identical, so every data-shaped assertion was blind by construction.

3. 🔎 Autoscaling decoupled cost from latency, and latency was the only accidental cost signal. Without it the job would have run 13 hours, missed the SLA by eight hours, and been caught on night one. A reliability feature removed the detection mechanism.

4. Every autoscaled workload needs a work-volume or cost signal, because it has traded its latency signal away.

5. Node-hours track work, not machines. 10× the input is 10× the cost whether 24 machines do it slowly or 64 do it quickly. Autoscaling changed the wall clock and not the bill.

6. An SLA met by 8 minutes and one met by 3 hours 42 minutes report the same green. Alert on remaining slack, not on breach — it would have caught this on night one as a reliability finding.

7. The duration alarm was absolute (8 hours) where the regression was relative (3.75×). A threshold set to catch a hung job cannot detect a job doing ten times the work.

8. Volume assertions were on outputs only. Input volume changed 10× and nothing watched it.

9. 📐 A soft gate beats a hard one. Some changes should cost ten times more; a gate that says no to legitimate work is removed within a quarter. The label forces exactly one thing: a human has seen the number.

10. 📐 An unexpected cost number is usually a signal that somebody built the wrong thing. The analyst wanted a ten-day rolling average of a daily metric — ten days of output, not ten days of input — and the correct implementation costs nothing extra.

11. Per-pipeline attribution turned "compute is up 69%, investigate" into two hours. Case Study 1's work paid for itself here.

12. The anomaly check's two "false" alarms were not false. A legitimate change acknowledged in an hour is the system working.

Questions for Discussion

1. Seven monitoring systems were green. Design the eighth. What does it measure, and what is its false positive rate?

2. The duration alarm was at 8 hours against a 1.3-hour baseline. What is the right way to set that threshold, and what does your answer cost in alert volume?

3. Alerting on 45 minutes of remaining slack would have caught this on night one — as a reliability finding. Is it better to catch cost problems through reliability signals, or is that a coincidence you should not rely on?

4. The soft gate lets anyone proceed by adding a label. Construct the case where that fails, and decide whether you would still choose it.

5. Five of eleven flagged changes were modified or abandoned after seeing a number. Is that a good conversion rate? What would a bad one look like?

6. The real fix was a window function over daily output. How would a reviewer have spotted the wrong implementation without the cost estimate?

7. This case study argues autoscaling removed a signal. Does the same argument apply to other reliability features — retries, caching, replication? Find one and trace what it hides.