Case Study 1: Eleven Tools, Four Engineers

"We didn't choose eleven tools. We chose one tool eleven times, each time for a good reason, and nobody ever counted."

Executive Summary

By mid-2025 Kestrel's data platform depended on eleven distinct systems, operated by four engineers. Each had been adopted for a defensible reason. None had been adopted after asking what the total operational surface would be.

The symptom was not any single failure. It was that the team's on-call engineer could competently debug about half the stack, that two systems had exactly one person who understood them, and that routine work — a schema change, a new source, a version upgrade — had become slow in a way nobody could point at.

This case study is the audit: how the team counted what it had, how it measured the operational burden, what it removed, and what it kept despite the burden. The removals saved less than expected; the consolidations saved a great deal, and the difference between the two is the transferable finding.

Skills applied: the systems-per-engineer heuristic (§5.1); buy / build / do without (§5.3); what to leave out (§5.5); tool evaluation (§5.8).

Background

The eleven systems, with the reason each was adopted and when:

Read the dates. Seven of the eleven arrived in a sixteen-month window, each in response to a genuine problem, each adopted by an engineer who was solving the problem in front of them.

Nobody was wrong. The count was never the input to any decision.

The Problem

The audit was triggered by an ordinary event: a Kafka version upgrade took eleven days, five of them spent working out how the existing configuration had been arrived at, because the engineer who set it up had left and the configuration was not in version control.

The engineering manager asked a question the team had not asked before: for each system, how many people can operate it?

Two systems with a bus factor of one, and they were the same person. That person was also the one who had been on the Kafka upgrade, and they had a holiday booked.

The systems-per-engineer ratio was 11 ÷ 4 = 2.75, against the ceiling of roughly 2 that §5.1 proposes. The heuristic was not something the team had heard of; they derived the same conclusion from the bus-factor table, which is a more concrete route to it.

⚠️ Failure Mode — The tool that only one person can operate

A bus factor of one is usually discussed as a risk if the person leaves. That framing understates it in two ways.

It is a problem while they are still there. Every question about that system routes to one person, which makes them a bottleneck and makes the work uninterruptible. They cannot take a holiday without the system being unsupported, so they do not fully take one.

It compounds. The person who understands a system is the one who works on it, which is how they came to understand it, which is why they keep being assigned to it. Knowledge concentration is self-reinforcing unless something deliberately breaks the cycle.

Two mechanisms that work, and both cost something real:

• Rotate the on-call and let the second person struggle. Slower incident resolution in the short term; this is the cost, and it is the point. A system nobody has debugged under pressure is a system nobody can debug under pressure.
• Require a runbook before a system goes to production, written by someone who is not the person who set it up. If they cannot write it, the system is not understood well enough to depend on.

Kestrel did both, and Chapter 26 §26.4 covers the runbook standard that came out of it.

The Analysis

The team measured operational burden over the preceding six months, from ticket and incident logs.

372 hours over six months — roughly 9.5 working weeks, or about 12% of the team's total capacity, spent operating the platform rather than building on it.

Two observations from that table shaped everything that followed.

The distribution is extremely uneven. The top four systems account for 260 of 372 hours — 70% — and the bottom four account for 36. Any reduction effort aimed at the bottom of the table is wasted.

Upgrade and maintenance hours nearly equal incident hours. 163 against 209. Most operational-cost reasoning focuses on incidents, because incidents are memorable. Routine maintenance is invisible, continuous, and almost as large.

💸 Cost Check — The burden nobody invoices for

372 hours over six months is 744 hours a year, or 0.36 of a full-time engineer — more than a third of a person, permanently, spent keeping the platform running.

Compare that to the infrastructure bill for the same period, which the team could state precisely because it arrives monthly. The infrastructure cost was a real number and it was substantially smaller than the labor, on any reasonable loaded-cost assumption for an engineer.

This is the same shape as Chapter 2's Case Study 2 and Chapter 3's Case Study 1, and by now the pattern should be familiar: the metered cost is rarely the expensive one. It shows up so consistently because metered costs are the ones a system generates a document for, and labor is the one you have to go and measure.

The practical consequence for how you argue: when proposing to remove a tool, lead with hours, not with the invoice. The invoice will make your proposal look trivial.

What they considered removing

Great Expectations (33 hours). The dbt tests covered most of what it was catching. GX added distribution checks and a richer expectation vocabulary that were used in exactly two places.

Decision: removed. The two distribution checks were rewritten as dbt tests with a custom generic test. Saved roughly 30 hours a year and one system.

Spark (52 hours). Adopted because sessionization was too slow single-node — a real reason at the time. But the sessionization job had since been rewritten, and the volume it handled had not grown.

They measured before deciding, which is the important part. The current sessionization workload on DuckDB took 14 minutes against Spark's 9. Slower, and comfortably inside the window.

Decision: kept, but narrowed. Spark was retained for the one job that genuinely needs it — a full-history reprocessing that runs quarterly over two years of clickstream — and removed from the nightly path. Ongoing burden dropped because a quarterly job breaks less than a nightly one.

This is the interesting outcome of the audit. "Remove the tool" and "keep the tool" were not the only options; "use it for less" was, and it was the right answer more often than either.

The managed ingestion tool (11 hours). Low burden, and it was doing genuine work on three SaaS sources.

Decision: kept. The lowest-burden system on the list that is not a commodity. This is what buying well looks like.

Kafka and Debezium (146 hours combined). The two most expensive systems and the two with a bus factor of one.

Decision: kept, and invested in. Kafka carries the clickstream, which cannot be handled another way at 14M events a day, and Debezium solves the hard-deletes problem that has no batch equivalent (Chapter 14 §14.2). Removing either would mean losing a capability rather than simplifying.

What changed instead was the bus factor. Configuration moved into Terraform (Chapter 28), a runbook was written by a second engineer, and on-call rotation stopped routing Kafka incidents to one person.

📐 Design Decision — Remove, narrow, or invest

The audit's most useful output was a three-way framing rather than a two-way one.

Remove when another system already covers the capability. Great Expectations, where dbt tests covered the same ground.

Narrow when the capability is genuinely needed but far less often than the tool is currently used. Spark, retained for a quarterly job and removed from the nightly path. This option is systematically overlooked because "should we use X" is naturally heard as a yes-or-no question.

Invest when the capability is essential and the cost is knowledge concentration rather than the tool itself. Kafka and Debezium, where the fix was runbooks, Terraform, and rotation — not removal.

What the three-way framing costs: it is slower. A yes-or-no decision takes an hour; deciding whether to narrow requires measuring the actual usage, which took the team about a day per system. That day is what turned "remove Spark" — which would have been wrong — into "narrow Spark," which was right.

The Decision

The team removed one system, narrowed one, invested in two, and — the change that mattered most — adopted a rule for adding the twelfth.

The rule, written into the ADR process (Chapter 3 §3.7):

Adding a system to the platform requires an ADR that states: (a) which existing system was evaluated for this capability and why it was insufficient, (b) the estimated annual operational hours, (c) who the second operator will be, and (d) the condition under which we would remove it.

Point (c) is the one that changed behavior. Naming a second operator in advance forces the knowledge-sharing question to the front, where it costs an afternoon, instead of to the moment someone leaves, where it costs eleven days.

What Happened

Twelve months later:

The removal of Great Expectations accounted for about 30 of the 128 hours saved. The rest came from narrowing Spark (about 34) and — the largest single contribution — from the bus-factor work on Kafka and Debezium (about 60), which nobody had predicted would reduce hours at all.

That is the finding worth carrying. The team expected savings from removing tools and got most of their savings from making the tools they kept operable by more people. An incident handled by whoever is on call takes less total time than an incident that has to wait for the one person who understands the system, and the difference across a year is large.

Two rules were added later, after the fact:

A system with no runbook does not get a production dependency. Enforced at review.

Quarterly, someone who does not normally operate a system runs its routine maintenance. The first time this was tried, the Airflow upgrade took a day longer than usual and produced a substantially better runbook.

Lessons

1. Count the systems. Nobody chose eleven tools; each was chosen once, well, and the total was never an input.

2. Bus factor is a problem while the person is still there, not just if they leave. It makes them a bottleneck and it is self-reinforcing.

3. Measure operational hours, not just incidents. Maintenance and upgrades were 163 hours against 209 for incidents — nearly equal, and invisible because nothing memorable happens.

4. The distribution is uneven; work on the top of it. Four systems accounted for 70% of the burden.

5. "Narrow" is the missing third option. Spark was neither kept as-is nor removed; it was retained for the one job that needs it. This option gets overlooked because the question is naturally posed as yes-or-no.

6. Measure before removing. Spark's replacement was slower — 14 minutes against 9 — and that was fine, but they knew it rather than assuming it.

7. Most of the savings came from bus factor, not from removal. Making kept tools operable by more people saved more hours than deleting a tool did.

8. Require a named second operator before adopting anything. It forces the knowledge-sharing cost to the front, where it is an afternoon.

Questions for Discussion

1. Seven of eleven systems arrived in sixteen months, each for a good reason. Design a lightweight process that would have surfaced the accumulating total without slowing down any individual decision.

2. The bus-factor table was produced by asking "how many people can operate this?" Who should answer that question — the individuals, the manager, or a test? What is wrong with each option?

3. Maintenance hours nearly equalled incident hours. Why do you think operational-cost discussions focus almost entirely on incidents? What would make maintenance visible?

4. Great Expectations was removed because dbt tests covered most of what it caught. What was genuinely lost, and how would the team know if that loss mattered later?

5. Spark was narrowed rather than removed, based on a measurement showing DuckDB was 5 minutes slower. At what margin would you have removed it entirely? What else would you want to know?

6. The new rule requires naming a second operator before adoption. What happens when the honest answer is "there isn't one yet"? Should that block adoption, and what would you do instead?

7. Sixty of 128 saved hours came from bus-factor work that nobody predicted would save time. What does that suggest about how teams estimate the benefit of knowledge-sharing work — and how would you argue for it before you had this number?