Case Study 2: The Company That Didn't Need a Data Platform

"Our worst query took four seconds. We spent nine months and two engineers getting it to two."

Executive Summary

A software company with about forty engineers and eighteen million rows of transactional data spent nine months building a data platform — object storage, Spark, an orchestrator, a warehouse, and a streaming pipeline — staffed by two engineers full time.

Their largest table was 18 million rows. Their slowest analytical query ran in four seconds on the existing PostgreSQL read replica. After the migration, it ran in two.

This case study examines how that decision got made, why nobody stopped it, what the platform actually cost, and what the company did afterwards. It is the counterweight to every other case study in this book: the failure here is not under-engineering, and the people involved were competent.

It also does the thing the chapter asks for — it identifies the specific conditions under which the platform would have been correct, so that the lesson is "check the premises" rather than "never build a platform."

Skills applied: when you need none of this (§5.7); the systems-per-engineer heuristic (§5.1); buy / build / do without (§5.3); reversal conditions (Chapter 3 §3.7).

Background

The company. A business-to-business software company, roughly 40 engineers, 400 customers, about $14M in annual recurring revenue. PostgreSQL running the application, with a read replica that analysts and a BI tool queried directly.

The data, honestly measured — and it is worth noting that nobody measured this until the audit described below, eight months into the project:

Fifteen and a half gigabytes. That number is the entire case study.

The complaints that started it, all genuine:

1. Analytical queries occasionally slowed the read replica, and once caused a two-minute delay in a customer-facing report.
2. Two analysts each maintained their own SQL, and their numbers disagreed.
3. Combining application data with Stripe billing data and Salesforce CRM data required manual CSV exports.
4. A new engineer had asked "where do I find revenue by cohort?" and nobody had a good answer.

Every one of those is a real problem. Note what they are not: none of them is "our data is too large" or "our queries are too slow."

The Problem

The proposal that got approved described a platform: object storage for raw landing, a cloud warehouse, Spark for transformation, an orchestrator, and a streaming pipeline for the event table. Nine-month timeline, two engineers.

Why it was approved. Four reasons, and only the first is about the problem:

1. The four complaints were real and the platform did address them.
2. A newly hired senior engineer had built exactly this at a previous company, where it had been necessary, and proposed it with justified confidence.
3. Nobody in the approval chain had a framework for asking whether the scale warranted it. The chief technology officer was a strong engineer with no data background, and the question "how big is our data?" was never asked in a meeting where the answer would have changed anything.
4. The proposal's framing was "modernize our data infrastructure," which is unopposable. Nobody argues for unmodern infrastructure.

⚠️ Failure Mode — Architecture transplanted without its premises

This is the single most common mechanism of over-engineering in data work, and it is not a knowledge failure.

The senior engineer was right about everything except applicability. At their previous company — 900 engineers, 40 TB, several thousand events per second — the platform was correct and they had the scars to prove it. They transferred a solution that had been earned, to a situation whose premises they did not re-derive.

Experience transfers; premises do not. The value of having built something before is substantially in knowing why each piece was there, and that knowledge is exactly what gets lost when the pattern is applied somewhere new — because the pieces are memorable and the reasons are not.

The inoculation is one question, asked before any platform project: what is our largest table, and how long does our worst query take on the system we already have? Ten minutes, and it would have reframed this entire project.

This cuts both ways, and it is worth saying: the same mechanism produces under-engineering, when someone from a small company joins a large one and insists that PostgreSQL is fine. The failure is not the direction; it is not checking.

The Analysis

Eight months in, with the platform mostly built and the migration underway, a new director of engineering asked for a benchmark comparing old and new.

The benchmark. Ten representative analytical queries, run against the PostgreSQL read replica and against the new warehouse.

Every query was faster. The slowest query went from 4.1 seconds to 2.0.

Then the director asked the question that ended the project as conceived: what happens to the PostgreSQL numbers if we add appropriate indexes?

With two indexes added — about thirty minutes of work — PostgreSQL was faster than the warehouse on the two slowest queries. Which should not be surprising: at 15.5 GB, the entire dataset fits in the page cache of a moderately sized machine, and a well-indexed local read beats a distributed one.

🔎 Read the Plan — Why nobody had looked at the indexes

The cohort retention query's plan on the original replica, reduced to the relevant line:

text Seq Scan on events (cost=0.00..412893.00 rows=18200000 width=64) Filter: ((created_at >= '2024-01-01') AND (account_id = 8841)) Rows Removed by Filter: 18,196,204

A sequential scan over 18.2 million rows to return 3,796 of them. There was no index on (account_id, created_at).

This is not an exotic finding. It is the first thing anyone would check, and nobody checked, for a reason worth naming: the complaint had been framed as an architecture problem from the first sentence. Once a problem is framed that way, the investigation looks for architectural answers, and a missing index is not one.

The general habit: before accepting that a system is too slow, look at one plan. Not a profile, not a benchmark suite — one EXPLAIN on the slowest query. It takes two minutes and it reframes a surprising number of conversations.

What the four original complaints actually needed

This is the part that generalizes. Taking each complaint on its own terms:

Roughly four weeks and one subscription, against nine months and two engineers.

And note that the second and third answers are tools from the modern data stack. The failure was not adopting modern tooling; dbt and managed ingestion were the right answers. The failure was adopting the storage and processing architecture, which is where all the cost was and which addressed none of the four complaints.

That distinction is the most useful thing in this case study. The complaints were about consistency, integration, and documentation. The project delivered scale and performance. Those are different problems, and the platform solved the one nobody had.

The Decision

The project was not cancelled — a decision the director defended and which is worth examining.

What was kept: the warehouse (already provisioned and paid for, and genuinely nicer for analysts), dbt, and the managed ingestion tool.

What was cancelled: Spark, the streaming pipeline, and the object-storage landing layer. Between them, roughly four of the nine months of remaining work.

What was added: the two indexes and a dedicated analytics replica, in the first week after the benchmark.

Why not revert entirely? The warehouse was already running and the migration was mostly done, so reverting had a cost too. This is a sunk-cost argument and the director was aware of that, but the forward-looking version also held: the marginal cost of keeping a provisioned warehouse was low, and it removed the analytical load from the application database permanently rather than by convention.

What would have been different with the benchmark on day one: dbt against PostgreSQL, a managed ingestion tool, two indexes, an analytics replica. About a month. The warehouse would have been a later decision, made when data volume or concurrency actually demanded it.

What Happened

Eighteen months later:

• The company had grown to 26 GB of analytical data. Still comfortably within PostgreSQL's range.
• The warehouse was retained and used, primarily for concurrency — 30 analysts and BI users hitting it simultaneously, which is genuinely where a warehouse earns its place, and which was never the reason it was proposed.
• dbt was the highest-value adoption by a distance. The two analysts' disagreeing numbers had been the most damaging original complaint and it was solved by a Markdown file and a models/ directory.
• Spark was never revisited.
• The senior engineer who proposed the platform left after a year. The exit interview recorded frustration at the project's reduction; they still believed it was the right architecture.

That last point deserves to be taken seriously rather than treated as a punchline. They may be right about the destination and were wrong about the timing, and the difference matters: a platform built for a scale you reach in five years costs five years of complexity you did not need. The correct move is not to refuse the architecture but to write down the condition that triggers it — which is exactly what Chapter 3's "what would reverse this" section is for.

The company later adopted that practice. Their adr-001 records, in effect: revisit the storage architecture when analytical data exceeds 500 GB, when concurrent BI users exceed 50, or when a single analytical query cannot be made to run in under 10 seconds with indexing. Three observable conditions, checked quarterly.

Lessons

1. Measure before you architect. Largest table, worst query, on the system you already have. Ten minutes.

2. Architecture transplanted without its premises is the most common over-engineering mechanism. Experience transfers; premises do not, because the pieces are memorable and the reasons are not.

3. Look at one query plan before accepting that a system is too slow. A sequential scan over 18 million rows was the actual problem, and nobody looked because the complaint had been framed architecturally from the first sentence.

4. Match the solution to the complaint. The four complaints were about consistency, integration, and documentation. The project delivered scale and performance. Read your complaint list and ask what category each one is in.

5. "Modernize our infrastructure" is unopposable and therefore useless as a project justification. Require a statement of which specific problem is being solved and what the current system's measured shortfall is.

6. Some modern tooling was exactly right. dbt and managed ingestion solved the two most damaging complaints in three weeks. The failure was the storage and processing architecture, not the category of tool.

7. A warehouse's real justification is often concurrency, not volume. Thirty simultaneous BI users is a genuine reason. It was never the reason given.

8. Write the trigger condition rather than refusing the architecture. The proposer may be right about where you are going and wrong about when. Three observable conditions, checked quarterly, converts an argument into a schedule.

Questions for Discussion

1. The four original complaints were all real. Rank them by damage to the business, and say whether your ranking changes which solution you would build first.

2. Nobody asked "how big is our data?" in a meeting where the answer would have mattered. Whose job was that? Design the single question that a non-specialist approver could ask on any infrastructure proposal.

3. The director did not cancel the project entirely, partly because the warehouse was already provisioned. Distinguish the sunk-cost part of that reasoning from the forward-looking part. Which would you have weighted more?

4. The senior engineer still believed the architecture was right and left after a year. Was the company's handling of this fair? What would you have done differently as their manager, before and after the benchmark?

5. The company's ADR now names three reversal conditions: 500 GB, 50 concurrent users, or a query that cannot be indexed under 10 seconds. Critique each threshold. Which is most likely to fire first, and is that the one you would most want to catch?

6. dbt was described as the highest-value adoption. It is also a "modern data stack" tool. What distinguishes it from the parts that were cancelled — is there a general principle, or was it just luck?

7. This case study is the counterweight to the rest of the book's warnings about under-engineering. Having read both, what single question would you ask on encountering an unfamiliar data platform to tell which failure you are looking at?