Case Study 1: One Label, Nine Billion Series

"They added a label to help debug an incident. The label caused a bigger incident, and it took out the tool everyone was using to debug the first one."

Executive Summary

At 14:20 on a Thursday, an engineer investigating elevated checkout latency added customer_id as a label to Kestrel's HTTP request metrics, so they could see which customers were affected.

The metrics database was out of memory in eleven minutes. It stayed down for two hours and forty minutes. During that window Kestrel had no infrastructure monitoring at all — including for the original latency problem, which was still ongoing and which nobody could now see.

This case study is about cardinality, and about a second-order property that makes it worse than it first appears: the observability system is the thing you lose exactly when you most need it.

Skills applied: cardinality in time-series databases (§12.5); metrics versus business events (§12.5); blast radius (Chapter 3 §3.6); loud versus plausible failure (Chapter 2, Case Study 1).

Background

The metrics stack. Prometheus scraping about 340 targets, roughly 180,000 active series, 15-second scrape interval, 15-day local retention with longer-term data in a remote store. Comfortable on its allocation of 16 GB of memory, running at about 9 GB steady state.

The metric in question:

http_requests_total{method, status, endpoint, service}

• 4 methods × 6 status classes × ~200 endpoints × 3 services ≈ 14,400 series, of a 180,000 total.

The incident that started it. At 13:50, checkout p99 latency rose from 180 ms to about 900 ms. Not an outage, and not obviously attributable — it affected a minority of requests and the pattern was not clear from the existing labels.

The reasonable thought. "If I could see which customers are affected, I could tell whether it is one large account, a geography, or a client version." That is exactly the right investigative instinct.

The Problem

The change was one line:

# before
REQUESTS = Counter("http_requests_total", "HTTP requests",
                   ["method", "status", "endpoint", "service"])

# after -- deployed 14:20
REQUESTS = Counter("http_requests_total", "HTTP requests",
                   ["method", "status", "endpoint", "service", "customer_id"])

The arithmetic nobody did:

$$4 \times 6 \times 200 \times 3 \times 1{,}900{,}000 = 27{,}360{,}000{,}000 \text{ potential series}$$

In practice, only combinations that actually occur are created — but Kestrel serves a large fraction of its 1.9 million customers in any given hour, and each active customer immediately creates several series.

The timeline:

14:20  deploy. Prometheus memory 9.1 GB, 180k series.
14:23  first scrape after rollout. 340k series.  memory 10.4 GB
14:26  1.2M series.   memory 12.8 GB
14:29  3.9M series.   memory 15.2 GB.  scrape durations rising
14:31  OOM kill. Prometheus restarts, begins replaying its WAL.
14:33  replay completes, scrapes resume, series count climbs again
14:34  OOM kill.  ← crash loop

Prometheus was now in a crash loop, because each restart replayed the write-ahead log, resumed scraping, and re-ingested the same high-cardinality metric within three minutes.

⚠️ Failure Mode — The crash loop that re-creates its own cause

An out-of-memory kill on a metrics database is not a single event. It is the start of a loop, and the loop is what turns an eleven-minute mistake into a three-hour outage.

The mechanism: on restart, the database replays its WAL to recover in-memory state, then resumes scraping. Both of those re-create the condition that killed it. The WAL contains the high-cardinality series; the scrape targets are still exporting them.

Breaking the loop requires stopping the input, and the input is on 340 machines, not on the one that is failing.

Three ways out, in the order Kestrel tried them:

1. Roll back the deploy. Correct, and it takes as long as a deploy takes — which was 22 minutes, because the deploy pipeline itself has metrics-based health checks that were now failing.
2. Drop the metric at scrape time with a metric_relabel_configs rule. Faster, and it requires editing the metrics configuration and reloading — which the team had not practised.
3. Delete the WAL and start clean. Loses recent data, breaks the loop immediately. This is what finally worked, and it is the option nobody wanted to take because it destroys evidence.

The general lesson: know, in advance, how to make your metrics system drop a metric without deploying anything. It is a five-line configuration change and it is the difference between a twenty-minute incident and a three-hour one.

The Analysis

The two hours and forty minutes divide into three parts, and the middle one is the expensive one.

14:31–14:52 — diagnosis, 21 minutes. Slower than it should have been for a specific reason: the tool you use to diagnose a metrics problem is the metrics system. With Prometheus in a crash loop, the team had no graphs, no alerting, and no historical view. They diagnosed it from container logs and from the deploy history — "what changed at 14:20?" — which is the right question and took longer to ask than it would have with a working dashboard.

14:52–17:00 — recovery, 128 minutes. Rollback attempted, blocked by the deploy pipeline's own health checks. Relabel rule attempted, misapplied twice because nobody had written one before. WAL deletion finally applied at 16:48.

17:00 onward — the original problem. The checkout latency issue was still occurring. It was eventually traced to a slow query against dim_product after a statistics refresh had not run — a fifteen-minute fix that had been invisible for three hours.

🔎 Read the Plan — Find your cardinality before it finds you

Every time-series database can tell you its cardinality, and almost nobody looks until it is a problem. In Prometheus:

```promql

total active series

prometheus_tsdb_head_series

the ten metrics with the most series -- run this weekly

topk(10, count by (name)({name=~".+"}))

cardinality of one label across a metric

count(count by (customer_id) (http_requests_total)) ```

And the one that matters most, because it catches the problem before it is fatal:

```promql

rate of series growth. A step change here is somebody's new label.

deriv(prometheus_tsdb_head_series[30m]) ```

A step change in series count is always a deploy, and the alert is worth having:

yaml - alert: SeriesCountJump expr: | prometheus_tsdb_head_series > 1.25 * (prometheus_tsdb_head_series offset 1h) for: 5m annotations: summary: "Active series up >25% in an hour. Check recent deploys for a new label."

At the observed rate, this alert would have fired at 14:24 — four minutes after the deploy and seven minutes before the first OOM kill, with enough time to roll back cleanly.

The deeper problem: what the label was for

The engineer wanted per-customer latency. That is a legitimate need, and it was never a metrics question.

Per-customer latency is the right-hand column. Kestrel's clickstream already carried request timings, keyed by session and customer, landing in the lakehouse. The answer the engineer wanted was available in a warehouse query, in about ninety seconds, and nobody thought of it because the investigation was happening in the metrics tool.

That is the most transferable finding here: the tool you are already in shapes the solution you reach for, and the metrics tool is the one you are in during an incident.

The Decision

Five changes.

1. A cardinality limit at ingest, so the database defends itself:

# prometheus.yml
scrape_configs:
  - job_name: kestrel-services
    sample_limit: 50000          # per target, per scrape -- refuse beyond this
    label_limit: 12
    label_value_length_limit: 128

sample_limit is the important one. A target exporting more than 50,000 samples has its scrape failed rather than ingested. The failure is loud, attributable to one target, and does not take the database down — Chapter 2's fail-loudly principle, applied to observability itself.

2. The series-growth alert from the 🔎 callout, firing at 25% growth in an hour.

3. A forbidden-label list, checked in CI. A lint rule on metric definitions rejecting labels matching a pattern:

FORBIDDEN_LABEL_PATTERNS = [
    r".*_id$",        # customer_id, session_id, order_id, request_id
    r"^email$", r"^url$", r"^path$",    # unbounded strings
    r"^user.*", r".*message.*", r".*query.*",
]

With an exception list, because service_id and region_id are bounded and legitimate — and because Chapter 4's Case Study 1 established that widening a threshold to silence a known exception degrades the check for everything else.

4. A written relabel-drop procedure, and a quarterly drill. Five lines of configuration, in the runbook, with the exact command to reload without a deploy.

5. The metrics stack got a second, independent one. A minimal external monitor — is the site up, is checkout responding, is Prometheus itself alive — running outside the main stack, on a different provider, with three alerts. You cannot debug your monitoring with your monitoring.

📐 Design Decision — How much observability of the observability?

Change 5 provoked the real debate: this is a monitoring system for the monitoring system, and that regress has no natural end.

The case against: it is another system to operate (Chapter 5 §5.1's ceiling), it will have its own failures, and it duplicates a small part of what you already have.

The case for, which won: the failure mode of a monitoring system is that it fails silently and takes your ability to notice with it. Every other system in the platform has an external observer — the monitoring stack. The monitoring stack does not.

The resolution was to make it deliberately tiny: three checks, one alert channel, a hosted service rather than something they run, and an explicit rule that it will never grow. Requests to add a fourth check are refused, because the moment it becomes a real monitoring system it acquires the same failure mode.

What that gives up: it will not tell you what is wrong, only that something is. That is accepted — it exists to answer one question, "is the thing that answers questions alive," and answering more would defeat it.

What Happened

Over the following two years:

• sample_limit has fired eleven times. Nine were genuine high-cardinality mistakes caught at one target, with no impact on anything else. Two were legitimate growth requiring a raised limit, granted with review.
• The series-growth alert has fired four times, twice for real problems and twice on days when a new service was deployed — expected growth, and the alert was correct to ask.
• The CI lint rule has blocked seven pull requests. Six were straightforward accidents. One was the same engineer, eight months later, adding order_id to a metric — which is the point of a check that does not depend on memory.
• No metrics outage.

The external monitor has fired twice: once for a genuine Prometheus failure, and once for a false positive during a planned maintenance window that nobody had told it about.

The most-cited outcome internally is not any of the controls. It is the §12.5 table — metrics versus business events — printed and stuck on a wall. "Is this a metrics question or a warehouse question?" is now asked routinely, and the team's estimate is that it prevents more problems than the technical controls do.

Lessons

1. Cardinality, not volume, is what kills a time-series database, and it arrives in minutes.

2. Never put an unbounded identifier in a label. Customer, session, order, request, email, URL, error message. Enforce it in CI, with an exception list rather than a loosened rule.

3. An OOM kill on a metrics database starts a crash loop, because restart replays the WAL and resumes scraping — both of which re-create the cause. Breaking it requires stopping input on every target.

4. Know how to drop a metric without deploying. Five lines of relabel configuration, in the runbook, drilled. It is the difference between twenty minutes and three hours.

5. Alert on the rate of series growth. A step change is always a deploy, and the alert would have fired seven minutes before the first kill.

6. Set sample_limit so the database defends itself. A failed scrape on one target is loud, attributable, and harmless.

7. The tool you are already in shapes the solution you reach for. The answer to "which customers are affected" was a ninety-second warehouse query, and nobody thought of it because the investigation was happening in the metrics tool.

8. You cannot debug your monitoring with your monitoring. A deliberately tiny external observer, with an explicit rule that it will never grow.

Questions for Discussion

1. The engineer's instinct — see which customers are affected — was correct. What would have made the warehouse the obvious place to answer it? Is that a tooling problem or a habits problem?

2. Rollback was blocked by the deploy pipeline's metrics-based health checks. Is that a good design? Construct the case for health checks that do not depend on the metrics stack, and say what they cost.

3. The team deleted the WAL, destroying evidence, because it was the only fast option. Design the procedure that would let them break the loop without losing data. What does it cost to have ready?

4. The CI lint rule uses .*_id$ with an exception list. Estimate the exception list's size after five years. How would you keep it from becoming the rule?

5. The external monitor has an explicit rule that it will never grow. Is that sustainable? What happens the third time someone has a good reason to add a check?

6. The original checkout latency problem was invisible for three hours and turned out to be a fifteen-minute fix. How would you account for that cost when arguing for the five controls?

7. The most-cited outcome was a table on a wall, not a technical control. What does that suggest about the ratio of effort you should spend on controls versus on shared mental models?