Case Study 2: The Feature That Was Right in Both Places and Wrong in Between

"Both implementations were correct. We tested both. We tested them against each other and they matched. What we never did was test them on the same customer at the same moment."

Executive Summary

Kestrel's fraud check runs at checkout, in under 200 ms, and uses six features. One of them, days_since_last_order, was computed by two different code paths — SQL over the warehouse for training, Python over the OLTP database for serving.

Both implementations were tested. Both passed. They disagreed on 5.37% of entities, and the disagreement was not random:

entities compared               20,000
disagreements                    1,074   (5.37%)
    boundary (< vs <=)             944
    null vs zero                   130

of customers who ordered ON the prediction day:
    disagreements                  944 of 963   (98%)

The skew was concentrated entirely on customers who had ordered recently — which is the population the fraud model is most confident about and most likely to act on. The model's best predictions were its most corrupted ones.

The fix was not better testing. It was one implementation, and the migration took four weeks. The skew rate has been 0.00% for fourteen months and is still computed daily.

Skills applied: training/serving skew (§32.7); the row-level diff; why aggregate comparisons miss it; and the discovery that two bugs were cancelling in exactly the cases an engineer spot-checks.

Background

The two paths existed for a good reason and neither author was careless.

The training path was written first, by a data engineer, as part of a dbt model:

SELECT customer_id,
       DATEDIFF('day', MAX(order_ts), :as_of) AS days_since_last_order
  FROM fct_orders
 WHERE order_ts <= :as_of
 GROUP BY customer_id

The serving path was written eight months later, by a backend engineer, because the checkout service needs the feature in under 200 ms and cannot query the warehouse:

def days_since_last_order(customer_id, as_of):
    row = db.query_one(
        "SELECT MAX(placed_at) AS last FROM orders "
        "WHERE customer_id = %s AND placed_at < %s",   # note: <
        (customer_id, as_of))
    if row.last is None:
        return 0                                        # note: 0
    return (as_of - row.last).days

Three differences, all invisible to the person who wrote the second one, because they had the feature's name and its description and not its SQL:

The third difference is the subtlest. The warehouse retains cancelled orders with a status flag; the OLTP table's application-level query filtered them out through a default scope nobody remembered.

What testing existed:

• The SQL model had dbt tests. Not null, non-negative, within a plausible range. All passed.
• The Python function had unit tests. Six cases, hand-written. All passed.
• Someone had compared the two. They pulled a hundred customers from each and compared the distributions. Mean 34.2 days versus 34.6 days. Judged equivalent.

The Problem

The model performed acceptably, which is why this took eleven months to find.

The trigger was a fraud analyst's complaint, not a monitoring alert: "the model keeps flagging customers who just placed an order two minutes ago."

That is precisely the boundary bug, described in the only vocabulary available to someone who does not know the code exists. A customer who orders today has days_since_last_order = 0 in training and whatever their previous gap was in serving — 12, 27, 8 days — so a customer in the middle of a normal purchase looks, to the deployed model, like one who has been dormant.

⚠️ Failure Mode — the distribution comparison that certified the bug

Someone did compare the two implementations, and the comparison passed. This is the part worth studying, because the check was not lazy — it was the wrong shape.

text WHAT WAS COMPARED training serving verdict mean days_since_last_order 34.2 34.6 "equivalent" p50 21 22 "equivalent" p99 187 187 "equivalent" null rate 0.65% 0.00% not noticed

Two independent samples of a hundred customers each. The distributions genuinely are almost identical, because the disagreement affects 5% of rows and shifts them by a few days each — which moves the mean by 0.4 days and moves nothing else at all.

The check that finds it is a different operation entirely: the same entity, at the same timestamp, through both paths, differenced.

text for entity, ts in sample: a = training_path(entity, ts) b = serving_path(entity, ts) if a != b: disagreements += 1

Comparing distributions asks "do these look alike?" Comparing rows asks "are these the same?", and only the second question has a correct answer of zero.

The null rate was visible in the table above and nobody read it. 0.65% versus 0.00% is the null-versus-zero bug, sitting in plain sight in the comparison that certified the feature — because the reviewer was looking at the three rows they expected to matter.

The Analysis

The row-level diff was written in an afternoon and produced the numbers in the summary. The structure of the disagreement is what changed the response.

customers by recency of last order      n        disagree     rate
────────────────────────────────────────────────────────────────────
ordered on the prediction day         963           944       98.0%
ordered 1-7 days ago                6,511             0        0.0%
ordered 8-30 days ago               9,834             0        0.0%
ordered 31+ days ago                2,562             0        0.0%
no prior order                        130           130      100.0%
────────────────────────────────────────────────────────────────────
                                   20,000         1,074        5.37%

🔎 Read the Plan — a 5% error rate that is really a 98% error rate

The headline number is 5.37% and it is misleading in the safe direction, which is the worst direction for a number to mislead in.

Two populations are affected and each is affected almost totally:

• Customers who ordered today: 98% wrong.
• Customers with no order history: 100% wrong.
• Everyone else: 0% wrong.

This is the general shape of training/serving skew and it is worth expecting. Skew comes from a logic difference, and a logic difference has a condition — a boundary, a null, a filter. So the error is total inside the condition and absent outside it, never spread evenly.

Which means the aggregate rate is close to meaningless. 5.37% invites a judgment about whether 5% matters; the real question is which 5%, and here it is the two populations a fraud model cares about most: people transacting right now, and people with no history.

The operational consequence at Kestrel: a customer placing their second order in a week — a normal, low-risk pattern — was scored as if they had been dormant for their previous inter-order gap. Elevated fraud scores on exactly the loyal, high-frequency customers, which is both the most damaging population to inconvenience and the one whose complaints reach a fraud analyst rather than a dashboard.

Report skew by segment, never as a single rate.

Then the second finding, which confused the team for a day.

🏭 From the Pipeline — nineteen customers where the two bugs cancelled

Of the 963 same-day orderers, 19 agreed. They are the customers whose first ever order was on the prediction day.

text training: sees today's order (<=) -> 0 days serving: finds no prior order (<) -> None -> 0 by the null bug both return 0

Two defects, opposite signs, identical output.

This explains the unit tests. The Python function's six hand-written cases included "a brand new customer" and "a customer who just ordered" — and both fall in the cancelling region. The tests were not weak; they were drawn from the same intuition that produces the cancelling cases, which is a systematic bias rather than bad luck. An engineer choosing example rows by hand gravitates toward the simplest ones, and the simplest ones are where compensating bugs hide.

And the genuinely confusing part: Kestrel fixed the null handling first, as the smaller change. The number of disagreeing customers went up, from 1,074 to 1,093, because the 19 cancelling rows stopped cancelling. A bug fix that makes a metric worse is alarming, and the team briefly reverted it.

The lesson is to expect it. When two defects interact, fixing either one increases the other's visible blast radius, and the correct response is to fix both rather than to trust the metric's direction over one change.

The Decision

One implementation. The rest is migration.

The feature is defined once, in a Python module both paths import:

# platform/ml/features/recency.py
def days_since_last_order(orders, as_of):
    """orders: iterable of order timestamps, INCLUDING cancelled.
    Returns None when there is no prior order. The boundary is inclusive."""
    prior = [o for o in orders if o <= as_of]
    return (as_of - max(prior)).days if prior else None

The training path calls it through a UDF; the serving path calls it directly. The three differences disappear, not because both were fixed, but because there is no longer a second thing to be different.

📐 Design Decision — one implementation, and the three costs of getting there

"Just share the code" is easy to say and took four weeks. The obstacles are worth naming because they are the same obstacles everywhere.

Cost one: the serving path must not query the warehouse. The shared function takes orders as an argument rather than fetching them, so each path supplies its own data access. This is the design move that makes sharing possible — separate the computation from the retrieval, and share only the computation.

Cost two: the sources had to be reconciled. The OLTP table excluded cancelled orders through a default scope; the warehouse retained them. Sharing the function does not fix that — it moves the disagreement from the function into its input, which is worse, because it is now invisible. Kestrel made the serving query explicit about cancelled orders and added an assertion comparing daily order counts between the two sources.

Cost three: a SQL path calling Python is slower. The training job's runtime went from 4 minutes to 11. This was accepted without much argument, and the reasoning generalizes: a nightly job that takes 7 minutes longer costs nothing, and correctness in a feature used by a production decision costs a great deal. The trade is only difficult when the shared implementation is on a latency-sensitive path, which is the serving side — and there it was already Python.

The alternative Kestrel rejected: generating both implementations from one specification. It is elegant, it is real (feature stores do it), and it was more machinery than a twelve-feature model justified. §32.9's threshold, applied to a build-versus-buy decision inside the fix.

And the monitor:

platform/ml/skew_check.py --daily
  sample 20,000 entities x their most recent scoring timestamp
  compute each feature through BOTH paths
  report disagreement rate PER FEATURE and PER RECENCY SEGMENT
  alert if any segment's rate is non-zero

What Happened

False-positive fraud flags on same-day repeat purchasers fell by 71%, which is the outcome the analyst had been describing eleven months earlier in the only language available to them.

The skew rate has been 0.00% for fourteen months and is still computed daily.

🧱 Kestrel Platform — why a metric that is always zero is worth computing

The daily skew check has never fired since the migration, and it has been proposed for deletion twice on the grounds that it is not telling anyone anything.

The argument for keeping it:

• It is the only monitor whose non-zero value means something immediately. No investigation, no threshold tuning, no "is this seasonal" — any value above zero is a defect. Chapter 25 §25.7's point about alerts that require interpretation, inverted.
• The failure it guards against is re-divergence, which happens through an ordinary change: someone optimizes the serving path, or the warehouse model is rewritten. The check turns a six-week investigation into a one-day one.
• It costs about ninety seconds of compute a day.

The general principle: a check that has never fired is not evidence that it is unnecessary. It is evidence that the thing it guards is currently correct — and the two are only the same if nothing ever changes.

Lessons

1. Two implementations of one feature will diverge. Not through carelessness — the two paths have different latency requirements, different languages, different data sources, and different authors separated by eight months.

2. ⚠️ A distribution comparison certifies the bug. Mean 34.2 versus 34.6, p50 21 versus 22 — genuinely equivalent, and genuinely wrong. Comparing distributions asks "do these look alike"; comparing rows asks "are these the same." Only the second has a correct answer of zero.

3. The null rate was in the comparison table and nobody read it. 0.65% versus 0.00%.

4. 🔎 A 5.37% skew rate was really a 98% rate on the population that mattered. Skew comes from a logic difference, a logic difference has a condition, and the error is total inside the condition and absent outside it. Never report skew as a single number.

5. The affected populations were customers transacting right now and customers with no history — the two a fraud model cares about most. The model's most confident predictions were its most corrupted.

6. 🏭 Nineteen rows agreed because two bugs cancelled, and they are exactly the cases the unit tests covered: "a new customer" and "a customer who just ordered." Hand-picked examples gravitate toward the simplest rows, and compensating bugs hide there.

7. Fixing one of two interacting bugs makes the metric worse. Kestrel briefly reverted a correct fix because the disagreement count rose from 1,074 to 1,093. Expect it.

8. 📐 Sharing an implementation means separating computation from retrieval. The shared function takes the data as an argument; each path fetches its own.

9. Sharing the function does not reconcile the sources — it moves the disagreement into the input, where it is invisible. Assert that the sources agree, separately.

10. A 7-minute-slower nightly job in exchange for a correct production feature is not a difficult trade, and it is only difficult when the shared code sits on the latency-sensitive path.

11. 🧱 A monitor that has never fired is not unnecessary. It is evidence the guarded thing is currently correct, and that is only the same thing if nothing ever changes.

12. The bug was reported by a fraud analyst in plain language eleven months before anyone found it. "It flags customers who just ordered" was a precise description of the boundary condition.

Questions for Discussion

1. The distribution comparison was a reasonable check that gave a wrong answer. What made someone choose that check, and what would make the row-level diff the obvious default?

2. The analyst's complaint was an accurate bug report in non-technical language. How would you build a path from complaints like that to the engineers who can act on them?

3. Skew was 0% for three of five recency segments. Design the segmentation you would use for a feature you own — and justify why those segments.

4. Kestrel rejected generating both implementations from one specification as over-engineering at twelve features. At what point does that flip?

5. The shared function takes orders as an argument. What does this design cost when a feature needs data the serving path cannot cheaply fetch?

6. The daily skew check has been proposed for deletion twice. Write the strongest case for deleting it, then rebut it.

7. Two bugs cancelled in the cases the unit tests covered. Is there a test-selection strategy that systematically avoids this, or is population comparison the only real answer?