Key Takeaways: ML Engineering and Feature Stores
The one thing
A large share of what presents as machine learning failure is data engineering failure — and it is diagnosed by the wrong specialty, because the symptom looks like a model problem and the cause is a join, a null, a timestamp, or two implementations of one feature.
The diagnostic that redirects the whole investigation
good in training, bad in production, IMMEDIATELY -> leakage or skew (yours)
good in production, degrading over weeks -> staleness (yours)
good in production, degrading over months -> drift (shared)
never good anywhere -> modeling or signal (not yours)
Ask about the timing before anything else. Case Study 1 spent six weeks inside the wrong frame because nobody asked.
"Never good anywhere" is not your problem, and saying so plainly is better than accepting responsibility for something you cannot change.
What an ML team needs
Five things, in the order they cause pain: the same feature computed once · correct history · freshness you can measure · labels with the same care as features · a way to ship a feature without you.
🔎 Four questions before building anything: When does the model need this — past or now? What timestamp should features be as-of? How old may they be? Who else uses it? Kestrel ran these against nine requests and built a feature-store integration for two.
Point-in-time correctness
A feature is a value about an entity, at a moment — and the third part is what gets dropped. A feature without an as-of is a fact about now, and a fact about now is what you must not train on.
⚠️ Leakage is measurable. Same feature, two joins: AUC 0.830 naive, 0.591 as-of, differing on 97.1% of rows. The gap is not a better model — it is the label, read back through a feature computed after the outcome.
The mechanism generalizes: any feature whose trajectory differs by outcome leaks when joined at its current value. Churners stop ordering, so churned and retained customers sit at 5.0 versus 6.5 orders at the prediction point and 6.9 versus 16.3 today. The 2.4× gap is the label.
It is invisible at every checkpoint. The query is a primary-key join. Every assertion passes. Cross-validation is tight. The metric is excellent — and the failure mode of leakage is good news, which nobody investigates.
Cross-validation measures consistency, not correctness. The leak is in every fold.
🔎 The one check: train to day N, evaluate strictly after. 0.830 → 0.594, and 0.594 against production's 0.59 is what ended the argument. Reproducing a failure beats explaining it.
⚠️ When a series of experiments produces the same result, the constant is the finding. Five models, five CV scores, one identical production number.
The as-of join
The boundary is inclusive — valid_from <= event_ts. The most common off-by-one here.
📐 NULL is not 0. 130 of 20,000 rows had no observation. COALESCE(x, 0) tells the model a
new customer is a dormant one, and the two behave nothing alike. Pass the null through, add an
indicator, or impute and record it — never silently. A null filled in without a trace is a fact you
have deleted.
The feature table must be append-only. A nightly full refresh has no history to join to, and the as-of join silently degrades into the naive one.
Test the invariant, not examples: no returned value may have a timestamp after its event, asserted over every row.
📐 Prefer removing the wrong option to documenting it. The wrong table was a one-line join, the right one a lateral join with a predicate — documentation does not change relative difficulty, and the failure gives no feedback for weeks. The current value is a special case of the as-of query, so deleting it cost nothing.
Feature age
📏 The number nobody computes. Trained at p50 11 days, served at 2 hours — a 132× mismatch on Kestrel's two strongest churn features.
Fresher-in-serving is the quiet failure and degrades performance by an amount that looks like noise. Staler-in-serving is loud and usually means materialization is behind.
The requirement is that training and serving agree, not that both are fast. Sometimes the fix is to serve staler features deliberately.
Training/serving skew
Two implementations of one feature will diverge — different latency needs, languages, sources, and authors eight months apart. Not carelessness; structure.
⚠️ A distribution comparison certifies the bug. Mean 34.2 versus 34.6, p50 21 versus 22. Comparing distributions asks "do these look alike"; comparing rows asks "are these the same." Only the second has a correct answer of zero.
🔎 Skew is never uniform, so the aggregate rate misleads. 5.37% overall was 98% of same-day orderers and 100% of customers with no history, and 0% everywhere else — because a logic difference has a condition, and the error is total inside it. The model's most confident predictions were its most corrupted. Report skew by segment, never as one number.
🏭 Nineteen rows agreed because two bugs cancelled, and they are exactly the cases hand-written unit tests covered — "a new customer", "a customer who just ordered." Hand-picked examples gravitate to the simplest rows, and compensating bugs hide there.
Fixing one of two interacting bugs makes the metric worse. 1,074 → 1,093. Expect it, and do not trust the metric's direction over a single change.
📐 The fix is one implementation, which means separating computation from retrieval. The shared function takes data as an argument; each path fetches its own. Sharing the function does not reconcile the sources — assert that separately.
A 7-minute-slower nightly job for a correct production feature is not a difficult trade.
The two stores
Offline: all history, columnar, immutable, for training as-of joins. A Chapter 20 Type 2 model is exactly this.
Online: latest value only, key-value, sub-10ms, for serving.
Materialization is the job that connects them and the component that fails. Chapter 25's freshness monitoring applies directly.
💸 The online store is priced by writes. 1,904,221 customers × 12 features refreshed hourly is 548,415,648 writes/day; changed-only is 71× less. It needs a weekly full refresh, because a diff-based job cannot notice a key the store silently lost.
The threshold
📐 You need a feature store when you have all three of: online inference · the same feature computed in two places · more than one consumer.
With one, buy nothing. With two, write the missing piece.
Seven of Kestrel's nine requests were satisfied by a dbt incremental model with valid_from /
valid_to — already tested by Chapter 23, catalogued by Chapter 30, and covered by Chapter 31's deletion
manifest, all of which a separate feature store would have needed re-solved.
Backfills, versions, labels
🔁 A backfill to a feature table invalidates the models trained on it. Kestrel's corrected values ran 3% lower for five weeks and the model over-predicted churn — which looked like drift. Store training sets as content-addressed snapshots, and fail a backfill that cannot enumerate its consumers from the catalog's lineage.
A materially changed definition is a new feature with a new name. §30.8's rule, in a new place.
Labels need a definition, two timestamps, a delay, and a known source. A 90-day churn definition means your newest training data is a quarter old — a hard constraint that changes what is possible.
If your labels come from a rules engine or an earlier model, you inherit its blind spots, and the loop tightens every retrain.
🔐 A training set is an immutable snapshot of personal data, which is what makes it undeletable by design. Keep the identifiers out of it, bound the retention to the model's life plus an audit cycle, and be honest that the position is about the training set rather than the model weights.
Monitoring
Four signals, all available immediately, all data engineering:
- Feature distributions, training versus serving — the highest-value monitor in ML infrastructure.
- Feature freshness — age at serving, per feature.
- Null rates — a feature going 0.6% → 40% null produces no error; the model scores every row.
- The prediction distribution — cheap, and a leading indicator.
Not accuracy. It needs labels, and a 90-day churn label makes it knowable 90 days late.
🧱 A monitor that has never fired is not unnecessary. Kestrel's skew check has been 0.00% for fourteen months and costs ninety seconds a day. It is the only monitor whose non-zero value means something immediately — no threshold tuning, no seasonality question. "It has never fired" is evidence the guarded thing is currently correct, and that is only the same thing if nothing ever changes.
The boundary
Own: feature pipelines, point-in-time correctness, the as-of join, both stores, materialization, freshness and skew monitoring, label plumbing, training-set snapshots, and the interface that lets a data scientist ship a feature without you.
Share: feature definitions, label definitions, drift investigation.
Do not own: model selection, hyperparameters, model performance in general, or whether the data contains enough signal.
"The model isn't good enough" is not your ticket. Answer the four questions you can — leaking, skewed, stale, mislabelled — in a week. If all four come back clean, say so with evidence rather than with a boundary.
The code
code/pit_join.py — leakage measured two ways, a sort-merge as-of join with feature-age reporting,
and a row-level skew diff with three injected bugs. Forty-two self-checks. --demo runs all three;
Exercises 32.11 and 32.13 extend it.