Quiz: ML Engineering and Feature Stores

DataField.Dev

Quiz: ML Engineering and Feature Stores

Twelve questions. Answers with explanations follow — work through them first.

1. A model scores well offline and badly in production from the day it is deployed. According to §32.2, which two causes should you suspect before the model?

A. Drift and insufficient signal
B. Leakage and training/serving skew
C. Overfitting and regularization
D. Label noise and class imbalance

2. §32.3 says a feature is a value about an entity, at a moment. Which part is the one that gets dropped?

A. The entity
B. The value
C. The as-of timestamp
D. The entity key

3. In pit_join.py, the same single feature scores AUC 0.830 with a naive join and 0.591 with an as-of join. What is the 0.239 gap?

A. The benefit of a better join algorithm
B. Noise, since the fixture is synthetic
C. The label, read back through a feature computed after the outcome
D. The difference between training and test distributions

4. Why does five-fold cross-validation fail to detect leakage?

A. Five folds are too few
B. The leak is present in every fold, so cross-validation measures consistency rather than correctness
C. Cross-validation does not compute AUC
D. The folds are too small to be representative

5. §32.5's 📐 callout says COALESCE(orders_to_date, 0) is wrong for the 130 entities with no observation. Why?

A. Zero is out of range for that feature
B. It changes the mean
C. Zero says "zero orders," which describes a dormant customer, while the entity is a new one
D. It makes the join non-deterministic

6. In the fixture, features at training time are a median of 11 days old, and serving fetches features 2 hours old. §32.6 says the fix is sometimes to:

A. Retrain more often
B. Make serving staler, deliberately
C. Increase the feature store's cache TTL
D. Add feature age as a model input

7. The skew fixture reports a 5.37% disagreement rate. Why does §32.7 argue that number is misleading?

A. It is too small to matter
B. Skew comes from a logic difference, which has a condition — so the error is total inside the condition (98% of same-day orderers) and absent outside it
C. The sample is too small
D. It should be measured per feature, not per entity

8. Nineteen of 963 same-day orderers agreed between the two code paths. Why, and why does it matter?

A. They were excluded from the comparison
B. Their orders were cancelled
C. Two bugs cancelled for customers whose first order was that day — and those are exactly the cases hand-written unit tests cover
D. Their feature values were null in both paths

9. §32.8: materializing only entities whose value changed gives a 71× reduction in online-store writes. What does it require you to also do?

A. Increase the refresh interval
B. A weekly full refresh, because a diff-based job cannot detect a key the store silently lost
C. Store the features twice
D. Disable the offline store during materialization

10. §32.9's threshold for needing a real feature store requires all three of:

A. More than ten models, a data science team, and a cloud budget
B. Online inference, the same feature computed in two places, and more than one consumer
C. Point-in-time correctness, backfills, and versioning
D. Streaming data, low latency, and a model registry

11. §32.10: a correct backfill to a feature table silently invalidated a model for five weeks. What was the visible symptom?

A. The training job failed
B. The online store fell behind
C. The model began over-predicting churn, which looked like drift
D. Feature nulls increased

12. §32.12 lists four things to monitor and deliberately excludes model accuracy. Why?

A. Accuracy is the modeling team's responsibility
B. Accuracy requires labels, and labels have a delay — a 90-day churn label makes accuracy knowable 90 days late
C. Accuracy is expensive to compute
D. Accuracy is already covered by the prediction distribution

---

Answers

1 — B. Leakage and training/serving skew.

The timing is the diagnostic. An immediate gap means offline and production are not evaluating the same thing — either a feature encoded information that did not exist at prediction time, or the serving path computes it differently from the training path. Drift and staleness cause gradual degradation, and a model that has never been good anywhere is a modeling or signal problem and not yours. Case Study 1 spent six weeks on hypothesis C before anyone asked the timing question.

2 — C. The as-of timestamp.

A feature without an as-of is a fact about now, and a fact about now is exactly what you must not train on. The dropped timestamp is what makes the naive join available at all: a feature table with no timestamp column cannot be joined correctly, and nothing about the resulting query looks wrong. Note that a Chapter 20 Type 2 dimension already carries the as-of, which is why teams that have one have often solved this without knowing it.

3 — C. The label, read back through a feature computed after the outcome.

Churners stop ordering, so a churner's current orders_to_date is close to their value at the prediction point while a retained customer's has grown substantially. The difference between then and now is the label. Measured at the prediction point the two groups barely separate (5.0 versus 6.5 orders), and measured today they are 2.4x apart (6.9 versus 16.3) — which is the whole of the 0.239.

4 — B. The leak is present in every fold.

Cross-validation checks that a model generalizes across random subsets of the same corrupted data. Five folds agreeing to ±0.004 means the leak is reliably present, not that the model is reliably good. This is why a time split is the diagnostic: train to day N, evaluate strictly after. Case Study 1's numbers were 0.831 by cross-validation and 0.594 by time split, and the second reproduced production's 0.59 exactly — reproducing the failure is what ended a six-week argument.

5 — C. Zero describes a dormant customer; the entity is a new one.

NULL means "no observation" and 0 means "zero orders," and a new customer and a dormant customer behave nothing alike. The damage is concentrated where it hurts most: 0.65% of rows sounds ignorable, and those rows are disproportionately new customers, who are disproportionately the population a propensity or churn model is deployed to act on. Pass the null through (trees handle it natively), or add an explicit indicator, or impute and record it — but never silently, because a null filled in without a trace is a fact you have deleted.

6 — B. Make serving staler, deliberately.

The requirement is that training and serving agree, not that both are fast. A model trained on features a median of 11 days old has learned the relationship between the label and an 11-day-old feature; serving it a 2-hour-old value gives it an input it has never seen. Serving the feature at the age it was trained on sounds perverse and is correct. Retraining on fresher features (A) is the other valid resolution, and which one you pick depends on whether fresh features are actually available in the offline history.

7 — B. The error is total inside a condition and absent outside it.

Skew comes from a logic difference — a boundary, a null, a filter — and a logic difference has a condition. In the fixture, 98% of customers who ordered on the prediction day disagree, 100% of customers with no history disagree, and every other recency segment is at exactly 0%. The aggregate rate invites a judgment about whether 5% matters; the real question is which 5%, and here it is the two populations a fraud model cares about most. Report skew by segment, never as a single rate.

8 — C. Two bugs cancelled, in exactly the cases hand-written tests cover.

The training path sees today's order and returns 0; the serving path finds no prior order and returns 0 through the null-handling bug. Two defects, opposite signs, identical output. The Python function's unit tests included "a brand new customer" and "a customer who just ordered" — both in the cancelling region. This is a systematic bias rather than bad luck: engineers picking example rows gravitate toward the simplest ones, and compensating bugs hide there. Expect also that fixing one of two interacting bugs makes the metric worse — Kestrel's disagreement count rose from 1,074 to 1,093 and the team briefly reverted a correct fix.

9 — B. A weekly full refresh.

A diff-based materialization compares the new value against the current online value and writes only on a change. It therefore has no way to notice that the online store has silently lost a key — the diff sees no change because it never sees the missing row. The periodic full refresh is the same reasoning as Chapter 20's periodic full rebuild alongside incremental loads: an incremental process needs a non-incremental backstop.

10 — B. Online inference, the same feature computed twice, and more than one consumer.

With one of the three, buy nothing. With two, write the missing piece. Seven of Kestrel's nine feature requests were satisfied by a dbt incremental model with valid_from / valid_to, which is already tested by Chapter 23's register, catalogued by Chapter 30, and covered by Chapter 31's deletion manifest — all of which a separate feature store would have needed re-solved. The two that justified the real thing were both online and both computed features the batch path also computed.

11 — C. The model began over-predicting churn, which looked like drift.

The corrected feature ran about 3% lower on average, and the model — having learned that lower values mean higher churn risk — shifted systematically. Nothing errored. The training job succeeded, the store was fresh, and the nulls were unchanged. The visible symptom belonged to a category ("drift") that sends the investigation to the modeling team, which is the same misdirection as Case Study 1. The fixes are content-addressed training snapshots and a backfill job that fails if it cannot enumerate its consumers from the catalog's lineage.

12 — B. Accuracy requires labels, and labels have a delay.

A 90-day churn definition means accuracy is knowable 90 days after the prediction. Feature distributions, feature age, null rates, and the prediction distribution are all available immediately, and all four are data engineering — they need no ML knowledge and they catch skew, staleness, and upstream pipeline breaks between them. Accuracy is the thing everyone wants to monitor and is usually the last signal to arrive.