Exercises: ML Engineering and Feature Stores

Solutions and grading notes are in the instructor companion. Exercises marked 🧪 use code/pit_join.py. None of these requires you to train a model — that is the point of the chapter.

Warm-Up

Exercise 32.1 — Whose problem is it?

Difficulty: ★☆☆ · Time: 15 minutes

For each symptom, say whether it is yours, shared, or not yours — and name the first diagnostic you would run:

1. The model scores 0.83 offline and 0.59 in production, from day one.
2. The model scored 0.72 at launch and 0.61 six months later.
3. The model has never exceeded 0.55 in any configuration.
4. The model scored 0.72 for eight months and 0.58 last Tuesday.
5. The model's predictions are well-calibrated but the business impact is negligible.

Then: §32.2 gives a four-line lookup for this. Write it from memory before checking.

Exercise 32.2 — Three parts of a feature

Difficulty: ★☆☆ · Time: 15 minutes

§32.3: a feature is a value about an entity, at a moment.

For each, name the entity, the value, and the as-of — and say which is missing or ambiguous:

1. customer.lifetime_orders in a nightly-refreshed table.
2. product.category in a Type 2 dimension.
3. session.pages_viewed, computed at session end.
4. customer.churn_probability, written by a model that runs weekly.
5. (customer, product).times_purchased.

Which of the five has a version as well as a timestamp, and why?

Exercise 32.3 — Read the leak

Difficulty: ★☆☆ · Time: 20 minutes · 🧪

python code/pit_join.py --leak

1. The naive join scores 0.830 and the as-of join 0.591. Explain the mechanism in two sentences, using the fact that churners stop ordering.
2. 97.1% of rows differ between the two joins. Why is that number so high, and what would make it lower?
3. The chapter says the failure mode of leakage is "good news." What does that imply about which defects get investigated?
4. Which of Chapter 23's twenty-two assertions would catch this? Be honest.

Core

Exercise 32.4 — Write an as-of join

Difficulty: ★★☆ · Time: 45 minutes

Without looking at pit_join.py:

1. Write an as-of join in SQL for your warehouse. Handle: the inclusive boundary, entities with no feature row, and multiple feature rows on the same timestamp.
2. Write the same thing in Python, as a sort-merge.
3. State the invariant your implementation must satisfy, in one sentence.
4. Write the test that asserts it over a whole dataset, not over three examples.
5. Now read pit_join.py's version and diff it against yours. What did you miss?

Exercise 32.5 — Null is not zero

Difficulty: ★★☆ · Time: 30 minutes · 🧪

§32.5's 📐 callout: 130 of 20,000 rows have no observation.

1. For each feature, say what NULL means and what 0 means, and whether they differ: orders_to_date · days_since_last_order · refund_count_90d · avg_basket_cents · is_email_subscriber
2. Two of the five are cases where 0 is the correct fill. Which, and why?
3. In pit_join.py, change row["value"] = None to row["value"] = 0. Re-run --self-check. Which assertions fail, and which do not? What does the set of passing assertions tell you about testing this decision?
4. Design the assertion you would add to Chapter 23's register to catch a silent COALESCE(x, 0) on a feature column.

Exercise 32.6 — Measure feature age

Difficulty: ★★☆ · Time: 40 minutes · 🧪

python code/pit_join.py --asof

1. The fixture's ages are p50 11 days, p90 34, p99 55, max 61. Where does that distribution come from? Trace it back to the fixture's generation.
2. For a real feature you have access to, compute the age distribution at training time. If you have no such feature, compute it for a warehouse table you refresh on a schedule.
3. §32.6 says the fix is sometimes to make serving staler. Construct the case where that is right, and the case where it is clearly wrong.
4. Build the four-column table from §32.6's 📏 callout for three features you own.

Exercise 32.7 — The row-level diff

Difficulty: ★★☆ · Time: 45 minutes · 🧪

python code/pit_join.py --skew

1. Case Study 2's distribution comparison found means of 34.2 and 34.6 and passed. Reproduce that: compute the mean of both paths in the fixture and confirm they look equivalent.
2. Now compute the row-level disagreement rate. State in one sentence why the first check cannot find what the second finds.
3. Break the segmentation: report the disagreement rate by a segment other than recency. Does the concentration still show up?
4. Add a fourth bug to serving_days_since_last_order — your choice — and predict its disagreement rate and segment before running it.

Exercise 32.8 — Do you need a feature store?

Difficulty: ★★☆ · Time: 30 minutes

§32.9's threshold: online inference and the same feature computed twice and more than one consumer.

Apply it to five scenarios:

1. One model, batch scoring nightly, one team.
2. One model, online scoring at checkout, one team.
3. Four models, all batch, sharing eight features across three teams.
4. Two models, one batch and one online, sharing four features.
5. Twelve models, all online, one team, no shared features.

For each: buy, build the missing piece, or use a table — and name the missing piece where relevant.

Exercise 32.9 — Labels

Difficulty: ★★☆ · Time: 40 minutes

§32.11's four properties: definition, two timestamps, delay, source.

For a prediction problem you know (or Kestrel's churn model):

1. Write the label definition precisely enough that two people would compute it identically.
2. Name both timestamps. How far apart are they?
3. What is the labeling delay, and what does it do to your newest training data?
4. Is the label produced by a human, a rule, or a model? If a model, describe the feedback loop.
5. §30.8's problem applies here. Grep for your label's name. How many definitions?

Exercise 32.10 — The monitoring set

Difficulty: ★★☆ · Time: 40 minutes

§32.12 lists four things to monitor and excludes model accuracy.

1. Implement the first — feature distributions in training versus serving — for three features. Report mean, p50, p99, and null rate in both.
2. Why is accuracy excluded, and when does it become available for a 90-day churn label?
3. Design the alert thresholds for all four. Which one should page (Chapter 26), and which should not?
4. §32.12 claims the prediction distribution is a leading indicator. Construct the scenario where it moves and nothing is wrong.

Advanced

Exercise 32.11 — Extend the lab

Difficulty: ★★★ · Time: 90 minutes · 🧪

Add three to pit_join.py, with self-checks:

1. A second leaking feature with a different mechanism — one that leaks through an aggregate window rather than a current value.
2. A --split mode implementing the time-split evaluation from Case Study 1, reporting both CV and time-split AUC.
3. Feature age reported per segment, so a segment with systematically staler features is visible.
4. A materialization simulator: apply changed-only writes and count them (§32.8's 71×).
5. An l-o-o entity check: assert that an entity's feature value never depends on another entity's future — a leak class the current fixture cannot express.
6. A --skew --segment-by flag taking any grouping expression.

Then: report how many self-checks you added, and name the one that would have caught a bug you did not deliberately inject.

Exercise 32.12 — Find the leak in a real pipeline

Difficulty: ★★★ · Time: 60 minutes

Take a training query from your own work, or write one for Kestrel's churn model.

1. List every feature and its as-of semantics. Any feature you cannot answer for is a suspect.
2. For each, ask: could this value have changed between the prediction moment and now, in a way correlated with the outcome? That question is the leak detector.
3. Run the time-split check. Report both numbers.
4. If the gap is small, construct a leak and confirm your check detects it. A diagnostic you have never seen fire is not yet a diagnostic.

Exercise 32.13 — Design the shared implementation

Difficulty: ★★★ · Time: 90 minutes

Case Study 2's fix separated computation from retrieval.

1. Take a feature currently computed in SQL. Refactor it into a function that takes data and returns a value, plus two thin retrieval layers.
2. What does the SQL path cost in runtime? Measure it, or estimate and say how.
3. Where does your design break? Case Study 2 names one: a feature needing data the serving path cannot cheaply fetch. Construct yours.
4. Implement the daily skew check, reported by segment.
5. Write the case for deleting your skew check in a year, and the rebuttal.

Exercise 32.14 — The backfill that invalidated a model

Difficulty: ★★★ · Time: 60 minutes

§32.10's 🔁 callout: a correct backfill silently invalidated a model for five weeks.

1. Design the control. A backfill to a feature table must enumerate its consumers before running. Where does the consumer list come from?
2. Content-address a training set. What exactly do you hash, and what breaks if you hash the query instead of the data?
3. The corrected feature ran 3% lower and the model over-predicted churn. Which of §32.12's four monitors would have caught it, and how fast?
4. §32.10 says a materially changed definition is a new feature with a new name. Argue against this, then argue for it, then say which you would enforce.

Exercise 32.15 — Hold the boundary

Difficulty: ★★★ · Time: 45 minutes

§32.13: "the model isn't good enough" is not a data engineering ticket.

1. Write the four questions you can answer in a week — leaking, skewed, stale, mislabelled — as a concrete checklist with the command or query for each.
2. Run all four against a model you have access to. Report the four answers.
3. Draft the message you send when all four come back clean. It must be useful, not defensive.
4. Construct the case where a data engineer should take on model quality, and say what changes.

Project Milestone

Exercise 32.16 — Feature infrastructure for Kestrel

Difficulty: ★★★ · Time: 3–4 hours · 🧪

Build the ML surface from §32.12's 🧱 callout on your own platform.

1. Build one point-in-time-correct feature table as a Chapter 20 incremental model with valid_from / valid_to.
2. Write one as-of join, with the invariant test over all rows.
3. Compute feature age for that feature, and state what serving age it would need.
4. Generate labels for a prediction problem, with both timestamps and a documented delay.
5. Run the time-split evaluation against a random split. Report both. If they agree, construct a leak and confirm the check catches it.
6. Implement the skew check for one feature computed two ways — even if you have to write the second way to have something to compare.
7. Publish the three daily numbers: skew rate, feature age ratio, null rate delta.
8. Apply §32.9's threshold to your platform and write the one-paragraph answer.

Deliverable: the feature model, the as-of join and its test, both evaluation numbers, the three published metrics, and the feature-store decision with its reasoning.

Step 5 is the one that matters. A team that has never seen its leak detector fire does not have a leak detector.