Further Reading: ML Engineering and Feature Stores

Sources are tagged Tier 1 (confident it exists, recommended without reservation) or Tier 2 (real and worth seeking, but confirm the current edition, version, or URL yourself).

A note on how to read this list. The ML literature is enormous and almost all of it is about models. The material below is deliberately the thin slice that is about data, and if you find yourself reading about architectures or optimizers you have wandered out of this chapter. The single best thing you can do after this chapter is not read anything — it is run the time-split check on a model you have access to.

Start here

• Chip Huyen, Designing Machine Learning Systems (O'Reilly, 2022). The best single book for this chapter's territory. The chapters on training data, feature engineering, and data distribution shifts are the relevant ones, and Huyen is unusually clear about the boundary between data problems and modeling problems — which is §32.2 and §32.13's whole subject. Tier 1.

• "Hidden Technical Debt in Machine Learning Systems" (Sculley et al., NeurIPS 2015). Nine pages, free, and the paper that named most of what this chapter is about — entanglement, undeclared consumers, pipeline jungles, and the observation that the model is a small box in a large diagram. Read it before anything else on this list; §32.10's backfill problem is its "undeclared consumers" section, arriving eleven years later. Tier 1.

• "The ML Test Score: A Rubric for ML Production Readiness" (Breck et al., Google, 2017). A checklist of twenty-eight tests, of which roughly half are data tests. Score a model you own against it — most score badly, and the data half is the half a data engineer can fix. Tier 1.

On leakage specifically

• Kaufman, Rosset, and Perlich, "Leakage in Data Mining" (KDD 2011 / TKDD 2012). The formal treatment, with a taxonomy that distinguishes leakage-in-training-examples from leakage-in-features. The Kaggle case studies are worth the read on their own — competitions where the winning solution had found a leak rather than a model. Tier 1.

• Kapoor and Narayanan, "Leakage and the Reproducibility Crisis in ML-based Science" (2023). Surveys hundreds of papers across disciplines and finds leakage in a startling share of them. It is the strongest available argument that this is a systemic problem rather than a beginner's mistake, and it should make you check your own work. Tier 1.

• Anything on time-series cross-validation — forward-chaining, walk-forward validation, purged k-fold. The vocabulary differs by field and the idea is one idea: never evaluate on data that precedes your training data. §32.4's time split is its simplest form. Tier 2 — scattered; scikit-learn's TimeSeriesSplit documentation is a concrete entry point.

On feature stores

• The Feast documentation, particularly get_historical_features and the entity-dataframe concept. Open source, readable, and the clearest available statement of what an as-of join API looks like. Read it even if you never deploy Feast — it will tell you what your dbt model has to do. Tier 1.

• The Tecton engineering blog, and Uber's Michelangelo papers. Michelangelo is where much of this vocabulary originated, and the write-ups on offline/online consistency are §32.7 from the people who hit it at scale first. Tier 2 — blog posts move; the ideas do not.

• Databricks Feature Store and SageMaker Feature Store documentation. The managed offerings. Read their materialization and point-in-time-lookup sections and compare against §32.8's cost arithmetic — the write-pricing discussion is usually absent from the marketing and present in the pricing page. Tier 1 — versioned.

• featureform, hopsworks, and the smaller open-source stores. Worth reading the data models of. Note how much of each one is a Type 2 dimension with an API in front of it, which is §32.9's point. Tier 2.

On the warehouse side, which is where most of the work is

• Chapter 20 of this book. Not a deflection: a Type 2 slowly changing dimension is a point-in-time-correct feature table, and teams that have one have already built the hard part.

• The dbt documentation on snapshots and incremental models. §32.9's seven-of-nine answer. Snapshots in particular are a feature-history mechanism most dbt users have not connected to ML. Tier 1 — versioned; this book pins dbt-core 1.9.1.

• Your warehouse's ASOF JOIN support. Snowflake, DuckDB, ClickHouse, and Databricks all have one now, with different names and semantics. Read the boundary semantics carefully — §32.5's inclusive-versus-exclusive question is answered differently in different engines, and the default is not always what you want. Tier 1 — versioned, and genuinely divergent between engines.

On monitoring

• "Monitoring Machine Learning Models in Production" material from the ML-monitoring vendors — Evidently AI's open-source library and documentation is the most useful, because it is a library you can read. Its data-drift metrics are §32.12's first monitor, implemented. Tier 1 for Evidently.

• Google's "Data Validation for Machine Learning" (Breck et al., SysML 2019) and TensorFlow Data Validation. The schema-inference-and-anomaly-detection approach is a genuinely different design from Chapter 23's explicit register, and the comparison is instructive: inferred schemas find things you did not think of and produce more false positives, which is Chapter 31 §31.2's trade in another domain. Tier 1.

• Chapter 25 of this book. Feature freshness is freshness; materialization lag is pipeline lag. The monitoring in §32.12 is Chapter 25 applied to a different consumer, and if you have built Chapter 25's you have most of it.

On labels

• Anything careful on weak supervision and programmatic labeling — the Snorkel line of work is the best-documented. Relevant here not for the labeling technique but because it makes explicit that labels are engineered artifacts with error rates, which §32.11 argues and most teams do not act on. Tier 2.

• Northcutt, Athalye, and Mueller, "Pervasive Label Errors in Test Sets" (2021). Found substantial label error rates in the standard benchmark datasets — the ones the entire field measures progress against. A useful corrective if you assume your labels are the reliable part. Tier 1.

Practice

• code/pit_join.py in this chapter. Leakage measured, an as-of join with the invariant test, feature age, and a row-level skew diff with three injected bugs. Forty-two self-checks. Exercise 32.5's break-it instruction is the most instructive five minutes in the file.

• Run the time-split check on a model you have access to. Exercise 32.12. If the gap is small, construct a leak and confirm your check detects it — a diagnostic you have never seen fire is not yet a diagnostic.

• Run the row-level diff on any feature computed in two places. Exercise 32.7. An afternoon, and the disagreement rate is almost never zero the first time.

• Compute feature age. Exercise 32.6. Nobody has, and the ratio is usually surprising.

A note on what to be skeptical of

Any feature store evaluation that does not start with §32.9's threshold. The products are good and most teams evaluating them do not have all three conditions. Kestrel scoped one for nine use cases and needed it for two.

Any claim that a feature store "solves" training/serving skew. It solves it for features defined inside it. A feature computed in the application and passed at request time is outside, and that is where skew survives.

Benchmarks of online-store latency. Chapter 11 §11.6's honesty checklist applies: the number is usually measured on a single-key lookup with a warm cache, and your serving path fetches twelve features for one entity and is bounded by the slowest of them.

And any offline metric quoted without saying how the split was made. After this chapter, "AUC 0.83" should prompt exactly one question: "split how?"