Chapter 32: ML Engineering and Feature Stores

"The model scored 0.83 in the notebook and 0.59 in production, and the data science team spent six weeks investigating the model. The problem was a JOIN."

Overview

This chapter is about the data problems that present as modeling problems, and there is a specific reason they are worth a chapter of their own: they are diagnosed by the wrong people.

A model performs worse in production than in training. The obvious hypothesis is that something is wrong with the model — the wrong architecture, overfitting, insufficient regularization — and the people with the skills to investigate that hypothesis are data scientists, who investigate it, sometimes for weeks.

The actual cause is frequently a join, a null, a timestamp, or two implementations of one feature. All four are data engineering, all four are invisible from inside a notebook, and all four are measurable in an afternoon by someone who knows to look.

What this chapter is not. It is not machine learning. There is no model architecture here, no hyperparameter tuning, and no loss function. The line this book draws — stated in Chapter 1 and held since — is that data engineers build the infrastructure ML depends on. This chapter is that infrastructure, and the argument is that a substantial fraction of what looks like ML failure is infrastructure failure.

Chapter 5 promised a threshold for when a warehouse table stops being a feature store and you need a real one. §32.9 delivers it, and the answer is more restrictive than the industry's.

32.1 What an ML Team Actually Needs From You

Ask a data scientist what they need and you will get a list of datasets. Ask what goes wrong and you get a different list entirely, and the second list is the one to build against.

Five things, in the order they cause pain:

1. The same feature, computed once. Not "a table with the columns in it" — one definition, used by training and by serving, because two implementations diverge and the divergence is §32.7.

2. Correct history. Not the current value; the value as of a moment in the past (§32.4). This is the single highest-value thing on the list and the one most warehouses cannot do.

3. Freshness they can rely on, or at least measure. A feature that is 11 days old in training and 2 hours old in serving is not the same feature (§32.6).

4. Labels with the same care as features. Labels have timestamps, delays, and definitions, and they are typically produced by whoever needed them fastest (§32.11).

5. A way to get a feature into production without you. The bottleneck complaint is almost always this one, and it is an interface problem rather than a data problem.

🔎 Read the Plan — the question that changes what you build

A data science team asks for "a table with customer features." The request is a proposed solution and the need is somewhere behind it, which is Chapter 29 Case Study 2's move in a new domain.

Four questions, ten minutes, before writing anything:

"At what moment does the model need this — and is that moment in the past or right now?" A batch model scoring nightly and a model scoring on a live request need genuinely different systems (§32.8). This one question determines most of the architecture, and it is skipped constantly.

"When you train, what timestamp are the features supposed to be as of?" If the answer is a blank look, you have found a leak before it is built (§32.4). If the answer is "the prediction time, obviously," ask how the current pipeline achieves that.

"How old is the feature allowed to be?" Not "how fresh do you want it" — everyone says real time. What breaks at one hour, at one day, at one week? Usually nothing breaks until surprisingly late, and the answer sets your entire cost profile.

"Who else uses this feature?" If the answer is nobody, a warehouse table is fine (§32.9). If it is three teams, you have a definition problem before you have an infrastructure problem — Chapter 30 Case Study 1, wearing a different hat.

Kestrel's team ran these against nine requests over a year and built a feature-store integration for two of them. The other seven were satisfied by a dbt model, which is the outcome §32.9 argues should be common and usually is not.

32.2 The Three Data Problems in ML, and Which Are Yours

Not everything that goes wrong with a model is yours to fix, and being clear about the boundary makes you more useful rather than less.

The three in the top rows are yours entirely, they account for a large share of production ML failures, and all three are invisible from a notebook — which is why they survive so long.

The distinguishing symptom is worth memorizing:

good in training, bad in production, IMMEDIATELY   -> leakage or skew     (you)
good in production, degrading over weeks           -> staleness           (you)
good in production, degrading over months          -> drift               (shared)
never good anywhere                                -> modeling or signal  (not you)

"Never good anywhere" is not your problem and you should say so plainly, because a data engineer who takes on responsibility for model quality in general has taken on a problem they cannot solve and will be measured against it.

32.3 What a Feature Is

A feature is a value about an entity, at a moment. All three parts matter and the third is the one that gets dropped.

entity        customer 90210
feature       orders_to_date
value         6
AS OF         day 137           <-- the part that gets dropped

An entity is the thing being predicted about — a customer, an order, a product, a session, or a pair (this customer, this product). The entity key is a join key, and Chapter 6's discipline about keys applies unchanged.

A feature without an as-of is not a feature; it is a fact about now. And a fact about now is exactly what you must not train on, which is the whole of §32.4.

Three kinds of feature, in increasing order of how much trouble they cause:

Attributes. customer.country, product.category. Slowly changing, and Chapter 20's SCD Type 2 is already the right answer — a Type 2 dimension is a feature table with point-in-time correctness built in, and teams that have one are usually not aware they have solved the hard part.

Aggregates. orders_to_date, revenue_30d, days_since_last_order. The bulk of real features and the source of most trouble, because a window has to be computed relative to something.

Derived / model outputs. An embedding, a propensity score, a cluster assignment. These have a version as well as a timestamp, because the model that produced them changes (§32.10).

32.4 Point-in-Time Correctness, and the Leak You Can Measure

Here is the whole problem in one query. A data scientist wants to train a churn model. They have labels — did this customer churn? — and a feature table.

-- The natural join. It is wrong, and nothing about it looks wrong.
SELECT l.customer_id, l.churned, f.orders_to_date
  FROM labels l
  JOIN customer_features f USING (customer_id);

customer_features holds the current value. The label describes something that happened months ago. So the model is trained on a feature computed after the outcome it is predicting.

This is measurable, and code/pit_join.py measures it. 20,000 customers, one feature, one label, two joins:

customers                      20,000
churned within 90 days         2,867  (14.3%)

naive join (current value):
    mean orders_to_date        14.92
    AUC                        0.830   <-- looks excellent

as-of join (value at event):
    mean orders_to_date         6.28
    AUC                        0.591   <-- the real signal

rows where the two differ      19,421 of 20,000  (97.1%)

AUC 0.830 is a shippable model. AUC 0.591 is barely better than a coin. The difference is not modeling; it is the label, read back through a feature computed after the outcome had already happened.

The mechanism is worth stating precisely, because it generalizes to every leak of this shape: customers who churn stop ordering. So a churner's current orders_to_date is close to their value at the prediction point, and a non-churner's has grown a great deal since. The difference between then and now is the answer, and the naive join hands it to the model.

⚠️ Failure Mode — the leak is invisible at every point where somebody looks

This is why leakage survives review, and it is worth walking through where each check fails.

• The query looks correct. It is a two-table join on a primary key. There is no window, no timestamp, no subquery — nothing to be suspicious of.
• The data looks correct. Every value in orders_to_date is a true fact about that customer. Chapter 23's assertions all pass: no nulls, no negatives, referential integrity intact. Leakage is not a data quality defect and no data quality test detects it.
• The model looks correct. Cross-validation is clean, because the leak is present in every fold. Splitting by customer does not help; splitting by time does, which is why time-based splits are recommended and why the recommendation is usually followed too late.
• The metric looks excellent, which is the actual problem: the failure mode of leakage is good news, and good news is not investigated.

It surfaces in production, weeks later, framed as "the model degraded." It did not degrade. It was never that good, and the deployed model — which correctly receives the as-of feature, because at serving time the future has not happened yet — is delivering exactly the 0.591 it was always worth.

The one check that finds it before deployment: "train on data up to day N, evaluate on day N+1 onward." If the evaluation score collapses relative to cross-validation, something in the feature set knows about the future. It is one line of code and it is the difference between six weeks of model investigation and an afternoon.

🎓 Interview Angle — "your model does great offline and badly in production. Walk me through it."

An extremely common question, and most answers go straight to the model — overfitting, distribution shift, regularization. Those are real and they are the third thing to check.

The answer that works starts by separating two cases, which shows you have debugged this rather than read about it:

"First I'd ask whether it was bad immediately on deployment or degraded over time. Those are different problems. If it was bad immediately, I'd suspect leakage or training/serving skew before I'd suspect the model — either a feature was computed with information that didn't exist at prediction time, or the serving path computes it differently from the training path. Both are data problems and both are cheap to check.

For leakage: retrain with a strict time split and see whether the score survives. For skew: take the same entity at the same timestamp through both code paths and diff the feature values. If the disagreement rate isn't zero, that's the bug.

If it degraded gradually, then I'd be looking at drift or feature staleness, and that's a shared investigation with the modeling team."

Then, if you want to be memorable: "the fastest single check is to compare the mean of each feature in training against its mean in serving. It's an afternoon, it needs no ML knowledge, and it finds skew immediately." §32.7.

32.5 Writing an As-Of Join

The fix is a join on entity AND time, taking the latest feature row at or before the event. Most warehouses can express it; few people write it correctly the first time.

-- The as-of join, portable form.
SELECT l.customer_id, l.event_ts, l.churned, f.orders_to_date
  FROM labels l
  LEFT JOIN LATERAL (
      SELECT orders_to_date
        FROM customer_features
       WHERE customer_id = l.customer_id
         AND valid_from <= l.event_ts        -- <= not <
       ORDER BY valid_from DESC
       LIMIT 1
  ) f ON true;

code/pit_join.py implements the same thing as a sort-merge, and writing one once is worth more than reading about it:

def asof_join(events, features, entity="customer_id", ts="day"):
    by_entity = {}
    for f in features:
        by_entity.setdefault(f[entity], []).append((f[ts], f["value"]))
    for k in by_entity:
        by_entity[k].sort()

    out = []
    for e in sorted(events, key=lambda x: (x[entity], x[ts])):
        rows = by_entity.get(e[entity], [])
        days = [r[0] for r in rows]
        i = bisect.bisect_right(days, e[ts])
        row = dict(e)
        if i == 0:
            row["value"] = None          # NOT zero
            ...

Four details, and only the first is about the join.

The boundary is inclusive. valid_from <= event_ts. A feature computed at exactly the prediction moment was available at the prediction moment. Getting this wrong costs one row's worth of information per entity and is the single most common off-by-one in this territory — and §32.7 shows what happens when the two code paths disagree about it.

The null is not a zero, and this matters more than the join does.

📐 Design Decision — NULL means no observation; 0 means zero

In the fixture, 130 of 20,000 events (0.65%) occur before the entity has any feature row at all — a customer scored before their first order.

The COALESCE(orders_to_date, 0) that everyone writes is wrong, and it is wrong in a specific, damaging way:

• 0 says "this customer has placed zero orders." That describes a dormant customer.
• NULL says "we have no observation for this customer." That describes a new one.
• A new customer and a dormant customer behave nothing alike, and the model has just been told they are identical.

The damage is concentrated where it hurts most: 0.65% of rows sounds ignorable, and those rows are disproportionately new customers, who are disproportionately the population a churn or propensity model is deployed to act on.

What to do instead, in order of preference:

1. Pass the null through and let the model handle it. Gradient-boosted trees handle missingness natively and learn a split for it; this is the right answer more often than people expect.
2. Add an explicit indicator — orders_to_date_is_null — and impute whatever you like. The information is preserved either way.
3. Impute, and record that you imputed, in the feature's documentation, so the next person knows.

What not to do is silently coalesce, which destroys information and leaves no trace. The general principle appeared in Chapter 18 with LEFT JOIN and again in Chapter 23's null assertions: a null that gets filled in without being recorded is a fact you have deleted.

The feature must be immutable once written. If customer_features is overwritten each night rather than appended to, there is no history to join to and the as-of join silently degrades into the naive one. This is Chapter 20's Type 2 requirement, and it is the reason a nightly full-refresh feature table cannot support training.

And the join must be tested against the property, not against examples. pit_join.py's self-check asserts, over all 20,000 rows, that no returned value has a timestamp after its event — which is the invariant, and which catches a class of bug that three hand-picked test cases will not.

🧪 Try It — see the leak, then close it

bash cd part-06-advanced-topics/chapter-32-ml-engineering-and-feature-stores/code python pit_join.py --leak python pit_join.py --asof

Then break it deliberately. In asof_join, change bisect_right to bisect_left. Re-run --self-check.

One assertion fails, and it is the boundary case: an event on the exact day a feature was computed now misses that feature. Predict which of the 42 checks fails before you run it.

Then change row["value"] = None to row["value"] = 0 and re-run. Notice that the self-check still mostly passes, and think about why the null-versus-zero decision needs a different kind of test than the join does.

32.6 Feature Age: The Number Nobody Computes

Every training row has a feature age — the gap between when the feature value was computed and the moment it is being used to predict. Almost nobody measures the distribution, and it explains a category of production failure.

From the fixture:

FEATURE AGE at the prediction point
    p50   11 days
    p90   34 days
    p99   55 days
    max   61 days

The model was trained on features that are, at the median, eleven days old. In production, the serving path fetches the feature from an online store that is updated hourly, so the model receives features that are two hours old.

These are not the same feature. orders_to_date at 11 days of age and at 2 hours of age have different distributions, and the model learned the relationship on the first one.

📏 Scale Note — the direction of the mismatch determines whether you notice

Fresher-in-serving-than-in-training is the common case and the quiet one. The model sees more recent values than it was trained on, its inputs shift slightly, and performance degrades by an amount that looks like ordinary noise. Nobody investigates, because the model still works.

Staler-in-serving-than-in-training is rarer and much louder. It usually means a materialization job is behind (§32.8), and the model's inputs go badly out of distribution at once. This one gets noticed within a day, which is fortunate, because it is also the one that does the most damage.

The measurement that covers both is one number per feature, in both environments:

text feature train p50 age serve p50 age ratio orders_to_date 11 days 2 hours 132x days_since_order 11 days 2 hours 132x country 41 days 6 hours 164x engaged_30d 1 day 1 day 1.0x OK

Kestrel's churn model had a 132× age mismatch on its two strongest features and nobody had computed the number, because computing it requires joining a feature to its own timestamp — which is exactly the operation the naive pipeline had already thrown away.

The fix is not always to make training fresher. Sometimes it is to make serving staler deliberately — to serve the feature at the same age it was trained on, which sounds perverse and is correct. The requirement is that training and serving agree, not that both are fast.

32.7 Training/Serving Skew

The same feature, computed twice, by two code paths, that disagree. It is the most common production ML data failure and the one with the clearest diagnostic.

Why it happens is structural rather than careless. The training path is SQL over the warehouse, batch, written by a data engineer. The serving path is Python or Java over an operational database or an API, low-latency, written by a backend engineer, often months later. Two people, two languages, two data sources, one feature name.

pit_join.py --skew runs the same 20,000 entities through both paths:

entities compared              20,000
disagreements                  1,074  (5.37%)
    of which null-vs-zero        130
    of which boundary            944

customer   as_of    batch    serving  kind
3          96       0        12       boundary
38         67       0        8        boundary
40         231      0        27       boundary

Three bugs are injected, and all three are real ones seen in production:

The boundary. The serving path uses < as_of where the training path uses <= as_of, so an order placed today is invisible to serving. 944 rows.

The null. Serving returns 0 where training returns NULL — §32.5's decision, made differently in two places. 130 rows.

The source. Serving reads the OLTP database, which excludes cancelled orders the warehouse retains. Not modeled in the fixture, and the one that took longest to find at Kestrel, because it produces a disagreement only for customers who have cancelled something.

⚠️ Failure Mode — 5.37% is not a small number, because skew is never uniform

The instinct on seeing a 5% disagreement rate is that it is survivable. It is not, and the reason is the distribution rather than the magnitude.

The disagreements are entirely concentrated. Of the 963 customers who ordered on the prediction day, 944 disagree — 98%. For customers who have not ordered recently, the two paths agree perfectly.

Which means the skew is concentrated on exactly the customers the model is most confident about, and most likely to act on: recent purchasers, the high-engagement segment, the population a retention campaign targets. The model's best predictions are its most corrupted ones.

A uniformly-distributed 5% error would be survivable. This is not that, and no aggregate metric reveals the difference — the mean of days_since_last_order shifts by a fraction of a day, and both distributions look fine.

The check that finds it is not a distribution comparison. It is a row-level diff: same entity, same timestamp, both code paths, count the disagreements. It is an afternoon of work and it produces a number you can put on a dashboard.

🏭 From the Pipeline — the nineteen rows where two bugs cancelled

Of the 963 same-day orderers, 19 agreed. They are the customers whose first ever order was on the prediction day.

Both paths return 0, for opposite reasons. The training path sees today's order and computes zero days since. The serving path finds no prior order at all and returns 0 through the null-handling bug. Two defects, opposite signs, identical output.

This is why spot-checking does not find skew. An engineer verifying a handful of rows by hand gravitates toward simple cases — a new customer, a customer with one order — and the simple cases are disproportionately the ones where the bugs cancel. The 19 rows that agree are the 19 an engineer would have picked.

Two things follow, and both are general:

• Compare populations, not examples. The disagreement rate over 20,000 rows found it instantly.
• When two independent bugs produce the same output, one of them will be "fixed" and the other exposed. Kestrel fixed the null handling first, and the same-day-order bug's blast radius grew, which is a genuinely confusing thing to debug and is worth expecting.

The structural fix is not better testing. It is one implementation. This is the strongest argument for a feature store, and it is §32.9's threshold: when the same feature must be computed in two places, you now have a problem that discipline does not solve — because discipline has to hold across two codebases, two teams, and every future change to either.

32.8 Online and Offline Stores

A feature store is two stores with one definition, and understanding why there are two is most of understanding the architecture.

                         one feature definition
                                   |
              +--------------------+--------------------+
              |                                         |
        OFFLINE STORE                              ONLINE STORE
   warehouse / lake, columnar                  key-value, low latency
   ALL history, immutable                      LATEST value only
   read: millions of rows                      read: one entity, <10ms
   used for: TRAINING (as-of joins)            used for: SERVING
              |                                         ^
              +---------- materialization --------------+
                          (the job that keeps them
                           consistent, and the thing
                           that breaks)

The offline store answers "what was this value on day 137, for two million customers?" Columnar, cheap per row, slow per lookup. This is a warehouse table, and Chapter 20's Type 2 pattern is exactly right for it.

The online store answers "what is this value now, for customer 90210, in under ten milliseconds?" Key-value — Redis, DynamoDB, or equivalent — holding only the current value.

Materialization is the job that copies the latest offline values into the online store, and it is the component that fails. Chapter 25's freshness monitoring applies to it directly: a materialization job that is four hours behind is serving four-hour-old features to a model expecting one-hour-old ones, and nothing about the model's output looks wrong.

💸 Cost Check — the online store is priced by a dimension nobody estimates

The offline store is warehouse storage and it is cheap. Kestrel's full feature history — 298,452 feature rows per feature set, three years — is a rounding error against the 341 GB/year of Parquet clickstream.

The online store is priced by writes, and the write rate is the materialization rate multiplied by the entity count, which is the number nobody computes before choosing a refresh interval:

text 1,904,221 customers x 12 features refreshed hourly = 22,850,652 writes/hour = 548,415,648/day refreshed daily = 22,850,652/day 24x less refreshed hourly, CHANGED ONLY = ~1.4% of entities change per hour = 319,909 writes/hour = 7,677,816/day 71x less

Writing only what changed is a 71× reduction and it is the single largest cost lever in a feature store. It requires the materialization job to diff against the current online value, which costs a read per entity — and reads are an order of magnitude cheaper than writes on every managed key-value store Kestrel priced.

The trap it introduces: a diff-based materialization has no natural way to detect that the online store has silently lost a key. Kestrel runs a full refresh weekly for exactly this reason, which is the same reasoning as Chapter 20's periodic full rebuild alongside incremental loads.

32.9 The Threshold: Do You Actually Need a Feature Store?

Chapter 5 promised this section would be explicit. Here it is.

You need a feature store when you have all three of:

1. Online inference. A model scoring on a live request, needing features in milliseconds. If every model scores in a batch job, a warehouse table is a feature store and you should use one.
2. The same feature computed in two places. §32.7's skew, which discipline cannot solve because it has to hold across two codebases forever.
3. More than one team or model sharing a feature. Otherwise the "reuse" benefit is theoretical.

With one of the three, buy nothing. With two, write the missing piece yourself. With all three, a feature store is doing something you would otherwise build badly.

📐 Design Decision — what Kestrel built instead, for seven of nine requests

Nine feature requests in a year; two got a feature store integration and seven got a dbt model.

The seven had this shape: a batch model, scoring nightly, features used by one team. The requirement is a table with correct history, which is a Chapter 20 incremental model with valid_from / valid_to and an as-of join at training time.

sql -- models/features/customer_order_features.sql {{ config(materialized='incremental', unique_key=['customer_id','valid_from']) }} SELECT customer_id, order_ts AS valid_from, LEAD(order_ts) OVER (PARTITION BY customer_id ORDER BY order_ts) AS valid_to, COUNT(*) OVER (PARTITION BY customer_id ORDER BY order_ts) AS orders_to_date FROM {{ ref('fct_orders') }}

That is a feature store, for these seven cases, and it costs nothing beyond a model already in the project, is tested by Chapter 23's register, is governed by Chapter 30's catalog, and is deleted from by Chapter 31's manifest — all of which a separate feature store would have needed re-solved.

The two that justified the real thing were both online: a fraud check on checkout and a live product ranking. Both needed a sub-10ms lookup and both computed features that the batch path also computed, which is criteria 1 and 2 simultaneously.

The honest note the team recorded: they had originally scoped a feature store for all nine, and the threshold above was written after the first two integrations turned out to cost six weeks each. Chapter 5's general point — do not buy the platform before you have the problem — with a specific number attached.

32.10 Backfills, Versions, and the Feature That Changed Meaning

A feature definition changes. Someone improves days_since_last_order to exclude cancelled orders. Three things now need deciding, and skipping any of them produces a silent failure:

Does history get recomputed? If yes, every model trained on the old definition is now inconsistent with the store it reads from. If no, the feature's meaning changes at a point in time and the model sees a discontinuity it will interpret as signal.

Is it the same feature? §30.8's question exactly. If the definition changed materially, it is a new feature with a new name — days_since_last_order_v2 — and the old one is deprecated on a schedule. This is unpopular and it is right.

What does the serving path do during the transition? Both definitions must exist simultaneously, and the model must be pinned to the one it was trained on.

🔁 Idempotency Check — a backfill that changed yesterday's answer

Kestrel backfilled orders_to_date after fixing a duplicate-order bug (Chapter 14's incident). The backfill was correct, it was idempotent, and it silently invalidated a model.

The model had been trained on the pre-backfill values. After the backfill, the offline store held different values for the same (customer_id, valid_from) pairs — better values — and the model's training set could no longer be reproduced.

Two things broke and only one was noticed.

The noticed one: an audit could not reproduce the training set, which mattered because the model made customer-facing decisions.

The unnoticed one, for five weeks: the serving path now returned corrected values while the model had learned the relationship on uncorrected ones. The corrected feature was on average 3% lower, and the model, having learned that lower values mean higher churn risk, began systematically over-predicting churn — which looked like drift.

The fix has two parts and the second is the one people skip:

• A training set is a snapshot with an identifier, stored, not a query re-run on demand. Kestrel's are content-addressed and referenced by the model registry.
• A backfill to a feature table triggers a review of every model that uses it. The dependency is in the catalog (§30.3's lineage), and the backfill job now fails if it cannot enumerate the consumers.

The general principle, third appearance in this book: correcting data is a change, and a change to a shared input needs the same care as a schema change. Chapter 17's contracts, in a new place.

32.11 Labels Are Data Too

Features get all the attention and labels cause at least as much trouble, because they are usually produced by whoever needed them fastest and rarely engineered at all.

Four properties of a label that need the same rigor as a feature:

A definition. "Churned" means what, exactly? No order in 90 days? Cancelled a subscription? Requested deletion? Chapter 30 Case Study 1's problem, and it arrives here with a model built on top of it.

A timestamp — two of them. The moment the labelled event occurred, and the moment you learned it. These differ, and the gap is the labeling delay.

A delay that bounds your training data. If churn is defined as 90 days of inactivity, you cannot label anything from the last 90 days, so your most recent training data is a quarter old. This is a hard constraint that surprises people and changes what is possible.

A source that may itself be a model. If your labels come from a rules engine or an earlier model, you are training on its outputs and will inherit its blind spots, and the feedback loop tightens every retrain.

🔐 Privacy & Governance — the training set is a copy of personal data

Chapter 31's manifest listed "ml training snapshots" as an open gap, and this section is why it is a hard one.

A training set is an immutable snapshot of personal data — that is what makes it reproducible, which is what §32.10 requires. It is also, for the same reason, undeletable by design.

The tension is real and there is no clean resolution, only three partial ones:

• Keep the snapshot's identifiers out of it. Train on features and a surrogate key, with the mapping held separately and deletable. The snapshot survives an erasure; the ability to link it to a person does not. This is the best available answer and it is not perfect.
• Bound the retention. A training snapshot older than the model that used it has no purpose. Kestrel's expire when the model is retired plus one audit cycle.
• Retrain rather than delete. Removing a person from a model is generally impractical; removing them from the next training run is trivial. The honest position is about the training set, not about the model weights, and stating that clearly is better than implying more.

What is not defensible is the common state: an unbounded pile of snapshots in a bucket, keyed by customer_id, that nobody has enumerated. That is Chapter 31 Case Study 1's finding, and ML platforms are where it accumulates fastest.

32.12 Monitoring a Model Is Monitoring Data

When a model degrades, the useful signals are almost all upstream of the model.

Four things to monitor, in the order they pay off:

1. Feature distributions, training versus serving. Mean, p50, p99, and null rate for every feature, in both environments. The single highest-value monitor in ML infrastructure, it needs no ML knowledge, and it catches skew, staleness, and upstream pipeline breaks simultaneously.

2. Feature freshness. §32.6. Age at serving, per feature, alerted when it exceeds what training saw.

3. Null rates. A feature that goes from 0.6% null to 40% null because an upstream join broke will not produce an error. The model will happily score every row.

4. The prediction distribution. Cheap, and a leading indicator: if the share of customers predicted to churn doubles overnight, something upstream changed and you will find it faster than the modeling team will.

Note what is not on this list: model accuracy. It is the thing everyone wants to monitor and it is usually the last signal to arrive, because it requires labels, and labels have a delay (§32.11). A churn model's accuracy is knowable 90 days later. The four signals above are available immediately, and they are all data engineering.

🧱 Kestrel Platform — the ML surface

text models/features/ # dbt: the offline store (7 of 9 cases) customer_order_features.sql # incremental, valid_from/valid_to customer_engagement_features.sql platform/ml/ asof.py # the as-of join, one implementation materialize.py # offline -> online, changed-only + weekly full skew_check.py # row-level diff, both paths, daily feature_age.py # section 32.6, both environments snapshots/ # content-addressed training sets

Three numbers the platform publishes daily, and the team's view is that these three replaced most of what a monitoring product would have been bought for:

text skew rate (disagreeing rows / rows compared) target 0.00% feature age ratio (serve p50 / train p50) target 1.0x null rate delta (serve - train), per feature target 0.0pp

The skew rate has been zero since the single-implementation migration — and it is still computed daily, because a metric that is always zero is the only kind whose non-zero value means something immediately. That argument is Chapter 23's, and it applies here with unusual force: the skew check is cheap, boring, and the one thing that would catch a re-divergence within a day.

🧭 Version Note — the feature store went from a product category to a pattern

Around 2019 a set of products appeared with "feature store" in the name, and the shape of the advice has changed twice since.

text era the claim what happened ───────────────────────────────────────────────────────────────────────── ~2019 you need a feature store most teams did not, and the ones that bought one used it as a table with extra steps ~2021 online + offline stores, with the dual-store sync is the hard automatic sync part, and it is where the bugs are ~2023 "just use your warehouse" true for offline; false for low-latency online serving now the PATTERN matters, not the point-in-time correctness, one product shared definition, and skew monitoring -- all of which you can build

The durable content is the middle column of the last row, and it is what this chapter teaches: point-in-time correctness, one as-of implementation used by both paths, and skew monitored by segment. None of those requires a product and all three are frequently absent from deployments that have one.

Two things that genuinely did change.

A warehouse's SCD Type 2 dimension is a point-in-time-correct feature table (Chapter 20 §20.10), and once that is noticed, a large fraction of the offline store's job is already done by something you built for a different reason.

And online serving latency is a real requirement that a warehouse does not meet. Sub-10 ms lookups at request time need a key-value store (Chapter 12), and that is the one part of a feature store that is genuinely infrastructure rather than discipline.

The evaluation question, from §32.8: multiple models sharing features, an online path with a latency budget, and enough churn that a shared definition saves real work. Kestrel meets one of three, which is why this chapter builds the pattern and not the product.

🔁 Idempotency Check — a feature backfill can invalidate a model

Recomputing a feature is not a neutral operation, because a model was trained on the values the feature had, not on the values it should have had.

text a feature backfill, and what it breaks ───────────────────────────────────────────────────────────────────── the OFFLINE store is corrected -> the next training run sees different data. Fine, and it is a different model. the ONLINE store is corrected -> the DEPLOYED model now receives features from a distribution it was not trained on. This is training-serving skew, created by a fix. both are corrected -> the deployed model is now stale relative to both, and nothing says so.

The middle row is the one nobody plans for. Correcting a bug in a feature definition is obviously right, and applying it to the online path without retraining is how a model silently degrades — the metrics do not move immediately, because the drift is in the input rather than in the output.

Three controls:

Version the feature definition, and store the version with the value. A model records which feature versions it was trained against, and a serving path that supplies a different version is a failed precondition rather than a silent substitution.

Make the backfill idempotent per entity and per timestamp — MERGE on (entity_id, feature_timestamp) — so a partial backfill can be resumed and a re-run is a no-op.

And treat "the feature changed" as a model event. The retraining trigger should include feature version changes, not only a schedule. Chapter 32's skew check is what catches it if you do not, and catching it is much later than preventing it.

The general shape, and it is Chapter 27 §27.7's deploy shapes applied to features: a feature correction is a definitional change, and definitional changes require restating history or marking the boundary. A feature store makes it easy to do neither.

🔐 Privacy & Governance — a feature is a derived personal attribute

days_since_last_order, avg_basket_size_90d, predicted_churn_score. None of them is a name, an email, or an address. All three are personal data, and they are the kind that no scanner finds.

text obligation does it reach the feature store? ──────────────────────────────────────────────────────────────────── erasure it must, and usually it does not access control the store is often read by every model and every engineer building one retention features are usually kept "as long as the model needs", which is undefined subject access request a person is entitled to the features held about them, including a score automated decision-making a score used to decide something about a person carries its own obligations in several jurisdictions

The bottom row is the one with the sharpest edge, and it is not an engineering matter: a feature that feeds an automated decision about a person is regulated differently from one that feeds a dashboard, in a way that depends on what the decision is. That is a question for counsel, and the engineering job is to know which features feed which decisions — which requires the lineage from feature to model to decision that almost nobody has.

Three things to do, and the first is free:

Tag features with a classification, in the same registry as the definition. A feature computed from customer_id is personal by construction, and that inference can be generated rather than asserted (Chapter 31's manifest).

Give every feature a retention, tied to the model that uses it rather than to the store. A feature whose only consumer was retired is a personal-data copy with no purpose, which is the clearest possible failure of data minimisation.

And make the feature-to-model-to-decision path a queryable artifact. It is three joins if the registry records model versions and their feature versions (see the 🔁 above), and it is the only way to answer "what does this score influence" without asking around.

32.13 What a Data Engineer Should and Should Not Own

A short section, because the boundary saves more trouble than any technique here.

Own: the feature pipelines, point-in-time correctness, the as-of join, the offline store, the materialization job, freshness and skew monitoring, label plumbing, the training-set snapshots, and the interface that lets a data scientist ship a feature without you.

Share: feature definitions (§30.8's problem), label definitions, and drift investigation.

Do not own: model selection, hyperparameters, model performance in general, or the question of whether the data contains enough signal to solve the problem.

The last one is worth being explicit about. "The model isn't good enough" is not a data engineering ticket, and accepting it as one is a reliable way to be accountable for something you cannot change. The useful response is to answer the questions you can: is it leaking, is it skewed, is it stale, are the labels right? All four are answerable in a week. If all four come back clean, the problem is not yours, and you have said so with evidence rather than with a boundary.

32.14 Summary

A large share of what presents as ML model failure is data engineering failure, and the three biggest are invisible from a notebook.

🔎 Four questions before building anything: when does the model need this, what timestamp should features be as-of, how old may they be, and who else uses it. Kestrel ran these against nine requests and built a feature store integration for two.

⚠️ Leakage is measurable. The same feature, two joins: AUC 0.830 with the naive join, 0.591 with the as-of join, on 97.1% of rows. The gap is not a better model — it is the label read back through a feature computed after the outcome. And it is invisible at every point where somebody looks: the query, the data, the cross-validation, and the metric all look correct, because the failure mode of leakage is good news. The one check: train to day N, evaluate on N+1 onward.

A feature is a value about an entity at a moment — and the third part is the one that gets dropped. Chapter 20's SCD Type 2 is already a point-in-time-correct feature table, which teams that have one usually do not realize.

📐 NULL is not 0. 130 of 20,000 rows had no observation, and COALESCE(x, 0) tells the model a new customer is a dormant one. Pass the null through, or add an indicator, or impute and record it — but never silently.

Test the as-of join against its invariant, not against examples: no returned value may have a timestamp after its event, asserted over all rows.

📏 Feature age is the number nobody computes. Trained at p50 11 days, served at 2 hours — a 132× mismatch on Kestrel's two strongest churn features. The requirement is that training and serving agree, not that both are fast — sometimes the fix is to serve staler features deliberately.

⚠️ Skew is never uniform, so 5.37% is not a small number. Of the 963 customers who ordered on the prediction day, 944 disagreed (98%) — the skew is concentrated on exactly the customers the model is most confident about. No aggregate metric shows this; a row-level diff shows it in an afternoon.

🏭 Nineteen rows agreed because two bugs cancelled — and they are precisely the simple cases an engineer spot-checking by hand would pick. Compare populations, not examples, and expect a blast radius to grow when you fix the first of two cancelling defects.

💸 The online store is priced by writes. Materializing only what changed is a 71× reduction — and it needs a weekly full refresh, because a diff-based job cannot notice a key the store has silently lost.

📐 The feature-store threshold is all three of: online inference, the same feature computed twice, and more than one consumer. With one, buy nothing. With two, write the missing piece. Seven of Kestrel's nine requests were satisfied by a dbt incremental model — which is already tested, catalogued, and covered by the deletion manifest.

🔁 A backfill to a feature table invalidates the models trained on it. Kestrel's corrected values ran 3% lower for five weeks and the model over-predicted churn — which looked like drift. Store training sets as content-addressed snapshots, and fail a backfill that cannot enumerate its consumers.

Labels need a definition, two timestamps, and a delay you plan around — a 90-day churn definition means your newest training data is a quarter old.

🔐 A training set is an immutable snapshot of personal data, and that is what makes it undeletable by design. Keep the identifiers out of it, bound the retention, and be honest that the position is about the training set rather than the model weights.

Monitor feature distributions, freshness, null rates, and the prediction distribution — not accuracy, which needs labels and arrives 90 days late. The first four are available immediately and are all data engineering.

And keep the boundary. "The model isn't good enough" is not your ticket. Answer the four questions you can — leaking, skewed, stale, mislabelled — and if all four come back clean, say so with evidence.

Chapter 33 is cloud cost optimization, the chapter this book has been building toward since the $3,840.00 Spark job in Chapter 1 — where the money goes, how to attribute it, and how to compute a cost before shipping a query rather than after receiving a bill.

Key terms: feature · entity · point-in-time correctness · as-of join · data leakage · training/serving skew · feature age · online store · offline store · materialization · backfill · feature versioning · labeling delay · drift