Chapter 31: Privacy Engineering

"Legal asked how long it would take to delete one customer completely. I said I'd find out. Nine days later I had a list of thirty-one places their data lived and no idea what to do about eleven of them."

Overview

This is the chapter where getting it wrong is a legal event rather than an operational one, and that changes the engineering in three specific ways.

The deadlines are external. Chapter 26's 6am SLA is a promise the company made to itself and can renegotiate. A statutory response deadline cannot be renegotiated, is measured from an event you do not control, and the counterparty has a regulator behind it.

The requirement is about a person, not a table. Everything else in this book operates on datasets. Privacy operates on one individual, across every dataset, which is an access pattern most data platforms are built to be bad at.

And it is retroactive. A design decision made in Chapter 6 determines whether an obligation arriving today takes a query or a fortnight. Kestrel's dim_customer.email_hash (§6.7) was a five-minute decision that paid for itself here; the clickstream landing raw IP addresses in undeletable Parquet was a five-minute decision that cost eleven weeks.

What this chapter is not. It is not legal advice, and the distinction it maintains throughout is between the parts that are engineering — where is the data, can we delete it, is this aggregate safe — and the parts that are a legal determination — is this personal data, what is our lawful basis, how long may we keep it. Confusing the two is the single most common failure in this territory, in both directions: engineers deciding what counts as anonymous, and lawyers specifying a retention period with no mechanism (§30.7).

Chapter 30 gave you classification and retention as governance. This chapter is what happens when the same two fields have a statute behind them.

31.1 The Regulations as Technical Requirements

You do not need to read GDPR to build for it. You need the small number of things it obliges a system to be able to do, and there are fewer than most engineers expect.

Strip the legal apparatus and roughly six capabilities remain, shared in substance across GDPR (EU/UK), CCPA/CPRA (California), and the growing set of state and national laws modeled on them:

Capability 1 is the one everything else rests on, and it is the one nobody budgets for. Export, delete, and correct are all "find, then do something." A platform that can find a person in a query does all four cheaply. A platform that cannot does all four as investigations, forever.

🔐 Privacy & Governance — the four words that carry the most engineering weight

"Personal data." Broader than most engineers assume. It is not a list of column types; it is any information relating to an identifiable person, which includes an IP address, a device identifier, a cookie ID, and a salted hash of an email — all of which Kestrel's platform treats as ordinary technical columns. If a column can be used to single someone out, on its own or combined with something else you hold, assume it is in scope and ask.

"Pseudonymized." Personal data with the direct identifiers replaced. It is still personal data, it still falls under every obligation above, and treating a hash as if it were anonymous is the most common technical misunderstanding in this territory. Pseudonymization reduces risk; it does not remove obligations.

"Anonymous." Genuinely outside the scope — and the bar is far higher than "we dropped the name." §31.4 is about measuring it rather than asserting it.

"Controller" and "processor." Who decides why data is processed, versus who processes it on instruction. It determines who is on the hook, it is a legal determination, and the reason to know the words is so you can ask which one you are for a given flow rather than assuming.

The deadlines. GDPR obliges a controller to respond to an erasure or access request within one month of receipt, extendable by two further months for complex requests. CCPA gives 45 days, extendable by 45 more. The details vary and change; the engineering consequence does not — you are building for a deadline measured in weeks, not for one measured in quarters, and §31.5's coverage report is how you find out whether you can meet it.

🧭 Version Note — this material dates faster than anything else in the book

Chapter 21's Spark APIs will be recognizable in five years. The privacy landscape will not. CPRA amended CCPA; a dozen US states have passed laws since; adequacy decisions and transfer mechanisms have been struck down and replaced twice.

What is stable is the shape: find, export, delete, correct, restrict, account — against a deadline, for one person, across everything. Build for the shape. A platform that can find and delete a person in a query adapts to a new statute in a week; one that cannot adapts in a quarter, every time, forever.

And the practical consequence for you: confirm the current rules with counsel, not with this book. Every number in this section is illustrative of a shape and may not be current when you read it.

31.2 Finding Personal Data You Did Not Know You Had

You cannot protect what you cannot locate, and every organization past a certain age holds personal data in places nobody has written down. This section is a measurement, and the measurement is discouraging in a useful way.

code/pii_scan.py implements the two standard strategies against a fixture of 40 columns, of which 18 are personal data:

Name matching. Tokenize the column name and match against a vocabulary — email, phone, dob, postcode, ip. Cheap, instant, needs no data access.

Value sampling. Read a sample of rows and match against value shapes — an email regex, an IPv4 pattern, a card-number prefix, a government-ID format. Slower, needs read access to the data itself.

The measured result:

strategy        found false+ missed   precision    recall
name only          12      9      6       57.1%     66.7%
value only          8      0     10      100.0%     44.4%
name+value         16      9      2       64.0%     88.9%

Three findings, and the second is the one worth carrying:

Neither strategy alone clears two-thirds. Name matching finds more (12 vs 8) and is wrong nine times; value matching is never wrong and finds fewer than half.

The totals are the wrong comparison. What matters is that value matching finds three columns name matching cannot find at any tuning — customer_ref (an email column named as a foreign key), external_key (IP addresses), and legacy_col_3 (government-ID shapes). No vocabulary reaches a column called external_key. That asymmetry, not the recall figure, is why you run both.

Nine false positives out of 22 non-personal columns. supplier_address_id is a foreign key. email_template_id is a template. phone_case_sku is a product category. card_present_flag is a boolean. dob_required_flag is a form setting. Every one of these trips a name rule and none is personal data.

⚠️ Failure Mode — the false positives are what kill the program, not the false negatives

A scan that misses two columns is a gap you close later. A scan that flags nine harmless columns is a report nobody finishes reading, and the second failure destroys the first one's value.

The arithmetic at real scale is brutal. Kestrel has 290 tables. A scanner at this precision (64.0%) applied to a warehouse with, say, 400 genuinely personal columns produces 225 false positives — and they arrive as a spreadsheet sent to owners who have work to do.

What happens next is predictable. The first ten owners investigate carefully. The next twenty notice the pattern and start dismissing. By the second run, the report is triaged in bulk, and the two real findings buried in it are dismissed along with everything else — which is Chapter 25 Case Study 2's alert fatigue, arriving in a fourth form.

Three things that actually help, in order of effect:

• Report confidence, and sort by it. high (name and values agree, or values alone) first; medium (name only) in a separate section that is explicitly a maybe. The tool does this and it is most of the fix.
• Suppress permanently. A dismissed column stays dismissed with a reason and a date, and the next run does not re-raise it. A scanner without a suppression file is a scanner that gets turned off.
• Never route a medium finding to anyone but the column's owner. They can answer it in ten seconds; a central privacy team cannot answer it at all.

And the two it misses are the honest limit. session_id is a hex hash — no name signal, no value shape. attr_07 holds first names in a column a migration named. Neither is a tuning problem; first names are not regex-detectable without a name dictionary that itself misfires on every city and product_name column in the warehouse.

🧪 Try It — run the scan and argue with it

bash cd part-06-advanced-topics/chapter-31-privacy-engineering/code python pii_scan.py --scan

The output marks false positives with ! and lists what it missed. Before reading the summary table, cover it and predict the three numbers. Most people predict name matching will be the precise one.

Then: add a rule that catches attr_07. Any rule you can write will also flag warehouse_code, carrier, or promo_code. Measure the cost — --scan prints precision, so you can see exactly what your rule bought and what it spent.

What to do with the result. The scan is an input to a conversation, not an answer. Every high finding gets a classification decision (§30.6) from the person who can make it, and the output of that conversation is the tag that everything downstream — masking, retention, deletion, residency — keys off. The scan finds candidates; a human classifies; the tag is what the platform enforces.

31.3 Four Things You Can Do With a Sensitive Column

Once a column is classified, there are four options and they are not interchangeable. Choosing badly is the most common source of the awkward conversation two years later where a capability you needed turns out to have been destroyed.

Keep it, with controls. Correct when the business genuinely needs the value — you cannot email a customer a hash. The control is access (§31.6), not transformation, and the price is that the column is in scope for everything in §31.1.

Hash it. A salted hash preserves equality: two records for the same email still join. The salt must be secret and stable — an unsalted hash of an email is trivially reversed by hashing a dictionary, and a rotating salt breaks every historical join. This is Kestrel's email_hash (§6.7), and it is still personal data (§31.1).

Tokenize it. Replace the value with a random token, keeping the mapping in a separate, tightly controlled vault. Everything a hash gives you, plus reversibility for the small number of legitimate cases, at the cost of running a vault that is now the most sensitive system you own.

Drop it. The only option that removes the obligation, and the one most under-used. Ask what decision the column changes (§30.1); a surprising share of retained personal data changes none.

📐 Design Decision — hash or tokenize, and the question that settles it

Teams argue about this for weeks. One question resolves most cases:

Will anyone ever need the original value back from this dataset?

If genuinely never: hash. It is simpler, has no vault to run, no vault to breach, and no vault to keep available. The irreversibility is the feature.

If ever, even rarely: tokenize. A hash cannot be un-hashed later when someone discovers a legitimate need, and the fallback — rebuilding from a source system that may no longer have the data — is exactly the scramble this decision was supposed to prevent.

The trap is that "never" is usually said by the person building the pipeline and contradicted by someone else eighteen months later. So ask the question of the business owner, not the engineer, and write down the answer with a date next to it, because the next person will need to know it was decided rather than defaulted.

And a real cost that gets forgotten: the vault is now on the critical path. If tokenization happens at ingestion (§31.5) and the vault is down, ingestion stops. Kestrel's vault sits behind the same availability requirement as the 6am SLA, and that requirement was discovered during an incident rather than during design.

💸 Cost Check — dropping the column is free and the alternatives are not

Kestrel's clickstream lands 14,000,000 events/day, 11.48 GB/day of JSON, and each event carried a full IPv4 address and user agent.

The IP address changed exactly one decision — a coarse country-level geography used in a monthly report — and the user agent changed one more, a device-class breakdown.

Replacing both at ingestion with the two derived fields they feed (country_code, device_class) removed two personal-data columns from 4.19 TB/year of raw JSON and 341 GB/year of Parquet, and from every downstream copy of both.

The saving is not the bytes. It is that these columns dropped out of the deletion coverage report, out of the access review, out of the residency question, and out of the classification conversation — permanently, for a one-day change. §31.11's argument in its cheapest form.

31.4 Anonymization, and Why It Usually Isn't

"We anonymized it" is the most over-claimed sentence in data work. Anonymous data falls outside these regulations entirely, which makes the claim enormously valuable and therefore enormously tempting — and the standard practice, dropping the direct identifiers and calling the result anonymous, does not come close.

The reason is quasi-identifiers. A postcode is not identifying. A birth year is not identifying. A sex is not identifying. The three together identify a large fraction of a population, and this is not a theoretical concern — it is the finding that produced the entire field.

pii_scan.py --k-anon measures it on 5,000 synthetic Kestrel customers:

quasi-identifiers               classes      k     rows<k singletons
--------------------------------------------------------------------
postcode + birth_year + sex        1509      1      2,411        446
postcode + age_band + sex           414      1        287         38
postcode_3 + age_band + sex         252      1        176         22
postcode_3 + age_band                92      1         26          6
postcode_3 only                       6    210          0          0

Read the first row. With postcode, birth year, and sex retained — a table most people would describe as anonymized, because the name and email are gone — 446 people are alone in their equivalence class, and 2,411 of 5,000 rows (48.2%) sit in a group smaller than five. Anyone who knows a neighbor's approximate age can pick them out.

k is the size of the smallest equivalence class. A dataset is k-anonymous when every combination of quasi-identifiers appears at least k times, so no individual can be narrowed below a group of k. The convention is k ≥ 5, and it is a floor, not a target.

⚠️ Failure Mode — generalization does not degrade gracefully

The intuition is that coarsening the data gradually increases privacy. The measurement says otherwise, and the shape of the table above is the point of this section.

Four of the five rows have k = 1. Truncating the postcode to three digits, banding the age into five-year groups, and dropping sex still leaves six people alone in their class. Each step reduces the damage — 446 singletons to 38 to 22 to 6 — but none of the first four steps reaches the threshold at all.

And the step that finally works throws away almost everything. postcode_3 alone reaches k = 210, which is comfortably anonymous and is no longer a dataset anyone wanted: no age, no sex, six groups total.

This is the honest shape of anonymization, and it is why the claim is usually false. There is rarely a comfortable middle where the data is both safe and useful. The choice is normally between a genuinely anonymous dataset that has lost the analysis, and a useful dataset that is pseudonymous — still in scope, still access-controlled, still deletable.

The failure is not choosing the second one. It is choosing the second one and calling it the first, because everything downstream — retention, sharing, the vendor agreement, the deletion obligation — was then decided on a false premise.

Two things k-anonymity does not give you.

Homogeneity. A class of 200 people is k-anonymous, and if all 200 have the same diagnosis, knowing someone is in it tells you their diagnosis. l-diversity requires the sensitive attribute to vary within each class.

Composition. Two k-anonymous releases from the same population can be joined, and the join is not k-anonymous. This is the failure mode of "we publish an anonymized extract every quarter."

Differential privacy is the rigorous alternative, and the honest summary is: it provides a mathematical guarantee that survives composition, it requires adding calibrated noise to every answer, and the noise is large enough to matter for most business analytics. It is the right tool for published statistics and research releases; it is rarely the right tool for an internal dashboard where somebody needs the actual revenue number. Know it exists, know what it costs, and do not claim it if you have not implemented it.

🎓 Interview Angle — "how would you anonymize this dataset?"

The answer that fails is a list of techniques: drop the name, hash the email, bucket the ages. It answers a question about mechanics and misses that the question is about a claim.

The answer that works has three moves:

Ask what the dataset is for. Anonymization is a trade against a specific analysis. "Anonymize this" is not answerable without knowing which columns the analysis needs, because those columns are exactly what you are negotiating over.

Name the quasi-identifiers, not just the identifiers. "Dropping the email is the easy part. The question is whether postcode, age, and sex are staying, because those three are usually what re-identifies people." This is the sentence that separates people who have done it from people who have read about it.

Say how you would check. "I'd compute the equivalence-class sizes and look at the smallest one before claiming anything." Measuring rather than asserting is the whole discipline, and offering to measure is a stronger answer than any technique.

A good follow-up to expect: "you got k to 5 — is it anonymous now?" The answer is that k = 5 is a floor for one release, says nothing about homogeneity within classes, and does not survive joining with your other releases.

31.5 Deletion: The Request That Has to Reach Everywhere

A deletion request is a distributed transaction against systems that were never designed to participate in one, and the honest first step is to find out how bad it is.

pii_scan.py --coverage takes a manifest of every place a customer's data lives and a statutory deadline, and sorts the locations into three groups:

DELETION COVERAGE -- statutory deadline 30 days

HAS A MECHANISM (8)
   postgres.customers                     cascading row delete
   postgres.orders                        cascading row delete
   snowflake.gold.dim_customer (SCD2)     delete all versions
   snowflake.gold.fct_order_line          delete by customer_id
   snowflake.silver.stg_orders            rebuilt from bronze
   BI extract cache                       scheduled refresh
   vendor: email platform                 API delete
   data science feature store             delete by entity

NO MECHANISM, BUT EXPIRES WITHIN 30d (3)  -- acceptable, if documented
   kafka topic: orders.cdc                expires in 7d
   kafka topic: clickstream               expires in 3d
   application logs (Datadog)             expires in 15d

GAP -- OUTLIVES THE DEADLINE, NO MECHANISM (9)
   s3 bronze/orders/*.parquet             never expires
   s3 bronze/clickstream/*.json.gz        expires in 1095d
   snowflake Time Travel                  expires in 90d
   snowflake Fail-safe                    expires in 97d
   rds automated backups                  expires in 35d
   analyst CSV exports (laptops)          never expires
   vendor: support desk                   never expires
   s3 access logs                         expires in 400d
   ml training snapshots                  never expires

  covered 11/20 = 55%   gap 9

The middle group is the contribution of this report and it is the thing most deletion projects get wrong in both directions.

📐 Design Decision — a location that expires inside the deadline needs no delete path

Kafka's orders.cdc topic has a 7-day retention and the deadline is 30 days. Building a delete path into a Kafka topic is genuinely hard — you must produce tombstones, wait for compaction, and reason about consumers mid-replay — and it is completely unnecessary here, because the data is gone before the obligation matures.

This is a real saving. Three of Kestrel's twenty locations moved from "hard engineering problem" to "write down the retention and monitor it" in one conversation.

But it holds only under two conditions, and both must be checked rather than assumed:

• The retention is enforced, not aspirational. §30.7's missing mechanism. A topic configured for 7 days that has never been verified is not a 7-day topic.
• The retention is monitored. A configuration change from 7 days to 90 days silently converts a compliant location into a gap, and nothing about the deletion process would notice. Kestrel alerts on retention configuration changes for exactly this reason.

Get either wrong and you have a false sense of coverage, which is worse than a known gap because a known gap gets worked on.

The nine gaps sort into three kinds, and they need different answers:

Undeletable file formats. bronze/orders/*.parquet is immutable files with no index by customer. Deleting one person means rewriting every file that might contain them, which for a three-year history is the whole dataset. This is what Chapter 10's table formats solve — Delta and Iceberg support row-level deletes with deletion vectors — and Kestrel's migration of bronze to Iceberg was justified by this requirement more than by any query performance argument.

Backups and time travel. Snowflake Time Travel (90 days), Fail-safe (97 days), RDS automated backups (35 days). All three outlive a 30-day deadline and none supports selective deletion.

🔐 Privacy & Governance — the backup problem, and the two answers that are honest

You cannot surgically delete one person from a backup, and any design that claims to is either restoring, deleting, and re-taking the backup for every request, or is wrong.

Answer one: document the expiry and defend it. The position is that data in backups is not processed, is inaccessible except through a restore, and will age out on a known schedule — and that any restore is followed by a re-application of the pending deletion log. This requires you to keep a deletion log, which is the part people skip, and which is the only thing that makes the position true rather than convenient.

Answer two: crypto-shredding. Encrypt each person's data with a per-person key; delete the key. The ciphertext in the backup becomes unreadable, which is a defensible form of deletion. It is real, it is used in practice, and it is a substantial architectural commitment: per-subject key management, key availability on the critical read path, and a re-encryption story for key rotation. It is worth it when your backup retention is long or your restore frequency is high, and it is over-engineering when your Time Travel window is 90 days and your deadline is 30.

What is not honest is the third answer, which is the most common: saying nothing about backups at all and hoping the question does not come up. The regulators' guidance is generally accommodating on this point; it accommodates a documented position, not an absence of one.

Copies nobody controls. Analyst CSV exports on laptops, the support desk vendor, ML training snapshots. These are not solved by engineering — they are solved by not creating them (§31.6's masked views remove the reason to export), by contract (the vendor's deletion API), and by policy with a mechanism behind it.

🧱 Kestrel Platform — the deletion pipeline

The request arrives at a queue and is processed as a DAG (Chapter 24), not as a script:

text 1. resolve email -> customer_id, plus every alias, device, and session id 2. record write to deletion_log (subject, requested_at, deadline) 3. snapshot export for the audit trail BEFORE deleting <-- see below 4. fan out one task per location in the manifest, in dependency order 5. verify re-run the resolver; assert zero rows anywhere 6. attest write completed_at, per-location evidence, and any exceptions

Four things worth copying:

Step 1 is where the work is. A person is not one identifier. Kestrel resolves an email to a customer_id, then to every device_id and session_id ever associated with it — and the clickstream is keyed by device, not by customer, so without this step deletion silently misses the largest dataset in the platform.

Step 3 is counterintuitive and necessary. You must be able to prove later what you deleted, and after deleting you cannot. The snapshot is written to a separate, short-retention, tightly restricted store — and it is itself subject to the retention it documents.

Step 5 is the only part that makes it real. It re-runs the find query and asserts zero. A deletion pipeline without a verification step is a deletion pipeline that has been failing silently since the last schema change — and Kestrel's caught exactly that, twice, both times a new table nobody had added to the manifest.

The manifest is generated, not written. It is derived from the catalog's classification tags (§30.6), so a new table carrying customer_id appears in the deletion manifest automatically and fails step 5 loudly if it has no mechanism. This is the single highest-leverage thing in the chapter: it converts "remember to add it" into "it is impossible to forget."

📏 Scale Note — the deletion rate is small and the failure is not throughput

Kestrel receives roughly 40 erasure requests a month. At that volume, nothing about deletion is a performance problem, and the instinct to batch or optimize is misplaced.

The cost is entirely in the rewrite. A single deletion from bronze/orders under the old Parquet layout rewrote every file that might contain the customer — with a three-year history partitioned by date, that is the whole table. One request cost more compute than a night of Spark.

After the Iceberg migration, a deletion writes a small deletion vector and the cost falls to effectively nothing until the next compaction. The economics inverted entirely, and the requirement that justified the migration was this one — not query speed, which was the argument everybody had been making without success for a year.

31.6 Masking and Row-Level Security

Most people who can see personal data do not need to. Masking is how you reduce the population with access without reducing the population who can work.

Dynamic masking applies a transformation at query time, based on the querying role. The stored value is unchanged; what you see depends on who you are.

-- Snowflake
CREATE MASKING POLICY email_mask AS (val string) RETURNS string ->
  CASE
    WHEN current_role() IN ('PII_READER', 'SUPPORT_AGENT') THEN val
    WHEN current_role() IN ('ANALYST') THEN regexp_replace(val, '.+@', '***@')
    ELSE '***'
  END;

ALTER TABLE gold.dim_customer MODIFY COLUMN email
  SET MASKING POLICY email_mask;

Row-level security filters which rows a role sees at all:

CREATE ROW ACCESS POLICY eu_rows AS (region string) RETURNS boolean ->
  current_role() = 'GLOBAL_ADMIN'
  OR region = current_account_region();

Three properties that make these worth the setup cost:

The policy is attached to the object, not to the query. Every path — BI tool, notebook, psql, a scheduled job — gets the same treatment. This is the difference from a masked view, which protects only the people who remember to use it.

It is auditable in one place. "Who can see raw email?" is a policy lookup, not a survey of every query (§30 Case Study 2's question).

It fails closed. A role not named in the policy gets the ELSE branch. A new role sees masked data by default, which is the opposite of how grants behave.

⚠️ Failure Mode — masking is not deletion, and the partial mask is not a mask

Two mistakes, and the second is subtle enough to survive review.

Masking does not satisfy an erasure request. The value is still stored, still in backups, still exported by a COPY run under a privileged role. It reduces exposure; it changes no obligation in §31.1. Teams that conflate the two discover this during an audit.

And a partial mask often is not one. regexp_replace(val, '.+@', '***@') above leaves the domain — which is fine for gmail.com and identifying for a corporate domain with four employees. Kestrel's version leaked more than that: an early policy masked the email but left first_name, last_name, and postcode visible, on the reasoning that the email was the identifier. §31.4 is the refutation — those three columns identify people without the email at all.

The check is not "is the direct identifier masked." It is "can a person be singled out from what remains", which is the same k-anonymity question, and it is why masking policy and anonymization analysis have to be done by the same person at the same time.

💸 Cost Check — masking is nearly free and the alternative is not

Snowflake evaluates masking policies during query execution. Kestrel measured the overhead on dim_customer scans at under 2% — inside the noise of warehouse contention.

Compare the alternative Kestrel had been running: a second copy of dim_customer with the PII columns removed, refreshed nightly. A 1,904,221-row table, an extra model, an extra test suite, an extra thing to be stale, and — the part that made the decision — the masked copy had drifted from the real one twice, once for eleven days, because a new column was added to the source and not to the copy.

The masked-copy pattern is the default people reach for and it is almost always worse: it costs storage, costs a pipeline, and fails open — a column added to the source is visible in the copy only if someone remembers, whereas a masking policy applies to a column only if someone attaches it. Neither is automatic, but the failure directions are opposite, and one of them leaks.

31.7 Consent and Purpose Limitation

Consent is not a boolean and it is not stored where you would like it to be.

Purpose limitation is the principle that data collected for one purpose may not be freely used for another. Technically, this means a row is not simply present or absent — it is present for some purposes, and the pipeline must know which.

The engineering shape is a per-subject, per-purpose state that changes over time, which is a slowly changing dimension (Chapter 20) with a legal deadline attached:

CREATE TABLE consent (
  customer_id   BIGINT      NOT NULL,
  purpose       TEXT        NOT NULL,   -- 'marketing','analytics','personalization'
  granted       BOOLEAN     NOT NULL,
  source        TEXT        NOT NULL,   -- where the choice was made
  effective_at  TIMESTAMPTZ NOT NULL,
  PRIMARY KEY (customer_id, purpose, effective_at)
);

Three consequences that surprise people:

Consent is temporal, so your pipelines are too. A marketing export must reflect consent as of the send, not as of the last nightly build. A withdrawal at 09:00 that is honored at the next 02:00 run is a 17-hour window in which you processed data you were not permitted to process.

Withdrawal must propagate to derived data. If a customer withdraws marketing consent, the segment they are in, the model trained on them, and the vendor who has their address all need updating. §31.5's manifest again.

The lawful basis matters more than the switch. Consent is one of several lawful bases; much ordinary processing runs on contractual necessity or legitimate interests, which have no switch at all. Which basis applies to which flow is a legal determination, and the engineering consequence is substantial: a flow running on consent needs a consent check; one running on contract does not, and adding one is a bug — it will silently drop rows you were obliged to process.

🔁 Idempotency Check — the pipeline that honored consent as of the wrong moment

Kestrel's marketing export filtered on the current consent table at build time — which is correct, and which made the export non-reproducible. Rebuilding yesterday's export today produced a different file, because consent had changed in between.

The rebuild broke a reconciliation — the vendor's record of who was sent to, versus the platform's record of who was exported — and the investigation took two days before anyone thought to ask whether the file was even supposed to be stable.

The fix is Chapter 20's valid_from / valid_to, applied to consent, so an export is a function of (as-of timestamp) and is reproducible forever:

sql WHERE c.purpose = 'marketing' AND c.granted AND c.valid_from <= :as_of AND (c.valid_to IS NULL OR c.valid_to > :as_of)

And now the subtle part, which cost a further half-day of argument: the as-of timestamp for a send is the send time, not the build time. A file built at 02:00 and sent at 09:00 must be filtered again — or built at 09:00 — because a withdrawal at 07:00 is binding at 09:00. Reproducibility and correctness pull in opposite directions here, and the resolution is that the export is reproducible given an as-of, and the send re-checks. Two mechanisms, because they answer two questions.

31.8 Residency: Where the Bytes Physically Are

Some data may not leave a jurisdiction, and this is one of the few requirements in this book that constrains physical infrastructure rather than logic.

Three things it touches, and the third is the one that gets missed:

Storage. The bucket, the warehouse, the database — in-region. Straightforward and usually the only part anyone checks.

Processing. The Spark cluster, the warehouse compute, the orchestrator's worker. A job reading EU data on a US-based worker has moved the data, regardless of where it was written back.

Everything peripheral. This is where residency programs fail. Logs containing sample rows. Monitoring that ships query text. Error-tracking that captures a payload. A support tool. A vendor's US-hosted dashboard.

⚠️ Failure Mode — the log line that crossed the border

Kestrel's EU data was correctly stored in eu-west-1 and processed by EU workers. The residency review passed.

What it missed: the pipeline's error handler logged the offending row on a parse failure, and logs shipped to a US-hosted aggregator. At a 0.02% parse-failure rate on EU orders, that is a steady trickle of EU personal data crossing the border, through a code path written by someone who was being helpful.

Nothing in the architecture diagram showed it, because a log line is not a data flow to anybody drawing an architecture diagram.

The general lesson: residency reviews follow the data flows people drew, and the violations are in the flows nobody drew. Logs, metrics, traces, error payloads, support exports, screenshots in tickets.

The check that works is not a review; it is a rule. Kestrel's is one line in the logging library — structured logging that refuses to serialize a field tagged as personal data, failing at the log call rather than at the border. It is Chapter 27's shift-left applied to a legal requirement, and it caught four more instances in its first month.

The architectural answer is a per-region deployment: separate buckets, separate warehouses, separate orchestrator instances, and only aggregates crossing the boundary. It is more expensive and more operationally complex than a single global platform, and the decision is a legal one about which data is genuinely constrained — which is worth establishing precisely, because the cost of applying residency to everything is high and the cost of applying it to nothing is higher.

31.9 Personal Data in Non-Production Environments

The staging warehouse is usually the largest privacy exposure in the platform, and it is usually unmonitored, because it does not feel like production.

Chapter 27 §27.6 established the rule — production data does not go to staging in raw form. Here is what the alternatives actually cost:

The dominant failure is the masked copy that is not completely masked. A new column arrives in production, the masking configuration is not updated, and the staging environment silently acquires real personal data with production-grade volumes and development-grade access controls.

🧭 Version Note — the masking-completeness check is the part to build

Whatever you choose, the durable artifact is a test that fails when a personal-data column reaches a non-production environment unmasked. Kestrel's runs after every staging refresh:

python for table, column in classified_columns(tier="confidential"): assert is_masked_in(STAGING, table, column), \ f"{table}.{column} is unmasked in staging"

It reads the classification tags from the catalog (§30.6), so a newly-tagged column is covered the moment it is tagged. This is the same generation-not-enumeration move as the deletion manifest (§31.5), and it is the one design pattern in this chapter that transfers to every other privacy control you will build.

It has fired eleven times at Kestrel in two years. Every single one was a new column, and none would have been caught by a review.

31.10 The Access Request: Export, Not Delete

A subject access request asks for a copy of everything you hold, and it is a harder engineering problem than deletion for a reason that is not obvious: deletion has a well-defined success state and export does not.

Deletion is done when the row count is zero. Export is done when you have produced everything, and everything is a judgment about scope — does it include derived tables, model features, log lines, the support ticket, the inferred segment?

Three specific difficulties:

Derived data. A customer's LTV score, their assigned segment, a churn probability. These are data about them, generally in scope, and usually not on anybody's list.

Comprehensibility. The obligation is generally to provide data in a form the person can understand, which a database dump with 340 columns of internal codes is not. Somebody has to map codes to meanings — and the catalog's description field (§30.2) is exactly that mapping, which is the best argument for a catalog anyone at Kestrel had made in two years.

Other people's data. An order shipped to a shared address, a support conversation, a referral. Producing one person's record must not disclose another's, and this is a genuine judgment that belongs to the same person who makes the classification calls.

The engineering that helps is the same engineering as deletion: the resolver (email → every identifier) and the generated manifest. Build them once, use them for capability 1, 2, 3, and 4 in §31.1. Kestrel's export DAG is the deletion DAG with step 4 replaced.

31.11 Privacy as a Pipeline Stage, Not a Review

Everything in this chapter is cheaper at ingestion than anywhere downstream, and the multiplier is not small.

A privacy review is a gate (§30.1's second failure pattern). It runs at the end, finds problems when they are expensive, produces a document, and is routed around when it becomes an obstacle.

A privacy pipeline stage is a by-product (§30.11). It runs on every load, produces its evidence automatically, and cannot be routed around because it is not a step anyone can skip.

Four controls, in the order they pay off:

1. Classify at ingestion. A new column arrives untagged and the load fails, or lands in a quarantine tier. The classification decision happens once, when someone is already thinking about the column, not two years later during an audit.

2. Transform before landing. §31.3's country_code instead of the IP. The strongest case for ETL over ELT in this book (Chapter 3 §3.4), and the only one that is not negotiable — data you legally may not store cannot be landed and cleaned later.

3. Tag propagation. A model reading a confidential column inherits the tag (§30.6). Directional, automatic, and the thing that makes masking policies maintainable — otherwise every new model is a new manual decision.

4. Assertions in the register. Chapter 23. email must not appear in gold. Staging must have no unmasked confidential column. Every published count must respect its bound (Chapter 30 Case Study 1). A privacy control that is an assertion cannot go stale (§30.11); one that is a document can.

🏭 From the Pipeline — what eleven weeks bought, and what one day would have

Kestrel's clickstream landed raw IP addresses and user agents in date-partitioned JSON for three years before anyone asked whether it should.

Removing them afterwards took eleven weeks: a classification decision, a rewrite of 4.19 TB of historical JSON, a migration of every downstream model, a re-verification of three dashboards, and a deletion-coverage re-run.

Doing it at ingestion would have taken one day — the day the clickstream loader was written, when the person writing it already had the schema in front of them and the question "do we need the full IP?" would have taken ten minutes to answer.

The ratio is roughly 55:1, and it is not the interesting part. The interesting part is that the eleven weeks were spent on data nobody wanted, generating no analysis, answering no question — pure liability, accumulated by default.

The generalizable form: for any personal-data column arriving in a new pipeline, the cost of asking now is minutes and the cost of asking later is weeks, and the question is the same question. It is the cheapest thing in this chapter and the most reliably skipped, because at ingestion time it feels like premature caution and by the time it feels urgent it is expensive.

31.12 When You Find Something Bad

You will, and how the first hour goes determines how the next month goes.

A worked example, from Kestrel. During the §31.2 scan, an engineer found a legacy_col_3 in a deprecated table holding what looked like government ID numbers, in a table with no owner, granted to nine roles.

What they did, in order:

Stopped and did not investigate further. The instinct is to query it and see how bad it is — which creates an access record and possibly makes it worse. They confirmed the shape from the scan output they already had and stopped.

Escalated within the hour, to the named person in the incident process (Chapter 26 §26.3), who escalated to legal. Not to a group channel, and not with a screenshot.

Preserved rather than deleted. The instinct to delete it immediately is wrong and can destroy evidence needed to determine what happened and what has to be reported. Access was revoked; the data was not touched.

Wrote down what was known and what was not, and kept the two apart. "A column of 8,412 values matching a government-ID format. Not confirmed as real. Nine roles have SELECT. Last written 2023-11. No owner."

Let legal determine notification. Whether this is reportable, to whom, and within what deadline is not an engineering call. The engineering call is producing the facts quickly and accurately.

🎓 Interview Angle — "you find unencrypted PII in a table you don't own. What now?"

A common question, and most answers fail by being too active. "I'd delete it," "I'd fix the permissions," "I'd message the team" — all reasonable-sounding, all wrong as a first move.

The answer that works names the escalation path first and the technical response second, and shows you understand this is a reportable-event question rather than a cleanup question:

"I'd stop looking at it, so I'm not creating more access records. I'd escalate to whoever owns incident response within the hour, and I'd write down exactly what I saw and what I don't know. I would not delete it — that can destroy evidence and there may be a notification obligation that depends on knowing what was exposed and for how long. The technical fix comes after someone with the authority to make that call has made it."

The strongest single sentence you can offer is "I would not delete it." It is counterintuitive, it is right, and it demonstrates that you know the difference between an operational problem and a legal one. Then add what you would build afterwards — the classification tag, the assertion, the scan in CI — because the interviewer is also asking whether you fix causes.

31.13 The Kestrel Platform

What exists after this chapter:

platform/privacy/
├── scan.py                  # section 31.2 -- runs weekly, writes findings
├── suppressions.yml         # dismissed columns, with reason and date
├── manifest.py              # GENERATED from catalog tags -- every location
├── resolver.py              # email -> customer_id -> devices -> sessions
├── dags/
│   ├── erasure.py           # the 6-step DAG in section 31.5
│   └── access_request.py    # same DAG, step 4 replaced (31.10)
├── policies/
│   ├── masking.sql          # dynamic masking policies (31.6)
│   └── row_access.sql       # residency and territory filters
└── checks/
    ├── staging_masked.py    # fires on every staging refresh (31.9)
    ├── no_pii_in_gold.sql   # an assertion in Chapter 23's register
    └── retention_config.py  # alerts when a TTL changes (31.5)

The numbers, measured:

And one honest row: the nine gaps are still nine gaps. Four have documented positions (backups, Time Travel, Fail-safe, access logs), three are being closed (bronze to Iceberg, the support vendor, training snapshots), and two — analyst CSV exports and the long-tail of copies — are not solved by engineering and are being reduced by removing the reason to make them (§31.6).

The point of the report is not that coverage is 55%. It is that 55% is a number, and last year it was a nine-day investigation producing an anxious guess.

31.14 Summary

Privacy is where an engineering decision becomes a legal exposure, and the discipline that survives contact with it is measurement rather than assertion.

🔐 Six capabilities cover the substance of the major regulations: find, export, delete, correct, restrict, account. Capability one carries the others — a platform that can find a person in a query does all six cheaply, and one that cannot does all six as investigations, forever.

⚠️ Finding personal data is a measurement, and neither strategy alone clears two-thirds. Name matching found 12 of 18 with 9 false positives; value matching found 8 with none; together 16 of 18, at 64.0% precision. The false positives are what kill the program — 225 of them on a real warehouse, a report nobody finishes, and the two real findings triaged away with the rest. Report confidence, suppress permanently, route only to owners.

📐 Four options for a sensitive column, and one question chooses between the two hard ones: will anyone ever need the original value back? Never → hash. Ever → tokenize. Ask the business owner, not the engineer, and write down the answer with a date — and remember the vault is now on the critical path.

⚠️ Anonymization does not degrade gracefully. With postcode, birth year, and sex retained, 446 of 5,000 people are alone and 48.2% sit in a group below five. Four of five generalization steps still leave k = 1, and the step that finally works throws away almost everything. The choice is usually between anonymous-and-useless and useful-and-pseudonymous — and the failure is not choosing the second, it is calling it the first.

📐 A location whose data expires inside the statutory deadline needs no deletion path — but only if the retention is enforced and monitored, or you have bought a false sense of coverage. Kestrel: 8 locations with a mechanism, 3 that expire in time, 9 gaps, 55% covered — and the 55% is worth more than the previous nine-day guess.

🔐 You cannot surgically delete from a backup. Two honest answers: document the expiry and keep a deletion log to re-apply after any restore, or crypto-shred. The dishonest third answer is saying nothing.

🧱 Generate the manifest from the catalog's tags, do not write it. A new table carrying customer_id joins the deletion manifest automatically and fails verification loudly if it has no mechanism. This single move — generation, not enumeration — is the highest-leverage design in the chapter, and it repeats for the staging masking check.

⚠️ Masking is not deletion, and a partial mask often is not a mask. Masking the email while leaving name, postcode, and age visible is §31.4's failure with extra steps. The check is not "is the identifier masked" but "can a person be singled out from what remains."

🔁 Consent is temporal, so exports are functions of an as-of timestamp — and the as-of for a send is the send time, not the build time. Reproducibility and correctness pull opposite ways here; you need both mechanisms.

⚠️ Residency reviews follow the flows people drew; the violations are in the flows nobody drew. A 0.02% parse-failure rate shipped EU rows to a US log aggregator through a helpful error handler. The fix is a rule in the logging library, not a review.

🏭 Everything here is cheaper at ingestion. Removing two columns from Kestrel's clickstream cost eleven weeks afterwards and would have cost one day at the loader — a 55:1 ratio on data nobody wanted, generating no analysis, pure liability accumulated by default.

🎓 When you find something bad: stop looking, escalate within the hour, and do not delete it. The strongest instinct is the wrong one — deletion destroys the evidence that determines whether there is a notification obligation. Produce facts fast and accurately; the call is not yours.

Chapter 32 moves to feature stores and ML engineering, where the same data has to be correct twice — once in training and once in serving — and where the gap between the two has its own name.

Key terms: personal data · pseudonymization · anonymization · quasi-identifier · k-anonymity · l-diversity · differential privacy · tokenization · crypto-shredding · dynamic data masking · row-level security · purpose limitation · lawful basis · data residency · subject access request