Key Takeaways: Privacy Engineering

The one thing

Privacy is where assertion becomes liability. Every claim in this chapter — we removed the PII, it's anonymized, we can delete a customer — is either measured or is a guess with a regulator attached. The discipline is producing a number where the industry produces an adjective.

The regulations, as engineering

Six capabilities cover the substance: find, export, delete, correct, restrict, account.

Capability one carries the other five. Export, delete, and correct are all "find, then do something." A platform that can locate a person in a query does all six cheaply; one that cannot does all six as investigations, forever.

The deadlines are weeks, not quarters — one month under GDPR, 45 days under CCPA, both extendable. Build for the shape, not the statute, because the statutes will change and the shape will not.

Pseudonymized is not anonymous. A salted hash is still personal data, still in scope for all six capabilities. This is the most common technical misunderstanding in the territory.

Know which questions are not yours. Is this personal data? What is our lawful basis? How long may we keep it? are legal determinations. Where is it, can we delete it, is this aggregate safe? are engineering. Confusing the two fails in both directions.

Finding it

Neither detection strategy alone clears two-thirds. Name matching: 12 of 18, 9 false positives, 66.7% recall. Value matching: 8 of 18, zero false positives, 44.4%. Together: 16 of 18, 64.0% precision.

The totals are the wrong comparison. Value matching finds three columns name matching cannot find at any tuning — customer_ref, external_key, legacy_col_3. The two strategies fail in different directions, which is why you run both.

False positives kill the program, not false negatives. 225 of them on a real warehouse, a report nobody finishes, and the real findings triaged away with the rest. Sort by confidence, suppress permanently, route only to owners.

A scanner without a suppression file is a scanner that gets turned off.

Know what yours cannot find. First names in a migration-named column and opaque hashes are not tuning problems.

The scan finds candidates; a human classifies; the tag is what the platform enforces.

What to do with a column

One question chooses between the middle two: will anyone ever need the original value back? Never → hash. Ever → tokenize. Ask the business owner, not the engineer, and write the answer down with a date.

The vault is on the critical path. If tokenization happens at ingestion and the vault is down, ingestion stops.

Dropping is under-used. Ask what decision the column changes (§30.1). Kestrel removed IP and user agent from 4.19 TB/year of clickstream — and the saving was not bytes, it was that those columns left the deletion report, the access review, the residency question, and the classification conversation permanently.

Anonymization

"We anonymized it" is the most over-claimed sentence in data work.

Quasi-identifiers do the damage. Postcode, birth year, and sex are each harmless and together identify people: 446 of 5,000 alone in their class, 48.2% below k = 5.

⚠️ Generalization does not degrade gracefully. Four of five rungs still had k = 1; the rung that worked threw away age and sex entirely. The choice is usually between anonymous-and-useless and useful-and-pseudonymous — and the failure is calling the second one the first.

📐 Pick the rung where suppression becomes affordable, not the rung where it becomes unnecessary. Case Study 2: 48.4% of rows lost at ZIP3, 3.1% at state, keeping both columns the recipient used.

Disclose what you suppressed. The removed rows are systematically the unusual people.

Anonymity comes from crowding, so a segment extract is far worse than its population. Same three columns: 6.41% at risk over the base, 48.35% over the 41,208-row slice — 7.5×. The more precisely a segment is targeted, the less anonymous it is.

🔐 Four safe releases are not one safe release. Behavioural columns that are not quasi-identifiers within a release become quasi-identifiers across releases, because a trajectory is nearly unique. k-anonymity is a property of a release, not of a sharing relationship.

k-anonymity does not give you homogeneity or composition. Differential privacy survives composition and costs accuracy — know it exists, know what it costs, and do not claim it if you have not implemented it.

Deletion

Three groups, not two. Has a mechanism · expires inside the deadline · gap.

📐 A location whose data expires inside the deadline needs no delete path — three weeks of Kafka tombstone work removed in one meeting. Conditional on the retention being enforced and monitored, and the verification immediately found a per-partition override at 30 days on a topic set to 3.

Kestrel: 8 with a mechanism, 3 that expire, 9 gaps, 55% covered. The number matters less than the fact that it is a number.

🔐 A person is not one identifier. Five identifier spaces at Kestrel; a deletion in one missed four, including the largest dataset in the platform, on three requests already marked complete. The resolver must use full history, not current state, and must be the same code for deletion and export.

🧱 Generate the manifest from classification tags; do not write it. A new table carrying customer_id joins automatically and fails verification loudly if it has no mechanism. This single move — generation, not enumeration — is the highest-leverage design in the chapter, and it repeats for the staging masking check.

Verification is what makes the pipeline real. Re-run the resolver, assert zero. Kestrel's has fired twice, both times on a table nobody had thought about.

🔐 You cannot surgically delete from a backup. Two honest answers: document the expiry and keep a deletion log to re-apply after any restore, or crypto-shred. The dishonest third answer — saying nothing — is the most common.

Snapshot before deleting. You must be able to prove what you deleted, and afterwards you cannot.

📏 The failure is not throughput. Forty requests a month is nothing; one erasure from Parquet bronze cost more compute than a night of Spark, and effectively nothing after the Iceberg migration.

Access control

Masking policies attach to the object, not the query — every path gets the same treatment, it is auditable in one place, and it fails closed.

💸 The masked-copy pattern is the default people reach for and is almost always worse. Masking overhead measured under 2%; the copy cost storage, a pipeline, and drifted twice, once for eleven days. Both require someone to act on a new column; only one of them leaks when they forget.

⚠️ Masking is not deletion, and a partial mask often is not a mask. The check is not "is the identifier masked" but "can a person be singled out from what remains" — which is the k-anonymity question again.

Consent, residency, environments

🔁 Consent is temporal, so exports are functions of an as-of timestamp — and the as-of for a send is the send time, not the build time. Reproducibility and correctness pull opposite ways; you need both mechanisms.

A consent check on a flow running on a different lawful basis is a bug that silently drops rows you were obliged to process.

⚠️ Residency reviews follow the flows people drew; the violations are in the flows nobody drew. A 0.02% parse-failure rate shipped EU rows to a US log aggregator through a helpful error handler. The fix is a rule in the logging library, not a review.

The staging warehouse is usually the largest exposure in the platform and the least monitored. The durable artifact is a test that fails when a tagged column reaches a non-production environment unmasked — Kestrel's has fired eleven times in two years, every one a new column.

The access request

Deletion has a well-defined success state and export does not. Zero rows is checkable; everything is a judgment about scope.

Derived data counts — LTV scores, segments, churn probabilities — and is on nobody's list.

Comprehensibility is an obligation, and the catalog's description field is the mapping from internal codes to meaning. The best argument for a catalog anyone at Kestrel had made in two years.

Building it in

🏭 Everything here is cheaper at ingestion. Removing two clickstream columns cost eleven weeks afterwards and would have cost one day at the loader — 55:1, on data nobody wanted, generating no analysis. Pure liability, accumulated by default.

Four controls, in the order they pay off: classify at ingestion · transform before landing · propagate tags directionally · put the rest in Chapter 23's register as assertions.

A privacy control that is an assertion cannot go stale; one that is a document can.

When you find something bad

🎓 Stop looking. Escalate within the hour. Do not delete it.

The strongest instinct is the wrong one. Deletion destroys the evidence that determines whether there is a notification obligation. Preserve, revoke access, write down what is known and what is not — keeping the two apart — and let legal make the call.

The strongest single sentence you can say is "I would not delete it."

The vocabulary fix

🔎 Three terms instead of one, because "anonymized" meant four different things to four people:

The middle row changed behaviour, because the claim is unusable without a number and the number requires running the measurement. Same move as §30.8's certified metric: not one word, but no two people meaning different things by the same word.

The code

code/pii_scan.py — detection scored three ways, k-anonymity over a generalization ladder, and deletion coverage against a statutory deadline. Eighty self-checks. --demo runs all three; Exercises 31.11 and 31.14 extend it.