Case Study 2: Two Hundred Datasets, Four Owners

"We archived ninety-six datasets. Two people complained, both about the same one, and it turned out they were using a copy of it anyway."

Executive Summary

A company's data lake had grown to about 200 distinct datasets over three years. An audit asked three questions of each — who owns it, what does it mean, is anything reading it — and could answer none of the three for 112 of them.

The remediation was not technical. It was a deadline with a consequence: every dataset must have a named owner and a one-paragraph description by a date, or it is moved to an archive prefix and access is revoked. Fourteen weeks later, 71 datasets had been claimed and documented, 96 were archived without complaint, and 33 turned out to be duplicates of one another.

This case study is the mechanics of that audit: how you determine whether anything reads a dataset (and what the method misses), how the deadline was set, what went wrong, and what the company built so the swamp does not refill.

Skills applied: the seven practices (§9.7); lake layout and ownership (§9.3); the catalog as minimum viable interface (§9.8); governance as an undercurrent (Chapter 2 §2.7).

Background

The company. Roughly 400 employees, a data platform team of six, a lake on object storage that had grown organically since 2022.

How it got there is the ordinary way, and no step was wrong:

• 2022. One team lands three datasets. Everything is known by everyone.
• 2023. Self-service is encouraged — a good decision, and correct for the growth stage. Any team can land data. Fifteen teams do. Roughly 60 datasets.
• 2024. Two acquisitions bring their own data. A machine learning team creates feature datasets. A partner integration lands external data. Roughly 140.
• 2025. Roughly 200, and the platform team can name perhaps 40 of them.

The trigger was a GDPR erasure request. Legal asked the platform team to confirm that a specific customer's data had been deleted everywhere. The team could not, because they could not enumerate where personal data lived. The honest answer — "we do not know what most of these datasets contain" — was not acceptable, and it was true.

The Problem

The audit script was straightforward. Interpreting its output was not.

"""Inventory every dataset in the lake. Deliberately dumb: it reports what is
there, not what it means, because what it means is exactly the thing missing."""
for prefix in top_level_prefixes(bucket):
    yield {
        "dataset":      prefix,
        "objects":      count_objects(prefix),
        "bytes":        total_bytes(prefix),
        "avg_file_kb":  total_bytes(prefix) / max(count_objects(prefix), 1) / 1e3,
        "newest_write": max_last_modified(prefix),
        "oldest_write": min_last_modified(prefix),
        "has_schema":   exists(f"{prefix}/_schema.json"),
        "in_catalog":   prefix in glue_catalog_tables(),
        "owner_tag":    bucket_tags(prefix).get("owner"),
    }

The results:

"Is anything reading it?" was the hard question, and this is the part worth studying.

Determining whether a dataset is read

Four methods, none of them complete:

They used all four and treated the union as "possibly read."

The gap that mattered: 30 days of access logs cannot see a quarterly job. A dataset read once a quarter looks identical to a dataset read never, and the company had at least four quarterly processes.

⚠️ Failure Mode — "Nothing is reading it" is a claim about your observation window

A 30-day access log answers "was this read in the last 30 days," which is not the question. Anything on a quarterly, semi-annual, or annual cadence is invisible, and those are precisely the processes nobody remembers — a compliance report, a board deck input, a tax calculation, an annual vendor reconciliation.

Three ways to close the gap, and the third is the one that works:

1. Extend log retention to at least 400 days before you audit. Correct, and it means waiting over a year to start.
2. Ask teams. Necessary and unreliable — people do not remember what they built in 2023.
3. Archive rather than delete, and make the failure loud. Move the dataset to an archive prefix and revoke access. A quarterly job that needs it fails, loudly, with a clear error, and someone reclaims it. You have converted an unanswerable question into a cheap, self-resolving failure.

That third option is the same move as Chapter 2's Case Study 2 (zz_deprecated_ rename) and the same principle as Chapter 2's "fail loudly, not plausibly." When you cannot determine whether something is needed, arrange for its absence to announce itself.

The Analysis

The platform team proposed deletion. Leadership refused, correctly — the risk of deleting something load-bearing was unbounded and unquantifiable.

The counter-proposal, which was accepted:

A claim period with a deadline and a reversible consequence.

1. Every dataset gets a page in a shared document: name, size, last write, and three blanks — owner, description, consumers.
2. Fourteen weeks to fill in the blanks. Announced in an all-hands, in writing, and in a weekly reminder naming the specific unclaimed datasets per team.
3. Unclaimed datasets are archived, not deleted: moved to s3://company-archive/, access revoked, storage class set to Glacier Instant Retrieval, retained for two years.
4. Reclaiming takes one ticket and about an hour.

📐 Design Decision — Archive, not delete; deadline, not exhortation

Two choices, each with a rejected alternative.

Archive rather than delete. Rejected: deletion, which would have been cheaper and faster and carried unbounded risk. Archiving to Glacier Instant Retrieval at $0.004/GB-month costs approximately nothing for two years and makes every mistake reversible in an hour. What it costs: the datasets are still there, so the "how much data do we have" number does not improve, and there is a temptation to never finish the process.

A deadline with a consequence rather than a request. Rejected: asking teams to document their datasets, which had been tried twice before and produced 11 and 6 entries respectively. A catalog nobody is required to fill in stays empty, and this is not a statement about those teams — it is a statement about how work gets prioritized when it has no deadline.

What the deadline costs: it is a political instrument, it consumes goodwill, and it can only be used occasionally. The team's own view is that they spent most of a year's credibility on it and would not have been able to run a second one for eighteen months.

The reason it worked: the consequence was real, reversible, and cheap to undo. A deadline whose consequence is severe gets negotiated away; one whose consequence is trivial gets ignored. An hour's inconvenience is the sweet spot.

The Decision

Fourteen weeks. Weekly emails per team listing their unclaimed datasets by name.

Results:

The 33 voluntary deletions are the most interesting number. Teams asked to document a dataset frequently discovered it was a duplicate of another one, or superseded, and deleted it themselves. The documentation requirement did more work than the archive threat.

Post-archive complaints: two. Both for the same dataset, an events export that a marketing analyst used monthly. It was reclaimed in an hour. On investigation, they had been reading a copy of it that a different team maintained, and the original had been unread for two years.

What Happened

Immediately. Storage dropped from 84 TB to 31 TB active plus 53 TB archived. The active-storage bill fell by about $1,220 a month, which was not the point and did not hurt.

The erasure request was answerable six weeks later, once the remaining 71 datasets had schemas and PII classification. It took two days rather than being impossible.

Eighteen months on, the lake has 94 datasets. It grew from 71, and every new one has an owner, because of the four controls the company adopted alongside the cleanup:

1. Ownership is a precondition for creation. Landing to a new prefix requires a catalog entry with an owner and a description. Enforced by a bucket policy that denies writes to prefixes not registered in the catalog — enforced, not requested, which is the whole lesson.

2. Quarterly unowned-dataset report. Automated, sent to engineering leadership. Currently zero.

3. Access logging retained for 400 days, so the next audit can see annual jobs.

4. An archive-first policy. Nothing is ever deleted directly; everything goes to archive for two years first.

🔐 Privacy & Governance — Why the erasure request was the trigger, and why that is typical

The audit happened because of a legal request, not because of engineering conviction. That is the normal path and it is worth being honest about, because it tells you something about how to get this work funded.

The engineering argument for a catalog is diffuse. Nobody is blocked today. It is slow onboarding, duplicated effort, and confusion — all real, none urgent, and none with a date attached.

The compliance argument has a statutory clock. GDPR Article 12 gives a controller one month to respond to an erasure request, extendable by two further months for complex requests. That is not a target. When the honest answer is "we cannot enumerate where this person's data is," the problem acquires a deadline and a sponsor.

Two practical consequences:

• If you want catalog work funded, the privacy framing is usually the one that lands. This is not cynicism; the obligation is genuine and the deadline is genuine.
• Do the work before the request arrives. The company had two days of slack in a one-month window. A larger lake, or a request arriving mid-audit, would have consumed it.

This book is not legal advice; confirm your obligations with counsel. Chapter 31 has the mechanics.

Lessons

1. A swamp is a lake without ownership, and it forms one reasonable decision at a time. Nobody made a mistake. Self-service was correct for the growth stage.

2. "Nothing is reading it" is a claim about your observation window. Thirty days of logs cannot see a quarterly job, and quarterly jobs are exactly the forgotten ones.

3. When you cannot determine whether something is needed, arrange for its absence to announce itself. Archive with access revoked, and let a failure reclaim it.

4. Archive rather than delete. Glacier Instant Retrieval costs approximately nothing and makes every mistake reversible in an hour.

5. A deadline with a cheap, reversible consequence works; a request does not. Two prior documentation drives produced 11 and 6 entries. The deadline produced 104.

6. The documentation requirement did more than the archive threat. 33 teams deleted their own datasets on discovering they were duplicates.

7. Make ownership a precondition for creation, enforced by policy. A bucket policy denying writes to unregistered prefixes is the control that stopped the lake refilling.

8. A deadline is a political instrument with limited supply. The team spent most of a year's credibility and could not have run a second one for eighteen months.

Questions for Discussion

1. The audit was triggered by a legal request rather than by engineering conviction. Write the business case you would have made before the request, and assess honestly whether it would have been funded.

2. Four methods were used to determine whether a dataset was read, and none was complete. Design a fifth. What would it cost, and would it have changed the outcome?

3. Fourteen weeks was chosen as the deadline. Argue for six weeks and for six months. What does each optimize for, and what does the right answer depend on?

4. Control 1 denies writes to unregistered prefixes. What does this cost a team that wants to land something quickly during an incident? Design the exception path.

5. Two complaints arrived, both for a dataset whose complainants were actually reading a copy. What does that suggest about the 96 archived datasets, and how would you find out without unarchiving them?

6. The company now has 94 datasets, up from 71. Is growth from 71 to 94 healthy or a warning? What would you measure to tell the difference?

7. This case study and Chapter 2's Case Study 2 both describe deprecation by making absence loud — zz_deprecated_ renames and archive prefixes. Where else in a data platform does that technique apply, and where would it be dangerous?