Key Takeaways: Migrating Legacy Pipelines

The one thing

A migration produces no new features and succeeds when nothing changes — which makes it the work most data engineers are hired to do, the hardest to fund, and the easiest to interrupt.

Why it is hard, and it is not the code

Nobody knows what it does. No documentation, author gone, the only specification is the output.

Nobody knows who uses it. And Chapter 25's usage measurement does not reach a file share.

The old system is wrong in ways people have adapted to. Correction factors, "ignore Tuesdays." Fixing the bug breaks the adaptation.

There is no feature to point at. The benefit is "the same numbers, from a system we can change."

Inventory before you read anything

🔎 Eight fields, all discoverable without reading a line: kind · consumers · owner · tests · size · schedule · undocumented dependencies · criticality.

Kestrel, in three weeks: 10 of 12 with no owner, 11 of 12 with no tests, 36 undocumented dependencies. These three numbers are also the most persuasive thing a proposal can lead with.

Finding consumers takes four methods — grep · access history · instrumented reads · breaking it on purpose — and only the fourth finds everybody.

⚠️ Instrumentation finds systems and misses people. Seven of nine of Kestrel's consumers were a person opening a file. It also misses periodic consumers (a week of monitoring sees a monthly macro 23% of the time, and periodic consumers are disproportionately finance), and cannot distinguish two purposes behind one identity.

🔎 The announced, bounded rename is a census, and the README is what makes it one. It says the absence is deliberate, gives one place to reply, asks "what do you use it for?" rather than "are you using this?", and restores in one command.

⚠️ The indefensible version: silent · on something critical · at month-end · repeatedly · without agreement. Check the finance calendar — thirty seconds, and it avoids the one genuinely damaging outcome.

The four kinds

⚠️ The load-bearing spreadsheet is not a technical migration. Separate rules from decisions, migrate only the rules, and give the decisions an explicit versioned input with a comment field. A manual step that survives is the correct home for a judgment — Kestrel's first rebuild matched history perfectly and was wrong its first live month, because that month's judgment adjustments had no equivalent. The owner is not the blocker; the owner is the asset.

The order

📐 (reach × risk) / difficulty — learning per week.

Not biggest-first: the hardest job attempted with the least knowledge, a quarter gone, nothing visible at the first review. Not easiest-first: you learn nothing that transfers and the second job surprises you exactly as much.

High risk is a feature of the first job, because that is where the surprises are and you want them while there is schedule left.

Kestrel's first job: 80 lines, six consumers, business-critical, no owner, no tests — a week that touched every mechanism the rest would need.

Strangler fig

Build to a different output · run both · reconcile daily · swap consumers when the rule passes · leave the legacy job running unread for one cycle · then delete it.

Step 6 is the one that does not happen. A job left running "just in case" is still running in three years, costing money, holding credentials, appearing in every audit. Delete, do not disable.

Big-bang is defensible only when running both is impossible.

Reconciling

Decide grain, tolerance, and window before the first comparison. Zero is the right tolerance default for counts and integer money.

Use a FULL OUTER JOIN. An inner join is blind to rows one system produced and the other did not — the most common difference, and the one that hides.

When the legacy system is the one that is wrong

⚠️ 196 legacy bugs against 146 new ones. Nothing measured the old job for years, so its errors accumulated. Nobody plans for this, because the migration was approved on the premise that nothing changes.

Each one is a conversation, not a fix. Match the old number, ship the correct one, fix both, or block?

Bug-for-bug compatibility is legitimate and a terrible default — it ships a deliberate error with a promise to fix it later that has no forcing function after cutover.

📐 Emit both numbers. It unblocks the engineering, makes the impact visible rather than described, removes the deadline from the decision, and converts an argument into an observation. Set the removal date in the same ticket as the cutover.

Decide the policy in advance: a materiality threshold, who decides above it, and that every difference is documented with a date.

The floor

📏 50.2% of differences are irreducible — rounding, timezone boundaries, legacy non-determinism (18.5%, the largest), and genuine source changes.

Legacy non-determinism cannot be matched, only reconciled against a property: totals, distributions, row counts.

So a cutover rule requiring zero differences never fires. Require only accepted kinds, all documented — and getting that right in advance prevents a migration that is technically complete and organizationally stuck at 99.6%.

Differences must decay. 238 in week one, 17 in the last. If they do not, you are not converging.

The job you cannot read

🏭 Ask what it is for. One Informatica workflow was 40% dead code — three branches for a supplier retired in 2021, a currency for an exited market, a "temporary" 2019 reprocessing step. None findable by reading it; all findable by asking.

A migration is the only time anybody asks what a pipeline is supposed to do. Budget for that discovery. The right first question is not "how does this work?" but "what is this for, and is that still true?"

Cutover

📐 The rehearsed rollback is worth more than every other readiness item and is always deferred, because rehearsing costs a day and everything else is a checkbox.

Kestrel's rehearsal found: credentials expired on a job idle 23 days · two consumers repointed somewhere the plan did not mention · 3h40 against a claimed "under an hour." All three would have been found at 2am by someone who did not write the plan.

And the ratchet: having measured 3h40, they kept the legacy job running through the overlap, taking the rehearsed rollback under 20 minutes.

The cost

💸 The engineering line exceeds the compute line and nobody budgets it. $14,112 of difference-triage against $8,790 of double-running.

The overlap ran 2.45× the plan — 147 days against 60 — almost entirely waiting for business decisions, not for engineering.

The overrun takes 579 days of steady-state savings to repay. So do not justify a migration on cost; it invites a comparison you lose. Justify it on jobs that have owners, tests, lineage, and can be changed by someone who joined last month, and footnote the saving honestly as small.

Not everything should be migrated

📐 Kestrel: 8 rebuilt, 2 deleted, 1 absorbed into an existing model, 1 left alone.

"Legacy" is not a synonym for "old." A system is legacy when it cannot be safely changed — a property of your relationship with it, not of its age.

The test: what specifically will we be able to do afterwards that we cannot do now? For one of Kestrel's twelve the answer was nothing, and it is still running — as a recorded decision rather than an oversight, which is what makes it a decision.

📐 Migrating a pipeline and migrating its consumers are separate projects. Preserving a CSV as a declared output port killed the legacy job without four spreadsheet conversations. Bundling them turns a three-day job into a quarter.

Decommissioning

A migration is finished when the old thing is gone, and four things must actually be removed: the job (deleted, not disabled) · the code · the credentials (Chapter 30's unattributable identity) · the infrastructure (Chapter 33's unattributed spend).

Set the date at cutover, in a ticket, with an owner. Kestrel closed 11 of 14 in-quarter; the three that were not are the ones that would have become permanent.

The politics

🎓 Publish the inventory early · deliver visibly in week one · report the differences as findings, not problems · and never promise that nothing will change.

It will not be true by week two, and having promised it turns each of 196 findings into a negotiation about your credibility rather than about the number.

Three weeks building, eleven weeks deciding, on the job whose output was a reported number. Budget more time for the conversation than for the code.

The code

code/migration_lab.py — an estate scored and ordered by reach, risk, and difficulty; sixty days of shadow differences classified into six kinds with verdicts; a weighted cutover rule; and an overlap cost model including the engineering line. Thirty-eight self-checks, including that the recommended first job is neither the biggest nor the easiest.