Further Reading: Migrating Legacy Pipelines

Sources are tagged Tier 1 (confident it exists, recommended without reservation) or Tier 2 (real and worth seeking, but confirm the current edition, version, or URL yourself).

A warning about this literature, and it is the opposite of most chapters'. There is very little written about data pipeline migration specifically, and a great deal written about legacy software migration, most of which transfers directly. The best material in this chapter's territory has "code" in the title, not "data" — and the data-specific material is mostly vendor migration guides, which are about moving to their product and are silent on §37.7's entire subject.

The software-migration books, which transfer almost completely

Michael Feathers, Working Effectively with Legacy Code (Prentice Hall, 2004). Twenty years old and still the best book on this. Its definition of legacy — code without tests — is §37.12's argument, and its central technique (find a seam, get it under test, then change it) is exactly what shadow running does for a pipeline. Read the first four chapters even if you never write another line of application code. Tier 1.
Martin Fowler on the Strangler Fig Application (2004). §37.5, from the person who named it. Short, free, and the naming is the value — it gives you a word for "grow the new one around the old one" that a stakeholder immediately understands. Tier 1.
Sam Newman, Monolith to Microservices (O'Reilly, 2019). The most practical modern treatment of incremental migration. The chapters on the strangler fig, parallel run, and — critically — verification are directly applicable; his "parallel run" section is §37.6 and §37.7 with different nouns, and he is honest about the cost. Tier 1.
Anything careful on "characterization tests." Feathers's term for a test that documents what the system currently does, correct or not, so you can change it safely. A shadow reconciliation is a characterization test with a live system as the oracle, and knowing the name helps you explain why matching a bug is sometimes right. Tier 1.

On the parallel run specifically

The GitHub Engineering "Scientist" library and its write-up (2016). A small library for running old and new code paths in production, comparing results, and reporting mismatches — §37.6 as a library, in Ruby with ports to most languages. The design decisions in the write-up are the useful part, particularly what it does when the two paths disagree and how it samples. Tier 1.
Anything on "dark launching" and "shadow traffic." The service-side equivalent, better documented than the data-side one, and the operational concerns transfer — sampling rate, cost, and what to do with the comparison output. Tier 2 — scattered across engineering blogs.
Chapter 27 §27.9 and Chapter 32 §32.7 of this book. Shadow running as a deployment technique, and the row-level diff. §32.7's insistence on comparing rows rather than distributions is §37.6's requirement, arrived at from a different direction.

On the parts that are not code

Anything sober on organizational change and technical projects that produce no features. The migration's real difficulty is §37.13's, and the software literature is weak here. Kotter's change material and the general project-management literature on "invisible work" are more useful than anything in the data field. Tier 2 — a large literature; you need the framing, not the method.
Fred Brooks, The Mythical Man-Month, on the second-system effect. Directly relevant and usually cited backwards. The warning is that a rewrite accumulates every feature the first system's authors wished for — which is exactly what §37.8's 40%-dead-code finding avoids by specifying from the requirement rather than from the implementation. Tier 1.
Case Study 1 and Chapter 33 §33.12 of this book. Delivering a finding about somebody else's numbers. The two-column technique is §33.12's "bring the alternative, not the problem" in its most literal form.

On finding your consumers

Chapter 30 Case Study 2 and Chapter 25 §25.12 of this book. Query-log analysis and reviewing against usage. Both find systems, which is Case Study 2's whole limitation.
Your warehouse's access history, again: Snowflake ACCESS_HISTORY, BigQuery Data Access logs, Postgres pg_stat_statements. The best available non-destructive census and it stops at the boundary of the warehouse. Tier 1.
Anything on shadow IT discovery. The information-systems literature has thirty years on finding systems nobody registered, and its consistent finding — that self-identification beats detection — is Case Study 2's header-comment fix. Tier 2.

On the legacy technologies you will actually meet

Your database's documentation on migrating stored procedures. Postgres, SQL Server, and Oracle all have official guidance, most of it about moving between databases and much of it applicable to moving out. The useful sections are on extracting the SQL a procedure executes, which is §37.8's first approach. Tier 1 — vendor-specific.
Anything on reading the SQL a GUI ETL tool generates. Informatica, DataStage, SSIS, and Talend all log or can be made to log their generated SQL. This is the single highest-value technique for the hardest category, and it is documented in operations manuals rather than in migration guides. Tier 2 — product documentation.
The literature on spreadsheet errors — Panko's work is the standard reference, and the reported error rates in production spreadsheets are higher than most engineers expect. Useful for §37.3's argument, and useful for approaching a spreadsheet owner with the right posture: the spreadsheet is probably wrong somewhere, and so is the stored procedure you are replacing it with. Tier 2.

Practice

code/migration_lab.py in this chapter. An estate scored and ordered, sixty days of classified shadow differences, a weighted cutover rule, and an overlap cost model. Thirty-eight self-checks. Exercise 37.5 — scoring your own estate — is the hour that changes a plan.
Take one inventory. Exercise 37.2. Five jobs, eight fields, without reading the code. The owner and test counts are usually worse than expected and are the most useful thing you will produce.
Rehearse a rollback. Exercise 37.9. Time it, and compare against what the plan claims.
Ask "what is this for, and is that still true?" of five things you maintain. Exercise 37.14. Two of Kestrel's twelve jobs were deleted rather than migrated on the strength of that question.

A note on what to be skeptical of

Any vendor migration guide. They are about moving to a product and are silent on the two things that actually cost time: finding your consumers, and what to do when the legacy system is wrong.

Any migration plan with no reconciliation ledger. If differences are not being written down with a verdict and a date, they are being argued about repeatedly.

Any estimate that prices only the compute. §37.10: the engineering line was larger, and the overlap ran 2.45× the plan.

Any claim that a migration will change nothing. It will not be true by week two (§37.7), and having claimed it is what turns every finding into a question about your credibility.

And any plan that migrates every job. After this chapter, "twelve jobs, twelve migrations" should prompt one question: which of these should be deleted, absorbed, or left alone?