Quiz: Migrating Legacy Pipelines

Twelve questions. Answers with explanations follow — work through them first.

1. §37.2 says the first instinct — reading the legacy code — is the wrong first move. Why?

• A. The code is usually in a language you do not know
• B. It takes weeks and leaves you knowing what the code does but not who depends on it or which parts matter
• C. Legacy code is usually obfuscated
• D. The code may not be the current version

2. Kestrel's inventory found which of these?

• A. 2 of 12 jobs with no owner
• B. 10 of 12 with no owner and 11 of 12 with no tests
• C. All 12 with owners but no tests
• D. 6 of 12 with no owner

3. §37.4 orders jobs by (reach × risk) / difficulty. What does that optimize for?

• A. Total business value delivered
• B. Shortest total migration time
• C. Learning per week — surprises early, while there is schedule to absorb them
• D. Lowest risk of an outage

4. Why is "biggest first" a bad migration order?

• A. Big jobs are usually less important
• B. The hardest job is attempted with the least knowledge of the estate, takes a quarter, and produces nothing visible by the first review
• C. It uses too much compute
• D. Large jobs cannot be shadow-run

5. §37.6 insists the reconciliation query use a FULL OUTER JOIN. What does an inner join miss?

• A. Rows with null keys
• B. Rows one system produced and the other did not — the most common difference
• C. Duplicate rows
• D. Type mismatches

6. Sixty days of shadow running produced 687 differences: 196 legacy bugs and 146 new-pipeline bugs. What does §37.7 say about this?

• A. It indicates the new pipeline was built carelessly
• B. It is unusual and suggests a bad legacy system
• C. It is expected — nothing was measuring the legacy job for years, so its errors accumulated — and you must decide in advance what to do about it, because each one is political
• D. Legacy bugs should be ignored

7. §37.7's ⚠️ callout calls bug-for-bug compatibility:

• A. Always wrong
• B. The correct default
• C. A legitimate choice and a terrible default — the promise to fix it later will not be kept
• D. Only acceptable for financial systems

8. 50.2% of Kestrel's differences were irreducible — rounding, boundaries, legacy non-determinism, and source changes. What follows for the cutover rule?

• A. The migration should be abandoned
• B. A rule requiring zero differences never fires; require only accepted kinds, all documented
• C. Tolerances should be widened until differences disappear
• D. Non-determinism should be eliminated first

9. §37.9 says one readiness item is worth more than all the others and is always deferred. Which?

• A. Consumers have been told, with a date
• B. The pipeline has run green for 14 days
• C. A rollback has been rehearsed, not just written down
• D. Every accepted difference is documented

10. §37.10: the planned 60-day overlap ran 147 days, at 2.45× the cost. What drove the overrun?

• A. Compute costs rose
• B. The rebuild took longer than expected
• C. Waiting for business decisions about corrected numbers — the engineering was done and the decisions were not
• D. Additional consumers were discovered

11. In Case Study 2, three discovery methods agreed there were two consumers. There were nine. What is the general lesson?

• A. Three methods are not enough
• B. Agreement among methods that share an assumption confirms the assumption, not the answer — all three looked for machine access, and seven consumers were people
• C. File shares should not be used
• D. Access logging was misconfigured

12. Case Study 1's stored procedure excluded gift-card orders for nine years because of an operator-precedence bug. Why was it not caught in review?

• A. Nobody reviewed it
• B. It was written correctly and became wrong when gift cards entered the catalog eleven months later — a condition matching nothing is invisible, and only a second implementation finds it
• C. The reviewers did not know SQL
• D. It was in a GUI tool

---

Answers

1 — B. It leaves you knowing the code and not the dependencies.

Twelve jobs at 400 lines is 4,800 lines of undocumented shell and SQL; reading it takes weeks, and at the end you still do not know who consumes each output or which parts matter. Everything in §37.2's inventory — kind, consumers, owner, tests, size, schedule, undocumented dependencies, criticality — is discoverable without reading a line, and it is what the ordering decision actually needs.

2 — B. 10 of 12 with no owner, 11 of 12 with no tests.

Plus 36 undocumented dependencies across the estate. These three numbers are also the most persuasive artifact a migration proposal can lead with (§37.13): they are just true, they took three weeks, and they reframe the migration as a response to a measured problem rather than as an engineering preference.

3 — C. Learning per week.

High reach means the job matters, so migrating it is worth something on its own. High risk means it has undocumented dependencies, no owner, no tests — which is where the surprises live, and you want them early, while there is schedule to absorb them. Low difficulty means you find out soon. Kestrel's first job was 80 lines, business-critical, ownerless, with two hidden dependencies — a week of work touching every mechanism the rest of the migration would need.

4 — B. The hardest job with the least knowledge, and nothing visible at the first review.

"Easiest first" fails differently and for a related reason: you migrate something nobody cares about, learn nothing that transfers, and the second job surprises you exactly as much as the first would have. Both instinctive orderings optimize the wrong thing in the first month, which is when you know least.

5 — B. Rows one system produced and the other did not.

An inner join compares only the rows both systems produced, which makes it blind to the most common kind of difference — and, per Chapter 36 Case Study 1, the kind that hides. A missing row produces no mismatched value; it produces an absence, and an absence is invisible to a join that requires presence on both sides.

6 — C. Expected, and the decision must be made in advance.

The legacy job ran for years with no tests, no owner, and nothing comparing it to an independent source — Chapter 23's argument applied retroactively, so its errors accumulated undetected. What makes this hard is not technical: each difference is a conversation about whether you are allowed to ship the correct number, on a project approved on the premise that nothing changes. Kestrel decided in advance: under a stated materiality threshold, ship correct and note it; above it, the number's business owner decides while the migration continues, with both values emitted in two columns.

7 — C. Legitimate and a terrible default.

It means shipping a new pipeline that deliberately reproduces a known error, with a comment explaining why, and a promise to fix it later — and after cutover there is no forcing function, so the promise is not kept. It is chosen most often precisely because the migration's mandate was "nothing changes," and a step change in a reported number is what that mandate forbids. The two-column approach converts this blocking decision into a non-blocking one.

8 — B. A rule requiring zero never fires.

Legacy non-determinism alone was 18.5% — a job whose output depends on the row order the query plan happened to produce, which you cannot reproduce and therefore cannot match. Add rounding, timezone boundaries, and genuine source changes and half the differences are permanent. The rule must be "only differences of accepted kinds, all documented", and getting that right in advance is what prevents a migration that is technically complete and organizationally stuck at 99.6%.

9 — C. A rehearsed rollback.

Rehearsing costs a day; every other item is a checkbox you can tick from a dashboard, which is why this one is always deferred. Kestrel's rehearsal found that the legacy job had not run for 23 days and its credentials had expired, that two consumers had been repointed in a place the plan did not mention, and that the rollback took 3h40 against a claimed "under an hour." All three would otherwise have been discovered during an incident, at 2am, by someone who did not write the plan — and having measured 3h40, the team kept the legacy job running through the overlap, which brought the rehearsed rollback under 20 minutes.

10 — C. Waiting for business decisions.

The 87 extra days were almost entirely §37.7's conversations. The compute overrun was real and secondary; the engineering line — about four engineer-days a month triaging differences, $14,112 over 147 days — exceeded all the compute at $8,790, and is the line nobody budgets. The overrun alone takes 579 days of steady-state savings to repay, which is why a migration justified on cost invites a comparison it loses.

11 — B. Agreement among methods sharing an assumption.

Grep, warehouse access history, and file-share timestamps all look for a machine reading a thing. Seven of nine consumers were a person opening a file: an analyst's notebook, a monthly finance macro, a hand-compiled supplier report, and four linked spreadsheets. The methods also miss periodic consumers — a week of monitoring sees a monthly job 23% of the time — and cannot distinguish two purposes behind one identity, which is how two people remoting into one reporting VM appeared as a single service account.

12 — B. It was correct when written and became wrong when the data changed.

AND ... AND ... OR ol.sku LIKE 'GC-%' parses with the OR binding loosest, leaving an unqualified clause that matches a gift-card line on any order. In 2016 gift cards were sold through a separate system and the clause matched nothing; it became live eleven months later. A condition that matches nothing today is not dead code — it is a trap armed for the day the data changes, and it is invisible to review because it does nothing. The only thing that finds it is a second implementation, which is what a migration is. Note also why it survived nine years: nothing reconciled against an independent source, and everybody's baseline was the legacy number, so every report was internally consistent forever.