Case Study 1 — The Rebuild That Almost Finished the Job

A small firm trusted its RAID 5 array to keep them safe, skipped backups because "we have redundancy," and then watched a routine disk replacement cascade into near-total loss. This is a RAID recovery worked the right way — image every member, reconstruct the geometry virtually — and a clean lesson that redundancy is not a backup, and that the most dangerous moment for an array is the rebuild itself.

Background

A seven-person architecture firm ran its entire practice — drawings, models, renderings, fifteen years of project archives — off a four-bay NAS configured as a single RAID 5 volume of four 4 TB disks (about 12 TB usable). It had run flawlessly for five years, which was precisely the problem: it had run so flawlessly that the firm had quietly stopped doing anything else. The external backup drive someone used to rotate offsite had died eighteen months earlier and was never replaced. "We don't really need it," the office manager had reasoned. "The NAS has RAID — if a drive dies, it just keeps working."

One Tuesday a drive did die. Disk 2 dropped out and the NAS chimed its degraded-array alarm. This is RAID 5 working exactly as designed: one disk gone, the array still serving every file by computing the missing blocks from parity. The firm had a window — days or weeks — to image the array calmly and replace the disk. Instead, the office manager did the intuitive thing. She bought an identical 4 TB disk, slotted it into bay 2, and clicked Rebuild.

Recovery vs. Forensics. This is a 💾 recovery engagement — the deliverable is the firm's project files, not courtroom evidence. But the recovery succeeds only because it borrows the forensic discipline this book preaches: image every member disk to a verified copy first, then do all reconstruction on the copies. A shop that reconstructs on the original disks is one bad sector away from destroying the client's only data.

The recovery

The rebuild ran for nineteen hours. To recompute the new disk 2 from parity, the controller had to read every sector of all three surviving disks — a brutal, sustained, end-to-end stress test on three other drives that were exactly as old as the one that had just failed. At hour nineteen, with the rebuild at 86%, aging disk 4 hit a sector it could not read — an unrecoverable read error (URE) — and the controller, unable to complete the parity math for that stripe, dropped disk 4 from the array. Now two members were missing. A RAID 5 survives one. The array went offline, and the NAS that had "kept them safe" presented nothing at all.

The firm's files were not gone — but they were now scattered, in pieces, across four disks of varying health, with the array's own controller refusing to assemble them. They brought the four disks to a recovery shop.

The shop's first act was the one the chapter insists on: do not touch the original disks except to image them. Each disk was attached through a hardware write-blocker and imaged individually with dcfldd, producing four files and four SHA-256 hashes. The per-disk imaging told its own story:

  Per-member imaging (illustrative)
  disk 1  ST4000VN008   sha256 4f2a...c1   imaged 100%, 0 bad sectors
  disk 2  ST4000VN008   sha256 9b7e...44   imaged 100%  <- the "first failure"
  disk 3  ST4000VN008   sha256 1d80...a9   imaged 100%, 0 bad sectors
  disk 4  ST4000VN008   sha256 6c33...02   imaged 99.997%, 214 bad sectors

The diagnosis was illuminating. Disk 2 — the original "failure" — turned out to be almost entirely readable; it had dropped not from a head crash but from a marginal firmware glitch that a patient, single-pass image read straight through. Disk 4, by contrast, the one that died during the rebuild, had genuine media defects: 214 unreadable sectors, exactly the kind of latent bad sector that lurks unnoticed until a full-surface rebuild drags a read head across it. The shop now had complete images of disks 1, 2, and 3, and a 99.997%-complete image of disk 4.

Next, reconstruct the geometry virtually. A RAID volume is not self-describing, so the shop recovered the five parameters that turn a pile of member images back into a readable volume. Scanning each image, they found the Linux md superblock — the NAS ran mdadm software RAID underneath — whose magic number confirmed the implementation and yielded the layout:

0x001000:  fc 4e 2b a9 01 00 00 00  ...   md superblock magic 0xa92b4efc
           ^^^^^^^^^^^                     (stored little-endian: FC 4E 2B A9)
  -> level 5, 4 devices, chunk (stripe) size 64 KB,
     layout left-symmetric, data offset 2048 sectors

With disk order, stripe size (64 KB), parity rotation (left-symmetric), data offset, and level all read directly from the metadata, the shop assembled the array read-only from the four images — never from the original disks — using mdadm --assemble against loop-mounted copies, then mounted the resulting ext4 file system read-only.

The volume came back. Because disks 1, 2, and 3 were complete and disk 4 was 99.997% intact, almost every stripe had at least three of its four blocks present — enough for parity to reconstruct the rest. The firm recovered roughly 98% of its data: the entire active project set, the full client archive, the renderings. The losses were narrow and specific, and they mark exactly the limit of RAID 5's math.

Limitation. The 2% that was lost lived in double-fault stripes — stripes where disk 4's unreadable sectors fell in the same stripe rows that also needed a block from the originally-failed member, leaving two blocks missing from one stripe. RAID 5 parity reconstructs one missing block per stripe; two is one too many, and those bytes were genuinely, mathematically gone. A RAID 6, with its second independent parity, would have reconstructed every one of them.

The analysis

  1. RAID is not a backup. Redundancy protects against disk failure and nothing else — not deletion, corruption, ransomware, fire, theft, or human error. The firm's reasoning ("we have RAID, we don't need backups") confused fault tolerance for data protection, and it is the single most common belief that turns into a recovery case.

  2. The rebuild is the most dangerous moment in an array's life. Replacing a failed disk forces the controller to read every sector of every survivor — a full-surface stress test that routinely surfaces a latent bad sector and triggers a second failure. On large disks the cumulative odds of a URE across a full rebuild are high enough that single-parity arrays now fail in exactly the moment their one redundancy is being spent.

  3. Never rebuild on the original hardware. The instinctive click of "Rebuild" wrote to the array and stressed the survivors. Imaging each member individually first, then reconstructing virtually, would have preserved every disk in the state it was in at the moment of the first failure — and disk 4's 214 bad sectors might never have mattered.

  4. A RAID volume must be reverse-engineered from five parameters. Disk order, stripe size, parity rotation/direction, data offset, and the level itself are what reassembly needs. Here the mdadm superblock (magic 0xa92b4efc) handed them over; when metadata is gone, you reverse them from the data itself.

  5. Know the exact boundary of the loss. The recovery wasn't "98%, roughly." It was "every stripe with at most one missing block, reconstructed; every double-fault stripe, lost" — a precise, explainable limit. Naming it honestly is the difference between a professional report and a vague apology.

Discussion questions

  1. The firm believed "we have RAID, so we're protected." State precisely what RAID 5 does protect against and the full list of disasters it does not. Why does the marketing word "redundancy" so reliably mislead non-specialists?

  2. The moment disk 2 dropped, the office manager clicked Rebuild. Describe what she should have done instead, in order, and explain why imaging the degraded array before replacing anything would have changed the outcome.

  3. Disk 2 — the original "failure" — imaged at 100%, while disk 4 — which died during the rebuild — had real defects. What does this reveal about why hardware rebuilds are so dangerous, and about not trusting a controller's verdict on which disk "failed"?

  4. ⭐ The final losses were confined to "double-fault stripes." Using the parity property P = D1 ⊕ D2 ⊕ D3, explain exactly why a single stripe missing two of its blocks cannot be reconstructed in RAID 5, and show how RAID 6's second (Reed-Solomon) parity would have recovered those same stripes. What does this cost in capacity and write performance?

  5. This is a hardware-failure story, but compare it to the ransomware anchor case (Chapter 12), where a business "had RAID" and watched malware encrypt every member at once. Different disaster, identical false comfort. What is the single piece of advice that would have saved both firms, and why does RAID specifically encourage people to skip it?