Case Study 1 — The Weekend That Cost a Quarter of the Data

A five-bay RAID 5 in a creative agency dropped one disk on a Friday. By Monday the array was offline and a third of the active projects were gone — not because the disks died, but because of what well-meaning people did to "fix" them. This is the chapter's central lesson as a cautionary tale: the rebuild, not the failed disk, is what kills arrays.

Background

Brightline, a twelve-person marketing and video agency, ran everything off a five-bay NAS: a RAID 5 across five 4 TB drives, left-symmetric layout, 128 KB strip, roughly 16 TB usable. It held client video masters, project files, and a decade of finished work. There was a backup — a single external drive someone was supposed to rotate weekly — but it had last been connected eleven weeks earlier and held nothing from the three campaigns currently in production.

On a Friday afternoon the NAS posted "Volume degraded — Disk 3 failed." Reads and writes continued normally; nobody noticed any slowdown. The office manager bought an identical 4 TB drive, and the most technically confident person in the office slotted it into bay 3 around 5 p.m. and clicked Repair, then left for the weekend with the rebuild running.

What happened

A RAID 5 rebuild must read every sector of all four surviving disks and XOR them to recompute the failed disk's contents onto the replacement. These five drives were the same make, the same batch, bought together, and had run in the same warm enclosure for four years. Six hours into the full-surface read, Disk 1 — which had not been asked to read its outer tracks in over a year — hit a cluster of unrecoverable read errors. With Disk 3 already absent and Disk 1 now unreadable for those stripes, the controller had two members unavailable for the same rows. It aborted the rebuild and marked the volume failed/offline.

Monday morning, the array was dark. Following a forum thread, the same staffer opened the controller BIOS and, hoping it would "re-detect the disks," chose Initialize / Create New Array with the four original disks plus the replacement. The controller wrote fresh DDF metadata to the end of each member and began a background initialization that zeroed parity — and, in this controller's implementation, zeroed data strips — across the first stretch of the array before someone realized the projects were still missing and pulled the power. Then they called us.

By the time the drives reached the bench, three destructive things had already happened on the original disks: a rebuild had run against them, the controller had overwritten metadata, and an initialization had zeroed the leading region of the array. Only the first of those — the rebuild — wrote solely to the replacement. The other two touched the originals we needed.

We did what should have been done on Friday: imaged all six drives (five originals plus the replacement) behind a write-blocker with ddrescue -d -r3, hashing each image and recording a bad-sector map.

member1.img  (Disk 1)  217 unreadable sectors   sha256: 4f9c…a1e2
member2.img  (Disk 2)  clean                     sha256: b73d…0c55
member3.img  (Disk 3)  clean (the "failed" disk — only kicked by a timeout)
member4.img  (Disk 4)  clean
member5.img  (Disk 5)  clean
replacement.img        excluded — holds only partial rebuild output

The cruel irony surfaced immediately: Disk 3, the disk the controller had declared "failed," was 100% readable. It had merely been kicked from the array by an I/O timeout, not physically died. Had anyone imaged it on Friday and reconstructed virtually, the recovery would have been complete — five good members, one of them with a few latent bad sectors that parity would have covered.

But the weekend's "fixes" had moved the goalposts. The controller's initialization had zeroed parity and overwritten data strips across the first ~9% of the array — exactly where the NTFS $MFT and a swath of the current-project directory lived. We reconstructed from the freshest original members with the correct parameters (read from a surviving superblock region: raid5, five devices, left-symmetric, 128 KB chunk, the original data offset), excluding the replacement entirely. Parity cleanly filled Disk 1's 217 bad sectors everywhere they did not overlap the zeroed region. The result:

Reconstruction summary
  Members used .......... 1,2,3,4,5 (originals); replacement excluded
  Geometry .............. RAID5, left-symmetric, 128 KB strip, 5 disks
  Recoverable ........... ~76% of files, intact and hash-verifiable
  Damaged/lost .......... files in the zeroed leading region, incl. two of
                          the three active campaigns' master timelines
  Cause of loss ......... controller Initialize zeroing data+parity on
                          originals (NOT the original disk failure)

Three quarters of a decade of work came back perfectly. The quarter that did not included the project the agency most needed — and none of it had been lost when Disk 3 dropped on Friday.

The economics stung as much as the loss. The image-first virtual reconstruction we would have performed on Friday — five members imaged, parameters read, volume assembled and verified — would have been a routine, near-certain, full recovery at a predictable price. Instead the agency paid more for a partial result, re-shot one campaign from scratch on an emergency timeline, and renegotiated a delivery date with a client who never learned why. The eleven-week-old external drive, meanwhile, had cost almost nothing and would have made the entire crisis a footnote had it simply been rotated weekly and verified. The cheapest insurance in the building had been allowed to lapse precisely because RAID had worked so flawlessly for four years that a real backup felt redundant — the exact trap the chapter warns about. We left Brightline with two artifacts: the recovered data on fresh media, and a one-page standing instruction taped to the NAS that begins with a single word in bold — Stop.

The analysis

  1. RAID is redundancy, not a backup — and "degraded" means zero redundancy. Brightline ran for four years feeling safe because RAID 5 silently tolerated the daily risk. The day Disk 3 dropped, the array had no margin left, yet the business kept writing to it as if nothing had changed. The eleven-week-old external drive was the only real backup, and it held none of the work that mattered.

  2. The rebuild is the moment of maximum danger, not normal operation. Latent sector errors hide for months in regions nobody reads; a rebuild forces the first full-surface read in a year across same-batch disks under sustained load. That is precisely when a marginal sibling fails. The failed disk did not kill the array — the rebuild did, exactly as the chapter's degraded-rebuild cascade describes.

  3. Image first, always — including the "failed" member. Disk 3 was perfectly readable; the controller's "failed" label was a timeout, not a death certificate. A single Friday-afternoon imaging pass of all five members would have turned this into a textbook virtual reconstruction with full recovery. Every later disaster flowed from acting on originals instead of copies.

  4. Controller "Initialize / Create" can be irreversibly destructive. "Repair," "Rebuild," "Initialize," and "Create New Array" sound reassuring and reversible; they are not. Initialization zeroed parity and data on the original disks — the one action in the whole sequence from which there is no recovery. When in doubt, power off; never click a controller action you cannot undo.

  5. The expensive losses were all self-inflicted after the failure. The original fault was a fully recoverable single-disk degrade. Everything unrecoverable — the zeroed $MFT region, the two lost campaign masters — was created by the rebuild and the initialization. The technical failure was trivial; the human response was the catastrophe (Theme 6).

Discussion questions

  1. Trace the timeline and identify the exact action at which Brightline's situation changed from fully recoverable to partially unrecoverable. Was it the disk failure, the rebuild, or the initialization — and why?

  2. The staffer "was just trying to help." Write the one-page standing instruction you would post next to the NAS so that the correct response to a degraded array — essentially, "stop and call" — is the obvious one. How do you make "do nothing" feel like the responsible action?

  3. ⭐ Suppose Brightline's array had been subject to a litigation hold at the time of failure. Which of the weekend's actions could expose the company to a spoliation finding, and how would you, as the examiner later retained, document what was altered and when (Chapter 25)?

  4. Estimate the cost-benefit the agency faced: a professional image-first recovery quoted on Friday versus the value of the two lost campaign masters plus the emergency reconstruction. What does this say about pricing prevention (a real backup) against the cost of a bad week?

  5. Draft the two-paragraph incident note you would hand Brightline explaining, in plain language, what was recovered, what was lost, and why the losses occurred after the original failure — without blaming the staffer so harshly that the real lesson (image first, never rebuild on originals) is lost.