Case Study 1 — The Race Against the Journal

A single mistyped command erases a directory on a live ext4 server, and the team's first instinct — reboot and see what's missing — would have finished the job the typo started. This is the file system that burns its own map on deletion, recovered against a clock no one can pause: the journal is a ring buffer being overwritten every second the machine runs.

Background

A small software company runs its customer-facing web application on a single Linux server — Ubuntu, ext4 root file system, 4096-byte blocks, 256-byte inodes. The application's source, configuration, and a directory of uploaded customer documents live under /var/www/app. Backups exist, but the most recent good one is nine days old; a misconfigured cron job had been silently failing since.

At 2:14 p.m., a junior engineer cleaning up a staging path runs a deployment script with the wrong argument expanded. Instead of removing a temporary build directory, the script executes the equivalent of rm -rf /var/www/app/uploads against production. Roughly 5,800 files — customer PDFs, images, and a SQLite store — vanish. The web app keeps running on cached objects, so for ninety seconds nobody notices. Then upload links start 404-ing, and the on-call engineer's instinct is the most natural and most dangerous one possible: reboot the box and see what's still there.

Recovery vs. Forensics. This is a pure 💾 recovery engagement — no court, no chain of custody required, the deliverable is the customers' files restored. But the senior engineer who takes the call works to forensic discipline anyway, for one reason: on ext4, the difference between a successful recovery and a total loss is measured in minutes of uptime, and every careless action spends that budget. The instinct to "just look" is exactly what destroys ext4-recoverable data.

The recovery

The senior engineer's first words on the call are "Do not reboot. Do not write anything to that disk. If you can, pull it offline now." That single instruction is the whole case. Here is why, in the chapter's terms.

On ext4, an unlink removes the directory entry, decrements the inode's i_links_count, and when that count hits zero it frees the inode, frees the data blocks in the block bitmap, and stamps i_dtime with the moment of deletion. Critically — unlike NTFS, which leaves the deleted record's data runs intact — ext4 with extents does not leave a trustworthy block map behind on the live deleted inode. The map to where the files' data physically sits has effectively been stripped.

What has a copy of those maps is the journal (jbd2, reserved inode 8), running in the default ordered mode. Before ext4 committed the metadata changes for the deletion, it wrote the prior state of those inodes — block maps and all — into the journal. That journal copy is the lifeline. But the journal is a fixed-size ring buffer. Every new metadata operation on the still-running server — every log write, every session update, every temp file — appends a new transaction and, once the ring fills, overwrites the oldest transactions, which are precisely the ones holding the deleted files' old inodes. The recovery window is not "until the disk is reused"; it is "until the journal wraps," and on a busy server that can be minutes.

The team gets it offline fast. Because pulling power on a database server is its own risk, they instead remount the file system read-only without replaying the journal — mount -o remount,ro,noload — which both stops new writes and prevents a journal replay from overwriting the transactions they need. Then they image the volume with dd to an external disk and compute a SHA-256, so all recovery work happens on a copy and the production disk is never the thing being experimented on.

  Engagement summary (illustrative)
  source     : server root disk, ext4, 4 KiB blocks, 256-byte inodes
  fs check   : superblock magic 0xEF53 (53 EF) at byte 1080; EXTENTS feature set
  deletion   : /var/www/app/uploads  ~5,800 files  i_dtime ~ 14:14
  action     : remount ro,noload at 14:21 (7 min uptime spent); image with dd
  image      : app_uploads.img   sha256: 9f41...c30b  (verified: match)

With a safe copy in hand, recovery runs the chapter's ordered ext4 strategy.

Step 1 — Mine the journal. Pointed at the image, extundelete --restore-directory /var/www/app/uploads and a parallel ext4magic run scan the jbd2 journal for older copies of the deleted inodes whose extent trees were still populated, then follow those recovered maps to the data blocks — which, having been freed only minutes earlier on a not-yet-busy region of disk, are still present and unreused. Roughly 4,700 of the 5,800 files come back this way, many with their original names intact, because the deleted directory entries were also still legible: ext4 deletion typically removes an entry by extending the previous entry's record length to swallow it, so the swallowed names linger in the directory block's slack until that block is rewritten.

Step 2 — Carve the remainder. For about 1,100 files the journal had already wrapped past the relevant transaction, or the blocks had been partly reused. Here metadata recovery is dead, but the content blocks for many files still sit in unallocated space. A signature-based carving pass (Chapter 7) pulls back several hundred more PDFs and JPEGs by header and footer — without names or paths, but openable. The SQLite store, heavily fragmented and partly overwritten by the server's own writes in those seven minutes, comes back corrupt and is restored instead from the nine-day-old backup with a documented gap.

The honest tally: about 88% of files recovered with names via the journal, another 6% carved without names, and roughly 6% genuinely lost or stale-restored. Good — but not the near-total recovery the same accident would have allowed on an NTFS HDD, and a far cry from what another ten minutes of uptime would have left them.

The analysis

  1. ext4 burns the map on delete; recovery runs through the journal, not the inode. Unlike NTFS, where the deleted MFT record's data runs lead you straight back, ext4's deleted inode no longer maps to the data. The journal's older copy of that inode is what makes recovery possible at all.

  2. The journal is a ring buffer, so uptime is the enemy. Every second the server ran, new transactions overwrote the old inode copies the recovery depended on. The recovery window was set not by disk reuse but by how fast the journal wrapped — minutes, not days.

  3. "Reboot and look" can finish the deletion. A reboot replays the journal and resumes heavy writing; both actions overwrite exactly the transactions needed. The senior engineer's "do not reboot, do not write" was worth more than any tool.

  4. Image first, even in a pure recovery. Remounting ro,noload stopped writes and prevented a destructive journal replay; imaging to a copy meant every extundelete experiment ran against a snapshot, never the irreplaceable original. The original is sacred whether or not a court is involved.

  5. Without a current backup, even a good recovery is partial. The journal and carving saved most of it, but the missing 6% and the stale database trace directly to a silently failing backup job. The cheapest, most reliable "recovery tool" is a backup that was actually tested — the same lesson the ransomware anchor drives home in Chapter 12.

Discussion questions

  1. NTFS recovery starts at the surviving MFT record; ext4 recovery starts at the journal. Explain precisely what each file system does to the deleted file's block map that forces these different starting points.

  2. The on-call engineer wanted to reboot. Walk through, step by step, how a reboot could have converted a mostly-recoverable situation into a mostly-lost one — name journal replay, ring-buffer wrap, and block reuse explicitly.

  3. The team remounted ro,noload rather than letting a normal rw mount proceed. What does noload prevent on ext4, and why is that the difference between preserving and destroying the evidence you need?

  4. ⭐ Suppose this had been an NTFS volume on a hard drive instead of ext4. How would the recovery prognosis, the urgency, and the method have differed? Use the per-file-system meanings of "deleted" to justify your answer, and say whether the "do not reboot" urgency would have been as acute.

  5. The real root cause was a backup job that had silently failed for over a week. Connect this to the ransomware-recovery anchor: what is the shared prevention lesson, and why does "we have backups" mean nothing until the day you prove a restore actually works?