Case Study 2 — The rm -rf at 2 A.M.: A Partial Recovery, Honestly Reported

Where Case Study 1 was a forensic win, this is a recovery job — and a humbler one. A solo founder destroyed three weeks of uncommitted work with one mistyped command on an ext4 SSD with no backups. The ext4 journal and a SQLite WAL handed back the most recent and most important pieces; TRIM and a circular journal kept the rest. The lesson is theme #5: knowing when to stop, and saying so plainly, is part of the job.

Background

A two-person robotics startup runs everything on a single Ubuntu workstation: an NVMe SSD, ext4, no RAID, no off-site backup, and — fatally — a ~/work/ tree full of three weeks of simulation results and tuning data that had never been committed to version control. At 2 a.m., bleary after a long session, the founder meant to clean a build directory and typed a destructive recursive delete one level too high. ~/work/ and roughly 40 GB of results vanished, along with the project's bookkeeping database, project.db (SQLite). There was no Trash for a shell rm; the files were simply unlinked. The founder called a data-recovery service at 9 a.m., seven hours and one anxious reboot later.

You take the job with the standard caution and a standard warning. The caution: the original is sacred, so the first move is to stop using the machine and image the SSD, then work only on the verified copy. The warning, delivered before you quote any odds: on a no-backup ext4 SSD, recovery is partial at best, and I will tell you exactly what came back and what did not. The founder, hoping for a miracle, agrees to a best-effort engagement.

The recovery

Two facts set your expectations before you ran a single tool, and honesty meant stating them up front.

The drive is an SSD, so TRIM is in play. As covered in Chapter 9 — SSD and Flash Recovery, when a file is deleted on a TRIM-enabled SSD, the OS tells the drive those blocks are free, and the controller can zero them in the background within seconds to minutes — independent of the file system. Seven hours and a reboot after the deletion, an unknown fraction of the data blocks were likely already gone at the hardware level. Carving — the metadata-free fallback — would therefore be a coin flip, not a guarantee.

The ext4 journal is small and circular. The jbd2 journal is typically ~128 MB regardless of disk size, and it overwrites itself constantly. On a workstation that kept running for seven hours and rebooted once, much of the journal that existed at 2 a.m. had been recycled. ext4-journal recovery is a fresh-deletion technique; this deletion was no longer fresh. What the journal still held would be the most recent transactions before the overwrite caught up — which, as it turned out, included the pieces that mattered most.

You imaged the SSD, hashed it, and worked the copy. First, enumerate the damage:

$ debugfs -R 'lsdel' /dev/loop0p2        (on the image, read-only)
 Inode   Owner  Mode    Size       Blocks      Time deleted
 786433   1000 100644   41250      11/11       Thu ... 02:03   project.db
 786434   1000 100644   0          0/0         Thu ... 02:03   (block map gone)
 786501   1000 100644   0          0/0         Thu ... 02:03   (block map gone)
 ... ~3,900 inodes, the overwhelming majority showing 0 blocks ...

That wall of 0/0 is the chapter's hard truth made visible: at deletion ext4 zeroes the inode's extent map, so for most files the inode no longer knows where its data was. A few recent inodes — project.db among them — still showed block counts, because the journal had not yet recycled their pre-deletion copies.

You went after the journal-backed inodes first with ext4magic, supplying a tight time window around 02:00:

$ ext4magic /dev/loop0p2 -j journal -a <02:00> -b <02:10> -r -d /recovery
[+] reading journal (jbd2, magic 0xC03B3998) ...
[+] inode 786433  project.db    extents intact in journal  -> recovered (41,250 bytes)
[+] 612 of ~3,900 deleted inodes recoverable from journal/extents
[-] remaining inodes: extent maps not present in surviving journal window

project.db came back whole — and then the SQLite skills from this chapter paid an unexpected dividend. The recovered project.db had a sibling project.db-wal that had also survived in the journal. The write-ahead log held page versions not yet checkpointed into the main database, so it contained the founder's last working session — the experiment records entered in the hours before the disaster, which the main .db file did not yet reflect. Recovering the .db alone would have returned a database hours stale; recovering the -wal with it returned the work right up to the deletion. You also pulled several deleted rows out of the database's freeblocks, restoring records the founder had removed weeks earlier and missed.

For the bulk simulation results — the 0/0 inodes — you fell back to signature carving with PhotoRec (Chapter 7), expecting little because of TRIM. The result confirmed the expectation: carving recovered roughly 30% of the result files by content, but with no filenames, no directory structure, and no original timestamps — just a pile of f0001234.dat-style fragments, some truncated where TRIM had already zeroed trailing blocks. Useful, but a far cry from the named, organized tree the founder had lost.

RECOVERY SCORECARD — Case 2025-074 (ext4, NVMe SSD, no backup, ~7h + 1 reboot post-deletion)
 project.db (+ -wal, + deleted rows) ...... FULL    [ext4 journal + SQLite WAL/freeblocks]
 612 named files via journal/extents ...... FULL    [ext4magic]
 ~30% of bulk results, content only ....... PARTIAL [PhotoRec carving; no names/paths/dates]
 remaining ~70% of bulk results ........... LOST    [extent maps gone; TRIM-zeroed blocks]

You wrote a report that any client could understand and any peer could check: what was recovered and by which technique, what was unrecoverable and why (extent maps gone, TRIM, circular journal), the SHA-256 of every recovered artifact, and a one-page section titled "How to make sure this never happens again" — version control committed and pushed, an automated off-site backup, and the plain arithmetic that a $10/month backup would have made this entire invoice unnecessary. You did not oversell the 30% carve as a save, and you did not pretend the lost 70% might still turn up. The founder was disappointed but not deceived, recovered the one database that ran the business, and became, overnight, a person who backs up.

The temptation in a recovery job is to promise the miracle the client is praying for. Resist it. "I recovered your database and the most recent files, and I can prove the rest is gone" is worth more than a hopeful "maybe" — to the client, and to your reputation the next time they need you.

The analysis

  1. Image first, always — even in a recovery, even at 2 a.m.'s panic. Continuing to use the machine and rebooting it cost data here; the one thing you control is not making it worse, which means stop, image, hash, and work the copy.
  2. The ext4 journal saves the recent, not the old. It is ~128 MB and circular, so it is brilliant for "I deleted it an hour ago" and weak for "seven hours and a reboot ago." It returned the most recent inodes — including the business-critical project.db — and nothing older.
  3. Collect the -wal with the .db or lose the last session. The recovered SQLite database was hours stale on its own; the surviving -wal held the final working session, and freeblocks gave back deleted rows. SQLite recovery is incomplete without its sidecars.
  4. TRIM is the wall on SSDs. On a TRIM-enabled SSD, deleted blocks can be zeroed by the controller within minutes, so carving — the metadata-free fallback — is a coin flip, not a promise. Recovering ~30% by content with no names is a realistic SSD outcome, not a failure of skill.
  5. Without backups, recovery is partial at best — and saying so is the deliverable. The honest scorecard (full / partial / lost, each with its reason) and the prevention plan are as much the product as the recovered bytes. Knowing when to stop, and reporting the limit plainly, is theme #5 in practice.

Discussion questions

  1. The founder rebooted the machine once and used it for seven hours before calling. Explain, using both the ext4 journal and TRIM, how each of those two actions independently reduced what was recoverable — and what you would tell a future client to do in the first five minutes after an accidental deletion.
  2. The recovered project.db was hours stale until you added the -wal. Explain to a non-technical client what the write-ahead log is and why "I recovered your database" and "I recovered your database and its -wal" can be very different deliverables.
  3. Carving returned ~30% of the bulk results as content-only fragments with no names, paths, or timestamps. For what kinds of files is a content-only carve still valuable, and for what kinds is the loss of filenames and structure nearly as bad as losing the data? Tie your answer to the difference between a recovery client's needs and a forensic examiner's needs.
  4. ⭐ Contrast this case with Case Study 1: both turned on the ext4 journal and SQLite internals, yet one was a near-total win and this one was partial. Identify the two situational factors (not skill differences) that most explain the gap, and explain how each would change your up-front estimate to a client before you run any tool.
  5. The report included a prevention section showing that a $10/month backup would have averted the entire loss. Some argue a recovery technician should "just recover and not lecture." Make the professional and ethical case that the prevention guidance is part of the service, and connect it to the book's theme that the human cost behind the data is real.