Appendix B — Python Forensics Toolkit

Purpose. A bench reference of small, well-commented, reusable Python scripts that perform the everyday primitives of recovery and forensics — hashing, carving, EXIF parsing, browser-database querying, timeline building, MFT decoding, and hash-set triage — so you can read the code, understand exactly what a "magic" tool does under the hood, and adapt the pieces to your own cases.

Learning paths: 💾 Data Recovery — the hashing, carving, and EXIF scripts are daily tools. 🔍 Forensic Examiner — every script here turns a manual artifact into a repeatable, documentable step. 🛡️ Incident Response — the SQLite, timeline, and hash-set scripts triage a host fast. 📜 Legal/eDiscovery — understanding how these computations work is what lets you defend them on the stand.

How to use this appendix — read this first

These scripts are teaching code. They are written for clarity over completeness, they have been authored carefully but are illustrative and have not been executed in a sandbox, and they make simplifying assumptions that real evidence will eventually violate. Treat every one of them the way you treat any tool you did not write yourself: validate it against known-answer data before you trust a single byte of its output in a case.

Six standing rules govern everything below. They are the book's themes, expressed as code discipline.

Recovery vs. Forensics. The same script serves both trades, but the standard differs. For a 💾 recovery client who wants their photos back, "the carver pulled 4,000 JPEGs" is success. For a 🔍 forensic report, you must also be able to say which tool, which version, which input hash, and what the tool cannot prove — a carved JPEG has no filename, path, or timestamp, and that limitation is part of the finding, not a footnote.

Environment and conventions

• Python 3.8+, standard library only — no third-party packages are required to run anything here (the EXIF parser is deliberately dependency-free so you can see the bytes; production EXIF work should use Pillow, exifread, or exiftool).
• All timestamp helpers return timezone-aware UTC datetime objects. Convert to local time for display only, never for storage — store and report UTC (see Chapter 21 — Timeline Analysis).
• Each script below also ships as a standalone file in the book's code/ directory (e.g., code/hash_image.py). Sections 1–7 each import from the shared module in Section 0 — put forensic_utils.py on your PYTHONPATH (or in the same folder) before running any other script.
• Sample outputs shown in text blocks are illustrative (hand-constructed, internally consistent) — they show you the shape of the result, not a real run.

The toolkit at a glance

0. Shared module — forensic_utils.py

Purpose. One small module holding the constants and helpers every other script needs: a single-pass multi-algorithm file hasher, and the five timestamp converters you will reach for constantly. Memorize the three epoch constants; you will type them more than any other numbers in this field.

"""forensic_utils.py -- shared helpers for the Appendix B toolkit.

Import only what you need, e.g.:
    from forensic_utils import hash_file, filetime_to_utc, webkit_to_utc

Pure standard library (Python 3.8+). No third-party dependencies.
EDUCATIONAL: validate against a known-answer test before trusting output.
"""
from __future__ import annotations
import hashlib
import datetime as dt

UTC = dt.timezone.utc

# --- Epoch constants: the numbers that bridge the clocks of forensics ---------
# 100-ns intervals between 1601-01-01 and 1970-01-01 (FILETIME of the Unix epoch)
FILETIME_EPOCH_DELTA = 116_444_736_000_000_000
# Seconds between 1601-01-01 and 1970-01-01 (Chrome/WebKit offset = above / 1e7)
WEBKIT_EPOCH_DELTA   = 11_644_473_600
# Seconds between 1970-01-01 and 2001-01-01 (Mac/Cocoa absolute-time offset)
MAC_EPOCH_DELTA      = 978_307_200


def hash_file(path, algorithms=("md5", "sha256"), chunk_size=1 << 20):
    """Return {algo: hexdigest}, computed in ONE streamed pass over the file.

    chunk_size defaults to 1 MiB so a multi-gigabyte image never loads into RAM.
    Reading in fixed blocks also means the same bytes feed every algorithm, so
    the digests are guaranteed to describe the identical input.
    """
    hashers = {name: hashlib.new(name) for name in algorithms}
    with open(path, "rb") as fh:
        for block in iter(lambda: fh.read(chunk_size), b""):
            for h in hashers.values():
                h.update(block)
    return {name: h.hexdigest() for name, h in hashers.items()}


# --- Timestamp converters: every one returns an aware UTC datetime (or None) --
def filetime_to_utc(filetime):
    """Windows FILETIME (100-ns ticks since 1601-01-01 UTC) -> datetime/None.

    Used by NTFS ($MFT), the registry, .evtx logs, and DPAPI. 0 and the
    all-ones 'never' sentinel both mean 'not set'.
    """
    if not filetime or filetime == 0xFFFFFFFFFFFFFFFF:
        return None
    # 100 ns == 0.1 microsecond, so divide the tick count by 10 for microseconds
    return dt.datetime(1601, 1, 1, tzinfo=UTC) + dt.timedelta(microseconds=filetime / 10)


def webkit_to_utc(webkit):
    """Chrome/WebKit time (microseconds since 1601-01-01 UTC) -> datetime/None.

    Chrome, Edge, Brave, Opera, Vivaldi. Note: same epoch as FILETIME, but the
    unit is microseconds (FILETIME / 10), NOT 100-ns ticks.
    """
    if not webkit:
        return None
    return dt.datetime(1601, 1, 1, tzinfo=UTC) + dt.timedelta(microseconds=webkit)


def unix_to_utc(seconds):
    """Unix/POSIX time (seconds since 1970-01-01 UTC) -> datetime/None.

    ext4, syslog/journald, the Sleuth Kit bodyfile, and many app databases.
    """
    if not seconds:
        return None
    return dt.datetime(1970, 1, 1, tzinfo=UTC) + dt.timedelta(seconds=seconds)


def prtime_to_utc(prtime):
    """Firefox PRTime (microseconds since 1970-01-01 UTC) -> datetime/None.

    WATCH OUT: in moz_cookies, creationTime/lastAccessed are PRTime microseconds
    but `expiry` is plain Unix SECONDS -- different divisor (Ch.18).
    """
    if not prtime:
        return None
    return dt.datetime(1970, 1, 1, tzinfo=UTC) + dt.timedelta(microseconds=prtime)


def mac_to_utc(seconds):
    """Mac/Cocoa absolute time (seconds since 2001-01-01 UTC) -> datetime/None.

    Safari history, many macOS/iOS plists and databases.
    """
    if not seconds:
        return None
    return dt.datetime(2001, 1, 1, tzinfo=UTC) + dt.timedelta(seconds=seconds)

Usage notes.

• The four epochs you will meet — and the one mistake that loses cases — are summarized below. Misreading an epoch is the single most common technical error in artifact dating and the easiest for opposing counsel to expose, so state the epoch for every timestamp in your notes.

• hash_file accepts any algorithm name hashlib knows ("sha1", "sha512", "blake2b"). It is the engine behind Sections 1 and 7.

1. Hashing and verification — hash_image.py

Purpose. Compute the MD5 and SHA-256 of any file or disk image in a single streamed pass, optionally verify against an expected value, and optionally write *sum-compatible sidecar files. This is the first thing you do to an acquired image and the last thing you do before handing off a working copy — the mathematical backbone of chain of custody.

#!/usr/bin/env python3
"""hash_image.py -- compute & verify MD5 and SHA-256 of a file or image.

    python hash_image.py evidence.E01
    python hash_image.py evidence.dd --verify <expected_sha256_or_md5>
    python hash_image.py evidence.dd --sidecar        # writes .md5 and .sha256

Why BOTH algorithms: SHA-256 is the integrity standard you cite in court; MD5
is retained only because many acquisition tools and legacy logs (FTK Imager,
dd-based flows) still emit it, so keeping it lets you cross-check older records.
NEVER rely on MD5 alone for tamper-evidence -- it is collision-broken (Ch.5).
"""
import argparse
import sys
import datetime as dt
from forensic_utils import hash_file


def main():
    ap = argparse.ArgumentParser(description="Compute/verify MD5 + SHA-256.")
    ap.add_argument("path")
    ap.add_argument("--verify", metavar="HEX",
                    help="expected SHA-256 or MD5; compare and set the exit code")
    ap.add_argument("--sidecar", action="store_true",
                    help="write <path>.md5 and <path>.sha256 next to the file")
    args = ap.parse_args()

    digests = hash_file(args.path, ("md5", "sha256"))
    stamp = dt.datetime.now(dt.timezone.utc).isoformat(timespec="seconds")

    print(f"# file     : {args.path}")
    print(f"# utc_time : {stamp}")
    print(f"MD5    = {digests['md5']}")
    print(f"SHA256 = {digests['sha256']}")

    if args.sidecar:
        for algo in ("md5", "sha256"):
            with open(f"{args.path}.{algo}", "w", encoding="ascii") as fh:
                # `md5sum`/`sha256sum`-compatible line:  <hash><space><space><name>
                fh.write(f"{digests[algo]}  {args.path}\n")

    if args.verify:
        want = args.verify.strip().lower()
        ok = want in (digests["md5"], digests["sha256"])
        print(f"VERIFY = {'MATCH' if ok else 'MISMATCH'}  (expected {want})")
        sys.exit(0 if ok else 1)


if __name__ == "__main__":
    main()

Usage notes.

# Acquire-time: hash the image and drop sidecars for the case folder
python hash_image.py WS-ENG-04.dd --sidecar

# Hand-off time: prove the working copy still matches the acquisition hash
python hash_image.py WS-ENG-04_copy.dd --verify 9f2c...e1

# file     : WS-ENG-04.dd
# utc_time : 2026-06-28T14:02:51+00:00
MD5    = 5d41402abc4b2a76b9719d911017c592
SHA256 = 9f2c1e0b7a4d3f8c6e5b2a1d0c9f8e7d6c5b4a3f2e1d0c9b8a7f6e5d4c3b2a1e

• The exit code (0 match, 1 mismatch) makes this drop straight into a shell pipeline or acquisition script. A mismatch is not a curiosity — it means the copy is not the evidence, and you stop and investigate before doing anything else.
• For directory-wide or recursive hashing with a recomputable audit log, prefer hashdeep -r -c md5,sha256 (it also does the known-file matching of Section 7). This script exists so you understand what hashdeep is doing.

Chain of Custody. A hash is a promise: this is the same data I acquired, unaltered. The promise is only as good as your records, so the script timestamps every run in UTC and the sidecars are plain text you can paste into the custody log. The hash proves integrity; your documentation proves you computed it, when, and on what (see Appendix F — Chain-of-Custody and Report Templates).

Limitation. A matching hash proves two byte streams are identical; it says nothing about authenticity (whether the data was genuine when acquired) or meaning. And while SHA-256 is collision-resistant in practice, MD5 is not — never use an MD5-only match to rebut an alteration claim.

2. File carving — carve.py

Purpose. Scan a raw image for known file headers and (where they exist) footers, and write out the bytes in between — recovering files with no help from the file system at all. This is a stripped-down teaching model of foremost/scalpel/photorec so you can see the core loop; the signatures match Appendix A and Chapter 7.

#!/usr/bin/env python3
"""carve.py -- educational header/footer file carver for a RAW disk image.

    python carve.py image.dd carved_out/

It carves header-to-footer with a size cap. It does NOT handle fragmentation,
nested files (a JPEG thumbnail inside another JPEG's EXIF), or structural
validation -- real cases need `scalpel`, `foremost`, or `photorec` plus a
human check of every carved file (Ch.7, Appendix H). Run against an IMAGE or a
verified copy, never a live original.
"""
import os
import mmap
import argparse

# (header_bytes, footer_bytes_or_None, max_carve_size, extension)
SIGNATURES = [
    (b"\xFF\xD8\xFF",            b"\xFF\xD9",              20_000_000,  "jpg"),
    (b"\x89PNG\r\n\x1a\n",       b"IEND\xae\x42\x60\x82",  50_000_000,  "png"),
    (b"GIF89a",                  b"\x00\x3B",              10_000_000,  "gif"),
    (b"GIF87a",                  b"\x00\x3B",              10_000_000,  "gif"),
    (b"%PDF",                    b"%%EOF",                100_000_000,  "pdf"),
    (b"PK\x03\x04",              None,                     50_000_000,  "zip"),  # also docx/xlsx/jar
]


def carve(image_path, out_dir, signatures=SIGNATURES):
    """Carve `image_path` into `out_dir`; return {extension: count}."""
    os.makedirs(out_dir, exist_ok=True)
    counts = {}
    with open(image_path, "rb") as fh:
        mm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)  # whole image, lazily paged
        try:
            for header, footer, max_size, ext in signatures:
                pos = mm.find(header, 0)
                while pos != -1:
                    if footer:
                        # search for the footer after the header, within max_size
                        window_end = min(pos + max_size, len(mm))
                        f = mm.find(footer, pos + len(header), window_end)
                        end = f + len(footer) if f != -1 else window_end
                    else:
                        end = min(pos + max_size, len(mm))   # no footer: cap the size

                    n = counts.get(ext, 0)
                    out_name = os.path.join(out_dir, f"{ext}_{n:06d}_off{pos}.{ext}")
                    with open(out_name, "wb") as out:
                        out.write(mm[pos:end])
                    counts[ext] = n + 1

                    pos = mm.find(header, end)   # resume past what we just carved
        finally:
            mm.close()
    return counts


if __name__ == "__main__":
    ap = argparse.ArgumentParser(description="Educational signature carver.")
    ap.add_argument("image")
    ap.add_argument("out_dir")
    args = ap.parse_args()
    for ext, n in sorted(carve(args.image, args.out_dir).items()):
        print(f"{n:6d}  {ext}")

Usage notes.

python carve.py wedding_drive.dd carved/

  4127  jpg
   312  png
    58  pdf
    19  gif
     7  zip

• Output filenames embed the byte offset where each file was found (jpg_000123_off1073745920.jpg). Record that offset — it is the only provenance a carved file has, and it lets you point back to the exact sector (offset / 512) in the image.
• The carver does one full mm.find sweep per signature for readability. A production carver makes a single pass matching all headers at once; if you carve large images often, that is the first optimization to make.

Limitation. Three things this carver (and signature carving in general) cannot do, which you must state in any report: (1) Fragmentation — if a file's clusters are not contiguous, header-to-footer carving splices in unrelated data or stops short; only smart carvers (PhotoRec) and file-system-aware recovery handle this. (2) No metadata — a carved file has no name, path, owner, or timestamp; you cannot prove when it was created or who saved it from the carved bytes alone. (3) Footerless formats — for PK\x03\x04 (ZIP/Office), MP4, and others with no reliable trailer, the size cap means you over- or under-carve; validate every such file. Knowing when carving has reached its limit is theme #5.

Ethics Note. A carver pulls everything matching a signature out of unallocated space, including material you were not looking for. If carved output includes apparent CSAM, stop, do not open further files, preserve, and follow Chapter 28 and 18 U.S.C. §2258A. Hash-set matching (Section 7) lets you flag known contraband by hash without viewing it.

3. EXIF extraction — parse_exif.py

Purpose. Read the camera make/model, capture timestamps, orientation, and (when present) decimal GPS coordinates out of a JPEG by parsing the APP1/Exif TIFF structure directly — no third-party library — so you understand exactly where that "the photo was taken here, then" evidence comes from.

#!/usr/bin/env python3
"""parse_exif.py -- minimal, dependency-free Exif reader for JPEG files.

Pulls common IFD0/ExifIFD tags (Make, Model, Software, DateTime,
DateTimeOriginal, Orientation) and, when present, decimal GPS coordinates from
the GPS IFD, by walking the APP1 Exif TIFF structure directly.

TEACHING parser: standard Exif in JPEG APP1 only. For real work use Pillow,
the `exifread` package, or `exiftool` (Ch.20) -- they handle MakerNotes, XMP,
HEIC/TIFF/RAW, and malformed tags. Treat embedded GPS as a LEAD to corroborate,
never as standalone proof of where a device (or person) was.
"""
import struct
import sys

IFD0_TAGS = {0x010F: "Make", 0x0110: "Model", 0x0131: "Software",
             0x0132: "DateTime", 0x0112: "Orientation"}
EXIF_TAGS = {0x9003: "DateTimeOriginal", 0x9004: "DateTimeDigitized",
             0xA002: "PixelXDimension", 0xA003: "PixelYDimension"}
GPS_TAGS  = {0x0001: "GPSLatitudeRef", 0x0002: "GPSLatitude",
             0x0003: "GPSLongitudeRef", 0x0004: "GPSLongitude"}

# TIFF field type -> size in bytes of ONE element
TYPE_SIZE = {1: 1, 2: 1, 3: 2, 4: 4, 5: 8, 7: 1, 9: 4, 10: 8}


def _find_app1(data):
    """Return the Exif TIFF payload (bytes after 'Exif\\0\\0'), or None."""
    if data[:2] != b"\xFF\xD8":              # not a JPEG SOI marker
        return None
    i = 2
    while i + 4 <= len(data):
        if data[i] != 0xFF:                  # marker structure broke
            return None
        marker = data[i + 1]
        seg_len = struct.unpack(">H", data[i + 2:i + 4])[0]   # includes the 2 len bytes
        if marker == 0xE1 and data[i + 4:i + 10] == b"Exif\x00\x00":
            return data[i + 10: i + 2 + seg_len]              # the TIFF block
        if marker == 0xDA:                   # Start Of Scan: pixel data begins, stop
            return None
        i += 2 + seg_len
    return None


def _read_ifd(buf, offset, bo):
    """Yield (tag, ftype, count, value_bytes) for each entry of an IFD.

    `buf` is the whole TIFF block; all offsets are relative to its start.
    Values <= 4 bytes are stored inline; larger ones live at a pointer.
    """
    count = struct.unpack_from(bo + "H", buf, offset)[0]
    entry = offset + 2
    for _ in range(count):
        tag, ftype, n = struct.unpack_from(bo + "HHI", buf, entry)
        total = TYPE_SIZE.get(ftype, 1) * n
        if total <= 4:
            raw = buf[entry + 8: entry + 8 + total]
        else:
            ptr = struct.unpack_from(bo + "I", buf, entry + 8)[0]
            raw = buf[ptr: ptr + total]
        yield tag, ftype, n, raw
        entry += 12


def _decode(ftype, n, raw, bo):
    """Turn raw value bytes into a str / number / list of numbers."""
    if ftype == 2:                                   # ASCII (NUL-terminated)
        return raw.split(b"\x00", 1)[0].decode("ascii", "replace")
    if ftype in (1, 3, 4, 9):                        # integer scalar(s)
        code = {1: "B", 3: "H", 4: "I", 9: "i"}[ftype]
        unit = {1: 1, 3: 2, 4: 4, 9: 4}[ftype]
        vals = struct.unpack(bo + code * n, raw[: unit * n])
        return vals[0] if n == 1 else list(vals)
    if ftype in (5, 10):                             # (S)RATIONAL = numerator/denominator
        code = "II" if ftype == 5 else "ii"
        nums = struct.unpack(bo + code * n, raw[: 8 * n])
        return [num / den if den else 0.0
                for num, den in zip(nums[0::2], nums[1::2])]
    return raw


def _gps_to_decimal(dms, ref):
    """[deg, min, sec] + 'N'/'S'/'E'/'W'  ->  signed decimal degrees."""
    deg, minutes, sec = (list(dms) + [0, 0, 0])[:3]
    dec = deg + minutes / 60.0 + sec / 3600.0
    return -dec if ref in ("S", "W") else dec


def parse_exif(path):
    """Return a dict of decoded EXIF fields for a JPEG (empty dict if none)."""
    with open(path, "rb") as fh:
        tiff = _find_app1(fh.read())
    if not tiff:
        return {}

    bo = "<" if tiff[:2] == b"II" else ">"           # II = little-endian, MM = big
    ifd0_off = struct.unpack_from(bo + "I", tiff, 4)[0]

    result, gps_off, exif_off = {}, None, None
    for tag, ftype, n, raw in _read_ifd(tiff, ifd0_off, bo):
        if tag == 0x8825:                            # pointer to the GPS IFD
            gps_off = struct.unpack_from(bo + "I", raw, 0)[0]
        elif tag == 0x8769:                          # pointer to the Exif SubIFD
            exif_off = struct.unpack_from(bo + "I", raw, 0)[0]
        elif tag in IFD0_TAGS:
            result[IFD0_TAGS[tag]] = _decode(ftype, n, raw, bo)

    if exif_off:
        for tag, ftype, n, raw in _read_ifd(tiff, exif_off, bo):
            if tag in EXIF_TAGS:
                result[EXIF_TAGS[tag]] = _decode(ftype, n, raw, bo)

    if gps_off:
        gps = {}
        for tag, ftype, n, raw in _read_ifd(tiff, gps_off, bo):
            if tag in GPS_TAGS:
                gps[GPS_TAGS[tag]] = _decode(ftype, n, raw, bo)
        if "GPSLatitude" in gps and "GPSLongitude" in gps:
            result["GPSLatitude_dd"] = _gps_to_decimal(
                gps["GPSLatitude"], gps.get("GPSLatitudeRef", "N"))
            result["GPSLongitude_dd"] = _gps_to_decimal(
                gps["GPSLongitude"], gps.get("GPSLongitudeRef", "E"))
    return result


if __name__ == "__main__":
    for key, val in parse_exif(sys.argv[1]).items():
        print(f"{key:20} {val}")

Usage notes.

python parse_exif.py carved_000123.jpg

Make                 Apple
Model                iPhone 13 Pro
DateTime             2024:03:14 09:41:07
DateTimeOriginal     2024:03:14 09:41:07
Orientation          6
GPSLatitude_dd       37.33182777777778
GPSLongitude_dd      -122.03118055555556

• How the bytes work, in one breath: a JPEG is FF D8 then marker segments; the FF E1 (APP1) segment whose payload starts with Exif\0\0 contains a complete little TIFF — a byte-order mark (II/MM), the magic 42, an offset to IFD0, then 12-byte entries each holding a tag, type, count, and either an inline value or a pointer. The GPS data lives in its own IFD reached through tag 0x8825. That is the entire format your eyes need.
• EXIF DateTimeOriginal is the camera's local clock with no time-zone field, and it is trivially editable. Corroborate it with file-system MACB times (Chapter 21) and, on modern phones, the maker-note offset that newer exiftool versions expose.
• For anything beyond this happy path — HEIC/RAW, XMP, thumbnails, vendor MakerNotes, damaged tags — use exiftool and cite it. This parser is for understanding, and for quick triage of plain JPEGs.

Limitation. Embedded GPS proves where the device that wrote the metadata recorded itself being — not necessarily where the subject is, and not at all if the field was forged or stripped. EXIF is an investigative lead that must be corroborated (cell-site records, other timestamps, witness accounts), never a lone conclusion. This nuance is exactly the kind of thing that distinguishes a careful 🔍 examiner's report from an overreach.

4. Browser history (SQLite) — browser_history.py

Purpose. Safely copy a Chrome or Firefox history database (with its WAL/SHM sidecars) and run a join over the visit tables, converting each timestamp from the browser's native epoch to UTC. This is the read-only, do-no-harm pattern every browser-forensics tool follows under the hood.

#!/usr/bin/env python3
"""browser_history.py -- safely copy and query a Chrome/Firefox history DB.

    python browser_history.py "/eo/History" chrome
    python browser_history.py "/eo/places.sqlite" firefox

Browsers keep their SQLite files open and use WAL journaling, so recent
activity may live in the -wal sidecar, not the main file. The safe pattern:
  (1) copy the DB *and* its -wal/-shm next to it into a work area, then
  (2) open the COPY read-only. Never query a DB inside a live profile or in
your evidence image in place. Hash the copy and log it (Ch.5, Ch.18).
"""
import sqlite3
import shutil
import os
import argparse
from forensic_utils import webkit_to_utc, prtime_to_utc

# Chrome: visits.visit_time is WebKit time (microseconds since 1601).
CHROME_SQL = """
SELECT u.url, u.title, u.visit_count, v.visit_time
FROM   urls u JOIN visits v ON u.id = v.url
ORDER  BY v.visit_time
"""

# Firefox: moz_historyvisits.visit_date is PRTime (microseconds since 1970).
FIREFOX_SQL = """
SELECT p.url, p.title, p.visit_count, h.visit_date
FROM   moz_places p JOIN moz_historyvisits h ON p.id = h.place_id
ORDER  BY h.visit_date
"""


def _open_readonly_copy(db_path, work_dir):
    """Copy db + -wal + -shm into work_dir; return a read-only connection."""
    os.makedirs(work_dir, exist_ok=True)
    base = os.path.basename(db_path)
    for suffix in ("", "-wal", "-shm"):          # main file first, then journals
        src = db_path + suffix
        if os.path.exists(src):
            shutil.copy2(src, os.path.join(work_dir, base + suffix))
    copy_path = os.path.join(work_dir, base)
    # mode=ro (not 'immutable') so SQLite still replays the -wal we copied.
    return sqlite3.connect(f"file:{copy_path}?mode=ro", uri=True)


def query_history(db_path, browser, work_dir="_work"):
    """Yield {utc, url, title, visits} rows from a browser history DB."""
    conn = _open_readonly_copy(db_path, work_dir)
    try:
        sql = CHROME_SQL if browser == "chrome" else FIREFOX_SQL
        to_utc = webkit_to_utc if browser == "chrome" else prtime_to_utc
        for url, title, visit_count, ts in conn.execute(sql):
            when = to_utc(ts)
            yield {"utc": when.isoformat() if when else "",
                   "url": url, "title": title or "", "visits": visit_count}
    finally:
        conn.close()


if __name__ == "__main__":
    ap = argparse.ArgumentParser(description="Query a browser history DB copy.")
    ap.add_argument("db")
    ap.add_argument("browser", choices=("chrome", "firefox"))
    args = ap.parse_args()
    for row in query_history(args.db, args.browser):
        print(f'{row["utc"]:25}  ({row["visits"]:>4}x)  {row["url"]}')

Usage notes.

2024-03-14T16:22:09+00:00  (   3x)  https://mail.example.com/u/0/
2024-03-14T16:31:55+00:00  (   1x)  https://drive.google.com/drive/u/0/my-drive
2024-03-14T16:44:02+00:00  (   1x)  https://wetransfer.com/

• The WAL matters. If you copy only History and ignore History-wal, you may miss the most recent browsing — exactly the activity an investigation cares about. Copy all three sidecars; opening with mode=ro (not immutable=1) lets SQLite merge the WAL so your query sees the current state.
• The same pattern reads any browser SQLite store: Cookies, Login Data, Web Data, and Downloads (Chrome), or cookies.sqlite and favicons.sqlite (Firefox). Swap the SQL and the converter. Firefox cookie trap: creationTime/lastAccessed are PRTime microseconds, but expiry is Unix seconds — use unix_to_utc for that one column (Ch.18).
• For Safari, copy History.db and convert visit_time with mac_to_utc (the join is history_visits.history_item -> history_items.id).
• For an end-to-end Chrome profile parse (transitions decoded, WAL handled, cache and downloads correlated) use Hindsight; this script is the minimal model of what it does.

Recovery vs. Forensics. Querying a copied database is the textbook expression of the original is sacred. The 💾 recovery tech copies-then-reads because the source profile may be on a failing drive that should be touched as little as possible; the 🔍 examiner copies-then-reads because querying in place would update SQLite's own internal counters and break the hash. Same shutil.copy2, two reasons.

5. MACB timeline to CSV — timeline_from_bodyfile.py

Purpose. Convert a Sleuth Kit bodyfile into a sorted CSV super-timeline, expanding each file's four times into MACB events exactly the way mactime does — so you can build, diff, and trust a timeline you understand line by line.

#!/usr/bin/env python3
"""timeline_from_bodyfile.py -- turn a Sleuth Kit bodyfile into a MACB CSV.

INPUT: a TSK 3.x bodyfile (from `fls -m / -r -o <offset> image.dd > body.txt`).
Pipe-delimited, times in Unix epoch, 0 = not set:
  MD5 | name | inode | mode | UID | GID | size | atime | mtime | ctime | crtime

OUTPUT: one CSV row per (timestamp, file), with a combined flag string in
mactime's column order  M A C B  -- so you can diff this against the canonical
`mactime` tool while learning (Ch.21). Build the bodyfile against a VERIFIED
image; the original is never touched (theme #2).
"""
import csv
import argparse
from collections import defaultdict
from forensic_utils import unix_to_utc

# bodyfile column index -> MACB letter
TIME_FIELDS = [(8, "m"), (7, "a"), (9, "c"), (10, "b")]   # mtime atime ctime crtime


def build_timeline(bodyfile_path):
    """Collapse a bodyfile into {(epoch, name, size): set_of_MACB_letters}."""
    events = defaultdict(set)
    meta = {}                                     # (key) -> (inode, mode) for output
    with open(bodyfile_path, "r", encoding="utf-8", errors="replace") as fh:
        for line in fh:
            line = line.rstrip("\n")
            if not line or line.startswith("#"):
                continue
            cols = line.split("|")
            if len(cols) < 11:
                continue
            name, size = cols[1], cols[6]
            for idx, letter in TIME_FIELDS:
                epoch = int(cols[idx] or 0)
                if epoch:                         # 0 means "not recorded"; skip it
                    key = (epoch, name, size)
                    events[key].add(letter)
                    meta[key] = (cols[2], cols[3])    # inode, mode string
    return events, meta


def macb_string(letters):
    """Render {'m','b'} etc. as a fixed 'macb' mask with dots: e.g. 'm..b'."""
    return "".join(ch if ch in letters else "." for ch in "macb")


def write_csv(events, meta, out_path):
    with open(out_path, "w", newline="", encoding="utf-8") as fh:
        w = csv.writer(fh)
        w.writerow(["utc_time", "macb", "size", "inode", "mode", "name"])
        for key in sorted(events):                # sort by (epoch, name, size)
            epoch, name, size = key
            when = unix_to_utc(epoch)
            inode, mode = meta[key]
            w.writerow([when.isoformat() if when else "",
                        macb_string(events[key]), size, inode, mode, name])


if __name__ == "__main__":
    ap = argparse.ArgumentParser(description="Bodyfile -> MACB CSV timeline.")
    ap.add_argument("bodyfile")
    ap.add_argument("out_csv")
    args = ap.parse_args()
    ev, mt = build_timeline(args.bodyfile)
    write_csv(ev, mt, args.out_csv)
    print(f"wrote {len(ev)} timeline events to {args.out_csv}")

Usage notes.

# 1) Build the bodyfile with The Sleuth Kit (offset from `mmls`), recursively:
fls -m "C:" -r -o 2048 WS-ENG-04.dd > WS-ENG-04.body
# 2) Convert to a sorted MACB CSV you can open in any spreadsheet:
python timeline_from_bodyfile.py WS-ENG-04.body timeline.csv

utc_time,macb,size,inode,mode,name
2024-03-14T16:40:02+00:00,...b,182734,71-128-3,r/rrwxrwxrwx,C:/Users/jrivera/Desktop/TurbineHousing_v7.sldprt
2024-03-14T16:44:55+00:00,m.c.,182734,71-128-3,r/rrwxrwxrwx,C:/Users/jrivera/Desktop/TurbineHousing_v7.sldprt
2024-03-14T16:45:01+00:00,.a..,4096,72-144-1,r/rrwxrwxrwx,C:/Users/jrivera/Desktop/

• Grouping multiple times that share an instant into one macb row (e.g., mac.) mirrors mactime and keeps the timeline readable. Sub-second precision is lost in the bodyfile's whole-second epoch — for NTFS 100-ns resolution, parse $MFT directly (Section 6) or use plaso.
• The shared converters in Section 0 also handle raw timestamp values you pull from other artifacts: feed a registry FILETIME to filetime_to_utc, a Chrome value to webkit_to_utc, and merge them into the same CSV to build a true cross-artifact super-timeline.
• For the comprehensive version — registry, event logs, browser, LNK, prefetch, all on one axis — run log2timeline.py/psort.py (plaso). This script teaches the file-system layer that sits at a timeline's core.

Why This Matters. A timeline is where isolated artifacts become a story a jury can follow: the document opened, the USB inserted, the file copied, the browser uploaded, the logs cleared. In anchor case #2 — the employee who covered their tracks — it is the timeline that exposes the theft, because altered $STANDARD_INFORMATION` times collide with the untouched `$FILE_NAME times the next script reads.

6. NTFS MFT record parser — parse_mft_record.py

Purpose. Decode a single 1,024-byte NTFS MFT FILE record the hard way — apply the fixup array, walk the resident attributes, and pull the filename plus both timestamp sets — so you can see why $FILE_NAME` times betray timestomping that `$STANDARD_INFORMATION times hide. Educational companion to Appendix G and Chapter 16.

#!/usr/bin/env python3
"""parse_mft_record.py -- decode ONE 1024-byte NTFS MFT entry (educational).

Reads the FILE record header, applies the fixup (update-sequence) array, then
walks RESIDENT attributes to extract the filename and the two timestamp sets:
  $STANDARD_INFORMATION (type 0x10) -- the times Explorer shows; easily forged
  $FILE_NAME            (type 0x30) -- mirror times the OS rarely rewrites

A $SI 'created' EARLIER than the matching $FN 'created', or $SI times with
zeroed sub-seconds, is a classic timestomping tell (Ch.21, Ch.30, case #2).

Get a record to feed it, e.g.:  `icat image.dd 0` dumps the $MFT; slice a
1024-byte entry from it. Use `istat`/Autopsy for the polished human view --
this parser is for LEARNING the on-disk bytes.
"""
import struct
import sys
import pprint
from forensic_utils import filetime_to_utc

SECTOR = 512


def apply_fixup(record):
    """Restore the real last 2 bytes of each 512-byte sector (NTFS fixup)."""
    rec = bytearray(record)
    usa_off, usa_cnt = struct.unpack_from("<HH", rec, 0x04)  # array offset, count
    usn = bytes(rec[usa_off:usa_off + 2])                    # value seen at each tail
    for i in range(1, usa_cnt):                              # entry 0 is the USN itself
        saved = rec[usa_off + i * 2: usa_off + i * 2 + 2]    # the true bytes
        tail = i * SECTOR - 2                                # last 2 bytes of sector i-1
        if bytes(rec[tail:tail + 2]) != usn:
            # Tail should equal the USN; a mismatch hints at corruption.
            pass
        rec[tail:tail + 2] = saved
    return bytes(rec)


def _four_filetimes(buf, base):
    """Decode 4 consecutive FILETIMEs (create, modify, mft-change, access)."""
    crtime, mtime, ctime, atime = struct.unpack_from("<QQQQ", buf, base)
    return {"B_created":    filetime_to_utc(crtime),
            "M_modified":   filetime_to_utc(mtime),
            "C_mft_change": filetime_to_utc(ctime),
            "A_accessed":   filetime_to_utc(atime)}


def parse_record(record):
    """Decode one MFT FILE record into a dict of flags, name, and timestamps."""
    if record[:4] != b"FILE":                      # 'BAAD' = a record TSK marked corrupt
        raise ValueError("not an MFT FILE record (bad signature)")
    record = apply_fixup(record)

    flags = struct.unpack_from("<H", record, 0x16)[0]
    info = {"in_use": bool(flags & 0x01),
            "is_directory": bool(flags & 0x02),
            "attributes": {}}

    offset = struct.unpack_from("<H", record, 0x14)[0]      # offset to 1st attribute
    while offset < len(record) - 4:
        atype = struct.unpack_from("<I", record, offset)[0]
        if atype == 0xFFFFFFFF:                    # 0xFFFFFFFF marks end of attributes
            break
        alen = struct.unpack_from("<I", record, offset + 0x04)[0]
        if alen == 0:                              # guard against a malformed record
            break
        non_resident = record[offset + 0x08]
        if non_resident == 0:                      # we only read resident attributes
            content_off = struct.unpack_from("<H", record, offset + 0x14)[0]
            base = offset + content_off
            if atype == 0x10:                      # $STANDARD_INFORMATION
                info["attributes"]["$STANDARD_INFORMATION"] = _four_filetimes(record, base)
            elif atype == 0x30:                    # $FILE_NAME
                fn_times = _four_filetimes(record, base + 0x08)   # after 8-byte parent ref
                name_len = record[base + 0x40]                    # length in UTF-16 chars
                name = record[base + 0x42: base + 0x42 + name_len * 2]
                info["attributes"]["$FILE_NAME"] = {
                    "name": name.decode("utf-16-le", "replace"),
                    "times": fn_times}
        offset += alen                              # advance by the attribute's length
    return info


if __name__ == "__main__":
    with open(sys.argv[1], "rb") as fh:
        pprint.pprint(parse_record(fh.read(1024)))

Usage notes.

{'attributes': {'$FILE_NAME': {'name': 'TurbineHousing_v7.sldprt',
                               'times': {'A_accessed': datetime(2024, 3, 14, 16, 45, ...),
                                         'B_created':  datetime(2024, 3, 14, 16, 40, 2, ...),
                                         'C_mft_change': datetime(2024, 3, 14, 16, 44, ...),
                                         'M_modified': datetime(2024, 3, 14, 16, 44, ...)}}},
                '$STANDARD_INFORMATION': {'B_created':  datetime(2021, 1, 5, 8, 0, 0, ...),
                                          'M_modified': datetime(2021, 1, 5, 8, 0, 0, ...),
                                          ...}},
 'in_use': True,
 'is_directory': False}

• The timestomping tell. In the illustrative output above, $STANDARD_INFORMATION` claims the file was created in **2021** while `$FILE_NAME records 2024 — and the $SI` times are suspiciously round (`08:00:00.000`). Tools like `timestomp` and `SetMACE` rewrite `$SI (what Explorer and most tools show) but typically leave $FILE_NAME` alone, because updating it requires moving/renaming the file. When `$SI predates $FN`, the `$FN times are usually the truth. This is the mechanism behind anchor case #2.
• The fixup step is not optional: NTFS overwrites the last two bytes of every 512-byte sector with a sequence number and stashes the originals in the update-sequence array. Skip apply_fixup and any field that straddles offset 510 or 1022 is corrupt. Applying it is what makes a hand-rolled parser trustworthy.
• This reads resident $STANDARD_INFORMATION`/`$FILE_NAME only — which is the common case for these two attributes. Non-resident $DATA (the file's actual content via cluster runs), attribute lists for large records, and ADS are out of scope here; use istat, MFTECmd, or analyzeMFT for full extraction, and read Appendix G for the complete structure.

Tool Tip. To get a record to feed this parser: icat image.dd 0 > mft.raw extracts the whole $MFT, and entry N is the 1,024 bytes at byte offset N * 1024. istat image.dd <inode> gives you the validated human-readable version to check your decode against — always confirm your bytes agree with istat before relying on them.

7. Hash-set lookup — hashset_lookup.py

Purpose. Classify a tree of files against known-good and known-bad hash sets in one pass, so you can filter out the millions of innocuous operating-system files and flag known contraband by hash, without ever opening it. This is the triage step that makes a 2-terabyte image humanly reviewable.

#!/usr/bin/env python3
"""hashset_lookup.py -- classify files against known-good / known-bad hash sets.

    python hashset_lookup.py /mnt/evidence --good nsrl.txt --bad projectvic.txt

Loads hash lists (one hex digest per line; '#' comments and a trailing
',label' both tolerated) into Python sets for O(1) membership tests, walks a
directory, hashes each file (SHA-256), and labels it:
  KNOWN-GOOD -> in the good set (e.g., NSRL RDS): filter out, de-prioritize
  KNOWN-BAD  -> in the bad set  (e.g., Project VIC / CAID): FLAG, DO NOT OPEN
  UNKNOWN    -> needs human review

KNOWN-BAD sets let you identify contraband by hash WITHOUT viewing the file --
essential for legality and examiner well-being (Ch.28). You hold and share the
HASHES, never the underlying material.
"""
import os
import argparse
from forensic_utils import hash_file


def load_hashset(path):
    """Read a hash list (hex, one per line) into a lowercase set."""
    out = set()
    with open(path, "r", encoding="utf-8", errors="replace") as fh:
        for line in fh:
            token = line.strip().split(",")[0].strip().lower()  # tolerate 'hash,note'
            if token and not token.startswith("#"):
                out.add(token)
    return out


def scan(root, good=frozenset(), bad=frozenset(), algo="sha256"):
    """Yield (label, digest, path) for every file under `root`."""
    for dirpath, _dirs, files in os.walk(root):
        for fname in files:
            full = os.path.join(dirpath, fname)
            try:
                digest = hash_file(full, (algo,))[algo]
            except OSError:                 # unreadable / special file: skip, keep going
                continue
            if digest in bad:
                label = "KNOWN-BAD"
            elif digest in good:
                label = "KNOWN-GOOD"
            else:
                label = "UNKNOWN"
            yield label, digest, full


if __name__ == "__main__":
    ap = argparse.ArgumentParser(description="Hash-set triage of a directory tree.")
    ap.add_argument("root")
    ap.add_argument("--good", action="append", default=[], help="known-good list (repeatable)")
    ap.add_argument("--bad",  action="append", default=[], help="known-bad list (repeatable)")
    args = ap.parse_args()

    good = set().union(*(load_hashset(p) for p in args.good)) if args.good else set()
    bad  = set().union(*(load_hashset(p) for p in args.bad))  if args.bad else set()

    for label, digest, path in scan(args.root, good, bad):
        if label != "KNOWN-GOOD":           # suppress the known-innocuous noise
            print(f"{label:11} {digest}  {path}")

Usage notes.

KNOWN-BAD   2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae  /mnt/ev/Users/x/AppData/.../cache/8f1a.bin
UNKNOWN     fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9  /mnt/ev/Users/x/Documents/notes.rtf
UNKNOWN     5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9  /mnt/ev/Users/x/Downloads/setup.exe

• Known-good sets (the NIST NSRL RDS, or a hash list of a clean OS install) let you eliminate files: there is no point hand-reviewing a stock kernel32.dll. Known-bad sets (law-enforcement programs such as Project VIC and the international CAID) let you flag previously identified contraband. Both turn an impossible review into a focused one.
• Membership is O(1) because the sets live in memory; tens of millions of NSRL hashes fit comfortably. For datasets too large for RAM, back the lookup with a SQLite table or a Bloom filter.
• The standard production tools for this are hashdeep -m -k known.txt and the matching built into Autopsy, X-Ways, and EnCase. Use them for the report; use this script to understand the mechanism.

Ethics Note. Hash-set matching is the technique that lets an examiner do the most difficult work in this field — identifying known CSAM — without viewing the material, which protects both the legal integrity of the case and the examiner's own well-being against secondary trauma. Distribute and store only the hashes; never the files. Pair this script with the mandatory-reporting and well-being guidance in Chapter 28 — Ethics. This is theme #6 made concrete: the technical skill exists to serve a human need and to limit human harm.

Validation checklist (before any of this output reaches a report)

Run every script through this gate. Self-written code that has not been validated is not evidence — it is a liability.

Do, don't just read

• Build your forensic_utils.py and prove the converters. Convert a known FILETIME (e.g., 133545504620000000) by hand, then with the function; they must agree to the second.
• Carve a practice image and grade yourself. Run carve.py against a download from Appendix J — Practice Images and Lab Setup, then run foremost on the same image and compare counts and false positives. The difference is your education.
• Decode an MFT record by hand, then by script. Pull entry 0 with icat, decode the FILE header offsets yourself, and confirm parse_mft_record.py and istat all agree.
• Diff your timeline against mactime. Generate timeline.csv and mactime output from the same bodyfile; any mismatch is a bug in your understanding worth chasing.

Reference companions: Appendix A — File Signatures Reference (the headers/footers the carver uses), Appendix D — Forensic Artifact Locations (where the databases and registry hives live), Appendix G — File System Reference (the full MFT/inode structures), Appendix H — Command-Line Reference (the validated tools these scripts model), and the Glossary. The standard tools these scripts illuminate are surveyed in Chapter 36 — The Forensic Toolkit.