Further Reading: File Formats and Serialization

Sources are tagged Tier 1 (confident it exists, recommended without reservation) or Tier 2 (real and worth seeking, but confirm the current edition, version, or URL yourself).

Format specifications are unusually good reading — precise, short, and written by people who had to decide something and explain why. Most of this list is specifications.

The specifications

The Apache Parquet format specification (parquet.apache.org/docs/file-format/), and the parquet-format repository's README and LogicalTypes.md. Recommended in Chapter 9 and again here for the encoding material: the pages on RLE, dictionary, delta, and byte-stream-split encodings are what §11.6's measurements are explained by. LogicalTypes.md is where you settle arguments about how a timestamp or decimal is actually stored, which matters when two engines disagree about one file. Tier 2 — the docs site moves; the specification is stable.
The Apache Avro specification (avro.apache.org/docs/), particularly "Schema Resolution." This is the section that defines backward, forward, and full compatibility, and it is the foundation Chapter 17's schema registry is built on. Read it before Chapter 17, not during. Ten pages, precise, and one of the better-written specifications in this space. Tier 1.
RFC 4180, "Common Format and MIME Type for Comma-Separated Values Files" (2005). Two pages, and the most useful thing about it is discovering how much it does not specify — it says nothing about encoding, nulls, dates, or numeric formats, which is exactly the list in this chapter's second case study. Read it once so you know what "valid CSV" means and how little that guarantees. Tier 1.
The Apache ORC specification (orc.apache.org/specification/). Worth skimming alongside Parquet's, because the two solve the same problem with different index structures — ORC's built-in bloom filters and row indexes versus Parquet's row group statistics plus optional bloom filters. The comparison teaches you which parts are essential. Tier 2.
The Protocol Buffers language guide and encoding documentation (protobuf.dev). Read the "Encoding" page specifically: varints, wire types, and field numbers. It explains why protobuf evolution rules are what they are — field numbers are the identity, names are not — which is a genuinely different model from Avro's and is the answer to Exercise 11.18. Tier 1.
The Apache Arrow columnar format specification. The in-memory counterpart to Parquet's on-disk format, and the reason "zero-copy" is possible between processes. Relevant to Exercise 11.19 and to understanding why pyarrow sits under so much of this book's tooling. Tier 1.

On compression

The Zstandard documentation and Yann Collet's benchmarks (facebook.github.io/zstd/). The compression-ratio-versus-speed frontier, with real numbers across levels 1–22. Read it before tuning a compression level, and it will probably persuade you not to. Tier 1.
The lzbench benchmark suite. A comparison of dozens of compressors on standard corpora. Useful as a reference for the shape of the trade-off; as this chapter's §11.6 demonstrates, the specific ratios are properties of the corpus and not of the codec. Tier 2.
David Salomon, Data Compression: The Complete Reference. If you want to understand why dictionary and run-length encoding behave the way they do on the columns in §11.6's table, rather than taking it on faith. Comprehensive, dry, and the reference rather than the introduction. Tier 2 — check the edition.

On benchmarking honestly

Mark Raasveldt et al., "Fair Benchmarking Considered Difficult: Common Pitfalls In Database Performance Testing" (2018), DBTest. Recommended in Chapter 8 and mandatory here. Eight pitfalls, and this chapter's Case Study 1 is pitfall-adjacent in a way the paper does not quite cover — sampling bias in the data rather than in the queries — which makes it worth reading with that gap in mind. Tier 1.
Any vendor's format comparison. Read two from competing vendors and note what each chose to measure and what data they used. The exercise is the point: after §11.6 you should be able to predict which choices produced which result. Tier 2, emphatically.

On the practical problems

The Python csv module documentation, particularly the Dialect class and the Sniffer. The Dialect attributes are, almost exactly, the seven unspecified properties from this chapter's second case study — which is a nice confirmation that the problem is well known and that the standard library's answer is "tell us which of the seven you mean." Tier 1.
The "Falsehoods Programmers Believe About..." genre, particularly the entries on CSV, on names, on addresses, and on time. Not rigorous and genuinely useful as a checklist of assumptions you did not know you were making — several of which appear in Case Study 2's seven failures. Tier 2 — scattered blog posts, findable by title.
The Unicode Consortium's material on encodings, and the "UTF-8 Everywhere" manifesto. Failure 6 in Case Study 2 is a UTF-8-versus-CP1252 mismatch producing 3Mâ„¢ with no error anywhere. Understanding why that specific mojibake pattern appears makes it recognizable on sight, which is worth the twenty minutes. Tier 2.

If you only read one thing

Do not read — run. Take this chapter's code/format_benchmark.py, replace the generator with a sample of your own real data, and measure.

Ten thousand rows is enough. The number you get will differ from every number in this chapter and from every number in every blog post, and understanding why it differs — using the per-column diagnostic from Case Study 1 — is the entire skill this chapter is trying to teach.

If you want reading as well, read the Avro schema resolution specification before Chapter 17. It is the one piece of format documentation that this book's later chapters actively depend on.