Chapter 11 — Key Takeaways (File Formats and Serialization)

The page for a format decision, and for reading somebody else's benchmark.

The two questions

1. Who reads it, and how?

a human, occasionally ─────────────────▶ CSV / JSON
a machine, whole records ──────────────▶ AVRO (+ registry) / Protocol Buffers
a machine, few columns of many rows ───▶ PARQUET (ORC if Hive-centric)

2. Where does the schema live? — this determines when you find out something is wrong

Splittable = can be divided at arbitrary boundaries and processed in parallel. A whole-file gzipped JSON/CSV is not, so it uses one core regardless of cluster size. Inside Parquet, compression is per column chunk — splittable whatever the codec.

Text formats

CSV leaves seven properties unspecified — delimiter, quoting, escaping, embedded newlines, encoding, null representation, date format. Three consumers → seven failures, one per property. Interchange format, never internal.

JSON's disqualifying property: an array is one value, so a 4 GB file is parsed end-to-end before any record is available. JSON Lines fixes exactly this — splittable, appendable, and a corrupt line costs one record rather than a file.

Avro on the wire, Parquet in storage

Kafka: AVRO + registry ──▶ landing: JSONL / JSON-in-Parquet ──▶ silver/gold: PARQUET + zstd
row, evolving, validated       row, raw, preserved              columnar, typed
at WRITE time

Avro's schema-evolution specification is what makes a registry possible (Ch. 17). It is row-oriented, so reading one field of a million records reads all fields of all million.

Schema-on-read vs. schema-on-write, by layer

The measurement — 300k Kestrel clickstream events

Three findings, none predictable

1. Gzipped CSV beat Parquet + snappy. Columnar's advantage is not primarily compression — general-purpose compressors capture much of the same redundancy. The codec mattered more than the layout: zstd was 1.6× smaller than snappy.

2. Sorting made it 3.4% WORSE. The events already arrived in timestamp order, which delta-encodes almost perfectly; sorting by session scattered them.

Sorting helps when the data has no useful order and hurts when it destroys one it already had. For append-only event data, sort by time — which usually means do nothing. Use Z-ordering for a secondary access pattern, and measure.

3. The ratio is a property of the DATA.

⚠️ Four fields decide a clickstream benchmark, and a generator gets all four wrong: paths · referrers · user agents · identifiers. Small vocabularies dictionary-encode almost perfectly, so every columnar format looks better than it will on your data.

LIMIT n is not a sample — it is the first n, and the first n has a shape.

Before quoting a compression ratio, say which data it was measured on.

Scan cost — where columnar actually wins

One column of twenty-eight, one day, 14M events:

~100× less, for the same answer — invisible in a compression comparison.

$1,176/year vs. $10 across 22 hourly dashboards on a per-byte meter. On a compute meter it appears as runtime instead: a minute of CPU against under a second. Same waste, different meter.

Codecs

zstd for stored data · snappy/LZ4 for intermediate (write once, read once) · gzip only for interchange. Do not tune the level without measuring.

Choosing — Kestrel

The three mistakes

1. CSV as an internal storage format
2. Row formats for analytical data — the 100×
3. Optimizing format before layout — format is worth a factor of a few, layout a factor of tens. Fix layout first.

The ten-second diagnostic

f = pq.ParquetFile(path); rg = f.metadata.row_group(0)
for i in range(rg.num_columns):
    c = rg.column(i)
    print(f"{c.path_in_schema:20} {c.total_compressed_size/1e6:7.2f} MB "
          f"{c.total_uncompressed_size/c.total_compressed_size:6.1f}x")

Compare per-column sizes, not totals. A column with a ratio near 1× is incompressible — random ids, hashes, IPs — and dominates the file. Check whether anything downstream needs it.