Appendix G: File Formats and Compression

Every figure in this appendix is measured on Kestrel's clickstream — 14,000,000 events/day — using chapter-11-*/code/. Your data will differ; the shape of the differences will not.

G.1 The Formats

*Protobuf is splittable only with a container format around it.

The decision is one line: row-oriented for writes and whole-record reads, columnar for analytical scans. Everything else is detail.

G.2 Size, Measured

One day of Kestrel clickstream, 14,000,000 events, 28 columns:

Two columns of the table carry the argument.

The size column is why the annual figure is 4.19 TB of JSON and 341 GB of Parquet.

The last column is the one that matters more. A two-column scan is 1.9 s in Parquet and 94 s in JSON — a 49× difference — because a row format must read and parse every byte to reach two fields. That is projection pushdown, and it is free.

G.3 Codecs

Two things people get wrong.

"gzip is not splittable" is about raw files, not Parquet. A .json.gz is one task no matter how big it is. Parquet compresses per column chunk, so the file remains splittable whatever codec you pick. A 40 GB .json.gz is one task and a 40 GB gzip-Parquet is not.

The codec is the small decision. JSON → Parquet is 11.3×; snappy → zstd is a further 1.09×. Getting the format right is worth ten times more than tuning the codec, and it is the tuning people do.

G.4 Parquet Internals

File
├── Row Group 0        <- the unit of parallelism and of skipping
│   ├── Column Chunk: order_id
│   │   ├── Dictionary Page
│   │   ├── Data Page 0    + min/max/null_count statistics
│   │   └── Data Page 1
│   └── Column Chunk: net_cents
├── Row Group 1
└── Footer             <- schema + every row group's statistics. Read FIRST.

How a reader skips work:

1. Read the footer. Now it knows the schema and every row group's statistics.
2. Projection pushdown: read only the requested column chunks.
3. Predicate pushdown: skip row groups whose min/max cannot match.
4. Within a chunk, skip pages by their statistics.

This is why SELECT * is expensive and why sort order matters: statistics only skip when the predicate column is correlated with the physical order.

Sizing:

G.5 The Small File Problem

Kestrel's clickstream, one day, written three ways:

11× the query time and 3,600× the requests, for the same bytes.

Three causes: a streaming sink writing per micro-batch, a Spark job with shuffle.partitions=200 over a small dataset, and partitioning by a high-cardinality column.

The fix is compaction (ch9), and the cost argument for it is weak — Chapter 33 prices Kestrel's at $2,131 a year. The latency argument and the deletion argument (ch31) are the strong ones.

G.6 Schema Evolution

"Silently" in the JSON and CSV column is the whole problem with schemaless formats. A field that was a number and is now a string does not fail; it produces a different type downstream, on a Tuesday.

CSV cannot survive a column being added, because readers address columns positionally when the header is absent and by name when it is — and both are broken by an insertion in the middle.

G.7 Choosing, in Five Lines

landing raw events from a stream          JSON Lines or Avro, compressed
                                           -- keep what the source sent (ch34)
anything you will query analytically      Parquet + zstd
a Kafka topic                             Avro or Protobuf + a schema registry
an interchange file for a vendor          CSV, and document the quoting
a file another system reads by contract   whatever they specified; wrap it (ch30)

And the conversion is usually the first thing worth doing. Kestrel lands raw JSON for fidelity and converts to Parquet in the same DAG — 12.3× on storage, 49× on a two-column scan, and the raw file stays for the argument you cannot otherwise win.

G.8 The Numbers Worth Remembering

JSON -> Parquet                    ~12x smaller
two-column scan, JSON -> Parquet   ~49x faster
snappy -> zstd                     ~1.09x smaller, negligible time
86,400 small files -> 24           ~11x faster query, 3,600x fewer requests
row group                          128 MB
file size                          128 MB - 1 GB

And the one that decides most arguments: at Chapter 33's rate card, Kestrel's entire 27.4 TiB of storage costs $644.76 a month, which is 2.1% of the bill. Format choice is a compute optimization that happens to save storage, and arguing it on storage loses.