Exercises: File Formats and Serialization

The single most valuable exercise here is 11.20(d): run the benchmark on your own data. Everything else is preparation for interpreting what it tells you.

Difficulty: ⭐ warm-up · ⭐⭐ standard · ⭐⭐⭐ deeper. Solutions: daggered (†) and odd-numbered problems are in appendices/answers-to-selected.md.

Part A — Warm-ups ⭐

11.1 † State the two questions a format choice comes down to. Which one determines when you find out something is wrong?

11.2 Why does a whole-file-gzipped 4 GB JSON file use exactly one core no matter how large your cluster is?

11.3 † List six things CSV gets wrong. Which one has bitten you, or seems most likely to?

11.4 What is JSON Lines' one advantage over JSON that makes it usable for landing? Give the consequence for a corrupt record.

11.5 † Why does Avro dominate streaming and lose at analytics? One sentence each.

11.6 Where does the schema live in each of: CSV, JSON, Avro with a registry, Parquet? For each, say where a mismatch surfaces.

11.7 † In the §11.6 benchmark, which format beat Parquet + snappy on size, and what does that tell you about what columnar formats are actually for?

11.8 State the rule for when sorting before writing helps and when it hurts.

Part B — Standard ⭐⭐

11.9 Run python code/format_benchmark.py --events 300000 and record every row. Then run it with --high-cardinality. Produce a table comparing the two, and write two paragraphs explaining the difference in terms of the four fields named in §11.6's ⚠️ callout.

11.10 † §11.6 found sorting cost 3.4%. Design an experiment that makes sorting help on the same data: change what order the generator produces rows in, or change the sort key, and measure. Report what you changed and by how much it helped. Then state the general rule your result supports.

11.11 Redo the §11.7 scan-cost arithmetic for a different query: SELECT customer_id, COUNT(DISTINCT session_id) FROM events WHERE event_date BETWEEN ... GROUP BY 1 over 30 days. Two columns of twenty-eight, 420 million events. Compute bytes read and BigQuery cost for gzipped JSON Lines and for Parquet + zstd. Then compute the same on a Snowflake compute meter, stating your assumption about throughput.

11.12 † The book lands bronze as a JSON payload inside a Parquet envelope. Write the three alternatives (plain JSON Lines, fully parsed Parquet, Avro) and for each state: what is gained, what is lost, and the specific Kestrel requirement it fails.

11.13 Take a real dataset you have access to — a log file, an export, an API response dump. Write it in at least four of the formats in §11.1 and measure. Then answer: which format would you choose for storage, which for interchange, and what surprised you?

11.14 † §11.8 says "do not tune the compression level without measuring." Measure: run the benchmark's Parquet writer at zstd levels 1, 3, 9, and 19. Report size and write time for each. Where is the knee, and what would you set?

11.15 A partner requires a daily full-table export of fct_order_item — 6,483,117 rows, 18 columns. They can read CSV, gzipped CSV, or Parquet. Recommend one and defend it, considering: their tooling, transfer cost at $0.09/GB egress, and what happens when a product name contains a comma.

Part C — Deeper ⭐⭐⭐

11.16 §11.6 found the same code produced 19.3× and 13.3× depending only on field cardinality. Design a "benchmark honesty checklist" — the questions to ask of any published format comparison before believing it. At least six items, each specific enough to be answerable from a benchmark's description.

11.17 † Chapter 8 §8.3 and Chapter 9 §9.6 both claimed sorting improves compression, and Chapter 11 measured an exception. Both earlier claims now carry a pointer to §11.6. Find another claim in this book that you suspect is stated too generally, design the measurement that would test it, and — if you can — run it.

11.18 Protocol Buffers appear in §11.1 and nowhere else. Research where they fit relative to Avro: schema evolution model, wire format, framing, and ecosystem. Then say why a data platform usually chooses Avro and a service mesh usually chooses protobuf.

11.19 † Arrow is listed as a format and described as "in-process interchange, zero-copy." Explain what zero-copy means here and why it matters. Then identify two places in this book's pipeline where an Arrow-based path would eliminate a serialization round trip.

Part D — The Kestrel Platform ⭐⭐⭐

11.20 — Increment 11: the format decision, measured.

(a) Run the benchmark in both modes and record the output verbatim in platform/docs/format-benchmark.md.

(b) Add a section to that file: the format decision for each place data sits in the Kestrel platform, using §11.9's table as a model, with one sentence of justification each. Where your answer differs from the book's, say why.

(c) Compute the scan-cost argument (§11.7) for Kestrel's three most common query shapes, on both meters. Record it.

(d) The exercise that matters. Swap in a sample of your own real data — ten thousand rows of anything: an application log, an export, a table you have access to. Re-run. Record the ratio and compare it to both synthetic modes.

Then write the paragraph you would send to a colleague who quotes a compression ratio from a blog post.

11.21 † Add platform/storage/format_policy.md: the platform's format rules as a short policy — what is used where, what is forbidden, and the one condition that would make you revisit each choice. Keep it under 400 words. Chapter 17 turns the wire-format half of this into a contract.

Reflection

A. §11.6's ⚠️ callout is this book's own benchmark falling into the trap Chapter 8's Case Study 2 warned about. Does finding that in a book you are reading make you trust it more or less? What would the alternative — quietly using the flattering number — have cost you?

B. The chapter says format is worth a factor of a few and layout is worth a factor of tens. Where else have you seen effort go to the more visible, smaller-payoff optimization?