Chapter 11: File Formats and Serialization

"Every format comparison you read was run on somebody else's data. Run it on yours."

Overview

This chapter is short on opinion and long on measurement, because format choice is one of the few decisions in data engineering where you can just check.

The received wisdom is roughly correct — Parquet for analytics, Avro for streaming, JSON for interchange, CSV for humans — and it is stated in a hundred blog posts, usually with a compression ratio and a speed multiplier attached. Those numbers are properties of the data they were measured on, and §11.6 will demonstrate, using this book's own benchmark, that the same code on the same formats produces 19.3× or 13.3× depending on nothing but the cardinality of a few text fields.

So the chapter teaches a decision procedure and a measurement, rather than a table of ratios.

Two things in it will surprise you if you have absorbed the conventional advice.

Sorting before writing does not always improve compression. Chapters 8 and 9 both said it does, and both were describing the common case. §11.6 shows the exception, measured, and the exception is not rare: append-only event data already arrives in a beneficial order, and sorting can destroy it.

gzip beats Parquet+snappy on this data. Not by a lot, and it is a useful corrective to the idea that columnar always wins on size. Columnar's advantage is not primarily size — it is what you can avoid reading, which is §11.7 and which does not show up in a compression ratio at all.

In this chapter, you will learn to:

• State the two questions a format answers and use them to narrow to two candidates.
• Explain why text formats persist despite losing on every measurable axis.
• Distinguish row-binary from columnar and name the access pattern each serves.
• Explain where the schema lives in each format and what that costs at read time.
• Run a benchmark on your own data and interpret it correctly — including recognizing when your synthetic data has flattered a format.
• Predict when sorting helps and when it hurts.
• Compute the scan-cost difference, which is where the real argument is.
• Choose a codec from three properties.

Who needs this chapter: everyone, and it is on the Quick Start path. §11.6 and §11.7 are the two sections to read even if you skip the rest.

11.1 The Two Questions

Every format choice comes down to two questions, and answering them narrows nine options to two.

1. Who reads it, and how?

• A human, occasionally → text. CSV or JSON.
• A machine, whole records at a time → row-binary. Avro or Protocol Buffers.
• A machine, a few columns at a time → columnar. Parquet or ORC.

2. Where does the schema live?

• Nowhere; the reader guesses → CSV.
• In every record → JSON.
• In the file header → Avro, Parquet, ORC.
• In a registry, referenced by id → Avro with a schema registry (Chapter 17).

The second question is the one people skip, and it determines more than the first about how a system fails. A format with no schema fails at read time, in the consumer's code, months later. A format with a registry fails at write time, immediately, in the producer's code — which is Chapter 2's fail-loudly principle applied to serialization.

The nine formats, placed

Splittable matters more than it looks: a splittable file can be divided at arbitrary boundaries and processed in parallel by different workers. A single 4 GB gzipped JSON file is not splittable, so it is processed by exactly one core no matter how large your cluster is — a common and infuriating source of "why is this job not parallelizing."

11.2 Text Formats

CSV

What it gets right: universal. Every tool reads it, every human can open it, and it will still be readable in thirty years.

What it gets wrong — and the list is long enough to be worth stating, because CSV problems are a recurring category of ingestion incident:

• No types. Everything is text. 007 becomes 7, 2025-11-28 becomes a date or a string depending on the reader, and a leading + disappears.
• No schema. Column order is the contract, and nothing enforces it.
• No standard. RFC 4180 exists and is widely ignored. Delimiters, quoting, escaping, line endings, and encoding all vary.
• Nulls are ambiguous. Empty string, NULL, \N, NA, -, or nothing. Every producer picks differently.
• Embedded delimiters break naive parsers. A product name containing a comma, quoted correctly, breaks anything that splits on commas.
• No compression built in, and gzipped CSV is not splittable.

Use it for: interchange with humans and with systems you do not control. Never as an internal storage format.

JSON and JSON Lines

JSON is self-describing, handles nesting, and is universal on the web. As a data file it has one disqualifying property: a JSON array is a single value, so a 4 GB file must be parsed from the first byte to the last, by one process, before any record is available.

JSON Lines (.jsonl, ndjson) fixes exactly this: one JSON object per line, newline-delimited. It is splittable, streamable, appendable, and — this is why the book uses it for bronze landing — a corrupt line affects one record rather than the file.

Both are verbose. Every record repeats every key. Kestrel's clickstream event is 809 bytes as JSON Lines in this chapter's benchmark, of which roughly 40% is key names repeated 14 million times a day.

📐 Design Decision — Why bronze lands as JSON Lines and not as Parquet

Kestrel's bronze layer stores the raw payload as JSON in a Parquet envelope (Chapter 9 §9.4, Chapter 10 §10.9), which looks like the worst of both worlds. The reasoning:

Why not parse into Parquet columns at landing? Because you must then decide what to do with a record that does not fit the schema, and every answer loses information — Chapter 9 §9.4's argument. The raw payload as an opaque string cannot fail to land.

Why not store the file as plain JSON Lines then? Because you lose the envelope's columns — _ingested_at, _offset, event_date, customer_id — which are what make partitioning and erasure possible. You would be scanning every byte of every file to find one customer's rows.

The hybrid gets both: a Parquet file whose columns are the envelope, with one string column holding the unparsed payload. Partition pruning and erasure predicates work on the envelope; the payload survives intact.

What it costs: the payload column is a large opaque string that compresses less well than the parsed columns would, so bronze is bigger than a fully parsed Parquet table. At Kestrel that is the difference between $7.84 and perhaps $5 a month, which is not a decision.

11.3 Avro: Row-Binary

Avro stores records in a binary row format with the schema in the file header (or referenced from a registry).

Why it dominates streaming:

Schema evolution is a first-class, well-specified feature. Avro defines exactly which changes are backward compatible (a reader with an old schema reads new data), forward compatible (a reader with a new schema reads old data), or full. That specification is what makes a schema registry possible, and Chapter 17 depends on it.

It is compact. No field names in the data — the schema supplies them positionally.

It is splittable, using sync markers between blocks.

It is row-oriented, which is right for a stream: a Kafka consumer wants whole records, one at a time, not a column.

Where it loses: analytical queries. Reading one field of a million Avro records reads all fields of all million records — Chapter 7 §7.2's problem, in a file.

The practical division, which this book follows throughout:

   Kafka topic          landing              analytical storage
   ┌──────────┐        ┌──────────┐         ┌──────────────┐
   │  Avro    │───────▶│  JSONL   │────────▶│   Parquet    │
   │ + schema │        │ or Avro  │         │              │
   │ registry │        │  bronze  │         │ silver/gold  │
   └──────────┘        └──────────┘         └──────────────┘
   row, evolving,      row, raw,             columnar, typed,
   validated at        preserved             query-optimized
   write time

In words: Avro on the wire where records move whole and schemas evolve; Parquet in storage where queries read few columns of many rows. Converting between them is a normal pipeline step, not a failure of planning.

11.4 Parquet and ORC: Columnar

Parquet (Chapter 9 §9.2) is the default analytical format. Broadest support, best tooling, and the format every engine in this book reads.

ORC is Parquet's contemporary from the Hive ecosystem. Technically comparable — arguably better built-in indexing and lightweight bloom filters — and less widely supported outside the Hadoop lineage.

Choose ORC when you are in a Hive or heavily Hortonworks-descended ecosystem, or when you are using ACID Hive tables. Choose Parquet otherwise, which is nearly always, and the deciding factor is ecosystem rather than technology.

Inside a Parquet file

Everything columnar formats do well follows from one layout decision, and it is worth being able to draw:

File
├── Row Group 0                 <- the unit of parallelism AND of skipping
│   ├── Column Chunk: session_id
│   │   ├── Dictionary Page         the distinct values, once
│   │   ├── Data Page 0             + min / max / null_count statistics
│   │   └── Data Page 1
│   ├── Column Chunk: event_ts
│   └── Column Chunk: event_type
├── Row Group 1
├── Row Group 2
└── Footer                      <- schema + EVERY row group's statistics
                                   read FIRST, and it is the whole trick

A row group is a horizontal slice of the table, typically 128 MB, stored column by column inside that slice. The footer at the end holds the schema and the statistics for every column chunk in every row group, which is why a reader opens a Parquet file by seeking to the end.

Two encodings do most of the compression work before any codec runs:

Dictionary encoding. A column with few distinct values stores each value once and then stores integer references. event_type with nine distinct values across 300,000 rows becomes 300,000 small integers plus nine strings. This is why low-cardinality columns are nearly free and why a high-cardinality column like session_id is the expensive one.

Run-length and delta encoding. Repeated values collapse to a count; monotonic values store their differences. event_ts arriving in timestamp order delta-encodes almost perfectly — which is precisely the property §11.6's Finding 2 destroys by sorting.

How a reader skips work

In order, and each step is cheaper than the one after it:

1. Read the footer. Now the reader knows the schema and every row group's statistics without having read a single row.
2. Projection pushdown. Read only the column chunks the query names. SELECT event_type from a 28-column table reads one twenty-eighth of the file, and it costs nothing to arrange.
3. Predicate pushdown. Skip any row group whose min/max cannot satisfy the WHERE clause. A row group whose event_ts runs 09:00–09:14 is skipped entirely by a predicate for 14:00.
4. Page skipping. Within a chunk that survives, skip individual pages by their statistics.

Steps 2 and 3 are where the 100× in §11.7 comes from, and they behave differently: projection pushdown always works, because the column list is known statically. Predicate pushdown only works when the predicate column correlates with the physical order — statistics on a randomly-ordered column overlap on every row group and skip nothing.

🔎 Read the Plan — proving the pushdown happened

Do not assume it worked. The plan says.

text FileScan parquet [event_type,event_date] PartitionFilters: [(event_date = 2026-11-28)] <- partition pruning PushedFilters: [IsNotNull(session_id)] <- predicate pushdown ReadSchema: struct<event_type:string,...> <- projection: 2 of 28 columns

Three things to check, in this order:

ReadSchema should list the columns you asked for and no others. A SELECT * buried in a view puts all twenty-eight back, silently, and this is the most common way a well-designed table gets read badly.

PartitionFilters should be non-empty on a partitioned table. Empty means a full scan — Chapter 1's $3,840 job, and the single highest-value line in any plan in this book.

PushedFilters is a hint, not a guarantee. A filter listed there was offered to the scan; the engine may still evaluate it after reading. The byte count is the truth, and every engine reports it somewhere: Spark's scan node, BigQuery's dry run, Snowflake's partitions_scanned.

📏 Scale Note — the three sizes, and what breaks at each end

Setting	Use	Too small	Too large
Row group	128 MB	statistics per group get cheap and useless; more footer	one task per group, so parallelism collapses
Page	1 MB (default)	overhead per page	page-level skipping stops helping
File	128 MB – 1 GB	per-request and per-task overhead dominates (Ch. 9)	poor parallelism, expensive rewrites

Row group size is the one people get wrong, and it is wrong in a specific direction: a Spark job writing many small files produces many small row groups, and a 4 MB row group's statistics describe so few rows that no predicate can skip anything. The small-file problem and the no-skipping problem are the same problem, which is why Chapter 9's compaction shows up as a query-speed fix rather than a storage fix.

And at genuine scale the footer itself becomes the cost. A 10,000-column table, or a directory of 340,000 files, spends real time reading footers before it reads any data — Chapter 9's Case Study 1, from the other direction.

11.5 Where the Schema Lives

The second question from §11.1, and it determines when you find out something is wrong.

The row that matters is the registry row. Every other option finds the problem in the consumer; a registry finds it in the producer, at write time, before the bad data exists. That is a categorical difference in where the cost lands, and it is why Chapter 17 exists.

Schema-on-read versus schema-on-write

Schema-on-write validates at write time and rejects bad data. Safe, and it requires knowing the schema in advance.

Schema-on-read stores anything and interprets at read time. Flexible, and it defers every error to the consumer.

Kestrel uses both, deliberately, at different layers — which is the resolution most platforms converge on:

11.6 Measuring: Kestrel's Clickstream

Now the measurement, and it is the most useful part of the chapter.

code/format_benchmark.py generates Kestrel clickstream events with realistic cardinality and null rates, writes them in nine format-and-codec combinations, and reports the sizes. Run it yourself — the point is the method, not the numbers.

300,000 events, low-cardinality generator:

Three findings, and none of them is the one you would predict.

Finding 1: gzipped CSV beats Parquet + snappy

19.93 MB against 20.35 MB. On size alone, a text format with a general-purpose compressor beat a columnar format.

This is a useful corrective. Columnar formats are not primarily a compression technology — general-purpose compressors are very good, and applied to a text file they capture much of the same redundancy. Columnar's advantage is what you can avoid reading, which does not appear in a size comparison at all. §11.7.

The codec matters more than the layout here: Parquet + zstd is 12.56 MB against Parquet + snappy's 20.35 MB — a 1.6× difference from changing one parameter.

Finding 2: sorting made it worse

Sorted by session: 12.99 MB. Unsorted: 12.56 MB. Sorting cost 3.4%.

Chapters 8 §8.3 and 9 §9.6 both said sorting improves compression, and both were describing the common case. Here is the exception, and the mechanism is worth understanding because it is not rare:

The events were generated in timestamp order, as append-only event data always arrives. In that order, event_ts delta-encodes almost perfectly — consecutive values differ by a constant. Sorting by session_id scattered the timestamps, destroying that delta encoding, and the gain on session_id did not repay it.

📐 Design Decision — When sorting helps, and when it destroys an order you already had

The rule that survives measurement:

Sorting helps when the data arrives in no useful order. A batch extract from a database, rows from a MERGE, output of a shuffle — these have arbitrary physical order and sorting gives you both compression and pruning for free.

Sorting hurts when it destroys a beneficial order that already exists. Append-only event data arrives in time order, and time order is already the best possible layout for a delta-encoded timestamp and for date-range pruning, which is what almost every query filters on.

So: for event data, sort by time — which usually means do nothing. For a secondary access pattern like session lookup, use Z-ordering (Chapter 10 §10.5), which interleaves rather than replaces, or accept that one pattern will be slower.

And measure it. This is a 3.4% effect on this data, and it might be 30% on yours in either direction. The instruction "always sort before writing" is not wrong often enough to be dangerous and it is wrong often enough that you should check.

Finding 3: the ratio depends on the data, not the format

Re-run with --high-cardinality, which gives paths, referrers, and user-agent strings near-unique values per event — which is what real traffic looks like, with query strings, session tokens, pagination, and tracking parameters:

Same code, same formats, same codec. Only the cardinality of a few text fields changed, and the ratio fell by a third.

And notice where the high-cardinality synthetic lands: 13.3× against Kestrel's real 12.3×. Adding realistic cardinality moved the benchmark from flattering the format to nearly reproducing the production figure.

⚠️ Failure Mode — Your benchmark flattered the format

This is Chapter 8's Case Study 2 — benchmark with your messy data — occurring inside this book's own benchmark, which is why it is worth being explicit about.

A synthetic generator naturally draws categorical fields from small fixed lists, because that is how you write a generator. Small vocabularies dictionary-encode almost perfectly, so every columnar format looks better than it will on your data.

The four fields that decide a clickstream benchmark's outcome, and they are the ones a generator gets wrong:

1. URL paths. Real ones carry query strings, IDs, and tracking parameters. Nearly unique.
2. Referrers. Same, plus full search-engine query strings.
3. User agents. Hundreds of distinct version strings, not four.
4. Identifiers — session, event, device, IP. High cardinality by construction and largely incompressible.

Before quoting a compression ratio, state which data it was measured on. "Parquet is 12× smaller than JSON" is not a fact about Parquet.

11.7 The Scan-Cost Argument

Size is the wrong metric, and this is where columnar actually wins.

The query SELECT event_type, COUNT(*) FROM events WHERE event_date = '2025-11-28' GROUP BY 1 needs one column of twenty-eight.

Using the high-cardinality measurement — 856 bytes/event JSON, 64.3 bytes/event Parquet — over one day of Kestrel's 14,000,000 events:

Roughly 100× less read, for the same answer. That gap is invisible in a compression comparison and it is the entire argument.

💸 Cost Check — Scan cost, on both meters

One day of events, the one-column aggregate above, on the frozen basis.

BigQuery on-demand, $6.25/TiB:

Format	Bytes	Cost/query	×365/year
Gzipped JSON	1.08 GB	$0.0061 | $2.24
Parquet	9.4 MB	$0.000053 | $0.02

Individually negligible. Now the number that is not negligible: run the same shape across the twenty-two dashboards, each refreshing hourly (Chapter 8's Case Study 1):

$$\$0.0061 \times 24 \times 365 \times 22 = \$1{,}176 \text{ per year, against } \$10.$$

On a compute meter (Snowflake, Spark) the difference appears as runtime rather than as bytes. Decompressing and parsing 1.08 GB of gzipped JSON takes on the order of a minute of CPU; reading 9.4 MB of Parquet takes under a second. Same waste, different meter — Chapter 8 §8.7's point, and the reason the habit "do not scan what you do not need" transfers between pricing models.

11.8 Codecs

Three properties distinguish compression codecs, and you rarely get all three.

Inside Parquet and ORC, compression is per column chunk, so the file stays splittable regardless of codec. Whole-file gzip is not splittable, which is the property that makes a 4 GB .json.gz single-threaded no matter what cluster you point at it.

This book's defaults, and the reasoning:

• zstd for stored analytical data. In the §11.6 benchmark it is 1.6× smaller than snappy at comparable speed. That is close to a free win, and zstd is the one codec choice this book is confident about.
• snappy or LZ4 for intermediate data — shuffle files, temporary output — where the data is written once, read once, and deleted, so compression speed dominates ratio.
• gzip only for interchange with systems that require it.

Do not tune the compression level without measuring. zstd levels 1 through 22 span an enormous range of speed for a comparatively small range of ratio, and the default (usually 1 or 3) is close to the best trade for most data.

🧪 Try It — Find your own numbers

Twenty minutes, and the output is more useful than any table in this chapter.

bash python code/format_benchmark.py --events 300000 python code/format_benchmark.py --events 300000 --high-cardinality python code/format_benchmark.py --self-check

Then answer four questions in writing:

1. What is your JSON-to-Parquet ratio in each mode, and how far apart are they?
2. Did sorting help or hurt? Explain the result in terms of what order the data was already in.
3. How much does the codec matter compared to the layout? Compare Parquet+snappy to Parquet+zstd, and Parquet+snappy to CSV+gzip.
4. Now the one that matters: swap in a sample of your own real data. Ten thousand rows is enough. How different is the answer, and which of the four fields from §11.6's ⚠️ callout explains the difference?

🏭 From the Pipeline — the CSV that cost a quarter's revenue restatement

A supplier sent a weekly CSV of wholesale orders. It had worked for three years.

In March the supplier's export tool was upgraded, and one field changed: a quantity column that had been 1200 became 1,200. Thousands separators, correctly, for a human reading a spreadsheet.

The loader did not fail. The column was declared VARCHAR at landing — bronze keeps what it was sent — and the silver cast was TRY_CAST(quantity AS INTEGER), which returns NULL rather than raising. A null quantity multiplied by a unit price is a null line total, and SUM ignores nulls.

Wholesale revenue for the quarter was 8.4% low. Every job was green. The row count was correct — the rows were all there, with a null in one column.

Three defences existed and none of them fired.

The not_null test was on order_id, not on quantity. Somebody had chosen the columns to test and had chosen the identifiers, which is the natural choice and the wrong one — the measure is what the business reads.

The volume assertion counted rows. Rows were fine.

And the reconciliation compared against the supplier's invoice total, which was also derived from their new export. Both sides had the same thousands separators and the same intent; the reconciliation was comparing two views of one document rather than two independent measurements, which is Exercise 23.18's blind spot in its purest form.

What was added afterwards, in order of value:

text 1. a null-rate assertion on every MEASURE column, not just on keys 2. a TRY_CAST failure COUNT, published as a metric -- the cast was already counting; nobody was reading 3. an explicit column schema on the CSV read, so the cast fails LOUDLY 4. a contract with the supplier (ch 17), status: observed

Item 2 is the uncomfortable one. The information existed. The loader knew, on the first Monday in March, that 4,100 casts had failed, and it wrote the number to a log nobody had subscribed to. The failure was not detection; it was routing — which is Chapter 25 §25.9's absence problem with the signal present and unread.

🧭 Version Note — what changed in Parquet, and what a stale reader still assumes

Parquet is twelve years old and the format has moved, quietly, in ways that matter to the advice you will find written about it.

text feature arrived what stale advice assumes ───────────────────────────────────────────────────────────────────────── zstd codec 2018 snappy or gzip are the choices column indexes (page-level min/max, separate from the footer) 2019 skipping is row-group granular bloom filters 2020 point lookups need an index nanosecond timestamps 2022 microseconds, or an int96 hack float16, unsigned int logical types 2023-24 everything widens to the next signed type variant / semi-structured logical type emerging nested JSON must be a string

Two of those change advice you will read as recently as last year.

Column indexes make page-level skipping possible, so "Parquet skips at row-group granularity" is now wrong on any modern reader — and it changes the sizing advice, because a large row group is no longer as costly to over-read.

And the int96 timestamp is a legacy Impala type that persists in old files and in some writers' defaults. A reader that does not handle it gets a nonsense date rather than an error, which is the single most common "my timestamps are in 1970" question about Parquet.

What has not changed, and is worth trusting: the layout (§11.4), the pushdowns, and the fact that the footer is read first. Everything in this chapter's mental model survives every version above, which is the useful distinction — the format's structure is stable and its encodings and types are not.

The practical rule: pin pyarrow and record the version in any benchmark you publish (§11.6 does this), because a size or speed number without a library version is not reproducible and will be quoted for years.

🔁 Idempotency Check — is the same input guaranteed to produce the same file?

You will want to compare two files and conclude something from the difference. A rebuild against the same input should produce the same bytes — and for most writers, it does not.

text what varies between two runs over identical input ───────────────────────────────────────────────────────────────────── the created-by metadata string library version, always present row group boundaries depend on the input's ORDER dictionary page contents depend on encounter order the file NAME a UUID, usually compression output deterministic for a given codec and level -- this part is fine

So md5sum on two Parquet files is not a comparison of their data, and a rebuild verification built on it will report differences that are not differences.

What to compare instead, in increasing order of cost:

```sql -- 1. row count and a checksum of the SORTED data, not the file SELECT count(*), sum(hash(order_line_id, net_cents, order_date)) FROM t;

-- 2. EXCEPT in both directions -- the definitive answer (ch 20 section 20.12) SELECT * FROM new EXCEPT SELECT * FROM old; SELECT * FROM old EXCEPT SELECT * FROM new; ```

The sum(hash(...)) form is order-independent, which is the property you want: it says the rows match without asserting anything about how they were laid out.

Two practical consequences.

Sort before writing if you want stable layout, and say so in a comment — it costs a shuffle and it makes row groups reproducible, which is worth it for a table you will diff and not otherwise.

And never make a downstream decision on a file's modification time or name. Chapter 9 §9.9's rule about not parsing filenames has this second half: the name is not a version, and a rewritten file with identical contents is not a change.

🔐 Privacy & Governance — the format decides what a deletion costs

An erasure request is a rewrite, and the format determines how much gets rewritten.

text format / layout deleting one customer means ──────────────────────────────────────────────────────────────────────── JSON Lines, one file per day rewrite the whole day's file -- and you must parse it to find them Parquet, no partitioning rewrite every file containing them, found by scanning every file Parquet, partitioned by date rewrite only the days they appear in Parquet, SORTED by customer_id within the partition rewrite only the ROW GROUPS they are in -- if your tooling can (§11.4) a table format with deletion vectors (ch 10) mark them, rewrite later, in bulk

The gap between the first row and the last is two orders of magnitude of work, and the decision that produces it is made in this chapter for reasons that have nothing to do with privacy.

Two specific format facts worth knowing before Chapter 31:

A JSON payload inside a Parquet envelope (§11.9) protects the fidelity and not the person. The envelope makes the file prunable; the personal data inside the payload string still requires a rewrite of every row group containing it, and you cannot mask a column that is a substring of an opaque blob. That is the honest cost of the bronze envelope and it should be stated in the ADR.

And compression makes selective deletion harder, not easier. A column chunk is compressed as a unit, so removing one value means decompressing, rewriting, and recompressing the chunk. There is no such thing as deleting a value in place, in any format in this chapter.

The practical instruction: when you choose a partition column (Chapter 9) and a sort key (§11.6), write down what they imply for erasure. One sentence in the ADR. It is free now and it is the difference between an afternoon and a six-week project later (Chapter 31 §31.5).

📏 Scale Note — where each format stops working

Every format in this chapter is fine at a gigabyte. The differences appear at three orders of magnitude, and it is worth knowing which wall you will hit.

text format comfortable to the wall, and what it feels like ───────────────────────────────────────────────────────────────────────── CSV ~1 GB per file parsing dominates; no pushdown; a quoting bug in row 4,000,000 JSON Lines ~10 GB/day parse cost; and the gzip variant is ONE task however large it is Avro very large fine for streams; hopeless for a two-column analytical scan Parquet petabytes per-file metadata at millions of files (ch 9), and wide schemas: a 10,000-column footer is slow to read even for one column

Two of those walls are not about volume at all.

A gzipped JSON file is one task, regardless of size, because gzip is not splittable — so a 40 GB .json.gz is a single-threaded read on a hundred-node cluster. The fix is many smaller files, and people discover this at exactly the moment they are trying to go faster.

And a very wide Parquet schema costs on every read. The footer holds statistics for every column chunk in every row group; at 10,000 columns and 100 row groups that is a million statistics entries parsed to read one column. Wide tables are a real anti-pattern in columnar storage and the symptom — a slow query that reads almost no data — looks like a bug in the engine.

The number to remember: below about 128 MB per file you are paying overhead, and above about 1 GB you are losing parallelism, and that range is the same in every format and every engine in this book.

🔎 Read the Plan — the footer, from the command line

You do not need an engine to see what a Parquet file will cost to read. The metadata is in the file and three commands will show you.

bash python -c " import pyarrow.parquet as pq f = pq.ParquetFile('part-00000.parquet') print('rows ', f.metadata.num_rows) print('row groups', f.metadata.num_row_groups) print('columns ', f.metadata.num_columns) print('created by', f.metadata.created_by) for i in range(f.metadata.num_row_groups): rg = f.metadata.row_group(i) c = rg.column(0) print(i, rg.total_byte_size, c.statistics.min, c.statistics.max) "

Four things to look at, in order:

num_row_groups against the file size. A 512 MB file with one row group cannot be read in parallel and cannot skip anything. A 512 MB file with 400 row groups has 1.3 MB groups whose statistics describe too few rows to be useful. Both are visible here and in no query plan.

The first column's min and max, per row group. If they overlap heavily, the data is not sorted by that column and predicate pushdown on it will skip nothing (§11.4). This is the single most useful check in this callout, and it answers "will my filter help?" without running the filter.

created_by. The writer's library and version. It explains dictionary behaviour, timestamp encoding, and — for old files — whether you are about to meet an int96 timestamp.

And num_columns against what you query. Twenty-eight columns of which a query reads two is the 14× projection saving; 400 columns of which a query reads two is a footer that costs more to parse than the data costs to read.

The habit: run this on a file from every dataset you inherit, once. It takes a minute and it answers questions that otherwise take an afternoon of benchmarking.

11.9 Choosing

The decision procedure, applied to each place data sits in Kestrel.

Who reads it, and how?
│
├─ A human, occasionally ──────────────────────▶ CSV  (exports, interchange)
│
├─ A machine, whole records at a time
│   ├─ over a message bus, schema evolving ────▶ AVRO + schema registry
│   └─ landing raw, must not reject anything ──▶ JSON Lines, or JSON-in-Parquet
│
└─ A machine, a few columns of many rows ──────▶ PARQUET + zstd
                                                  (ORC if Hive-centric)

The three mistakes

1. Using CSV as an internal storage format. No types, no schema, no standard, ambiguous nulls. It is an interchange format. Every hour spent debugging a CSV parsing problem is an hour spent on a problem that a typed format does not have.

2. Storing analytical data in a row format. The §11.7 arithmetic. It is invisible until someone looks at scan volume, and then it is 100×.

3. Optimizing the format before the layout. Chapter 9's Case Study 1: 340,000 files of perfectly good Parquet were 19× slower than 1,712 files of the same Parquet. Format is worth a factor of a few; layout is worth a factor of tens. Fix layout first.

Nested data, and when to flatten

Clickstream events are nested and orders are not, which is the ordinary case: event payloads arrive as objects containing objects, and relational sources arrive flat.

Parquet stores nesting natively — a struct column, a list of structs, a map — using repetition and definition levels to record, for every value, how deep it was and whether it was present. You do not need to flatten to store, and this surprises people who learned columnar formats through a warehouse that made them.

Flattening is a query-side decision and it has a real cost:

one event with a 4-element `items` list
  stored nested   1 row,  items read only if projected
  exploded        4 rows, every scalar column duplicated 4x

explode multiplies every other column in the row. An event with twenty-six scalar columns and a four-element list becomes four rows of twenty-seven columns — a 4× read amplification on columns the query may not even want. On Kestrel's clickstream, exploding items before filtering turned a 9.4 MB scan into 38 MB and a two-second query into eleven.

The rule that holds: filter first, explode last. Push every predicate you can above the explode, because a row eliminated before the multiplication is a row eliminated four times.

Three cases and the answer for each:

The third row is Kestrel's bronze envelope (§11.9's table): the raw JSON payload kept whole in a string column, with event_id, event_ts, and event_date lifted out as real columns beside it. The envelope makes the file prunable and the erasure tractable (Chapter 31) without ever deciding what the payload means — which is the point of bronze.

🧱 Kestrel Platform — the formats, and the one that is not Parquet

text layer / path format codec ──────────────────────────────────────────────────────────────────────── kafka topic clickstream Avro + registry snappy s3://bronze/clickstream/event_date=… JSON payload in Parquet zstd s3://bronze/orders/ Parquet zstd s3://silver/ Parquet (Iceberg) zstd s3://gold/ Parquet (Iceberg) zstd exports to supplier-b CSV, RFC 4180, quoted gzip spark shuffle (internal) Spark's own lz4

Six of the seven rows are one decision made once. The interesting row is the export: CSV, deliberately, to a partner who cannot read anything else — and the quoting convention is written into the contract (Chapter 17) because that is the field that breaks.

The single format change with the largest measured effect was not on this list. It was compaction: the same Parquet, in 1,712 files instead of 340,000 (Chapter 9). Format is worth a factor of a few. Layout is worth a factor of tens, and the ordering of that sentence is the chapter's most useful sentence.

🎓 Interview Angle — "why is Parquet faster than JSON?"

The weak answer is "it's compressed" or "it's columnar." Both are true, and §11.6's Finding 1 is the trap waiting under the first one: gzipped CSV beat Parquet + snappy on size. A candidate who leads with compression has an answer that measurement contradicts.

The strong answer is about what is not read:

"Compression isn't really it — a general-purpose codec on a text file gets you most of the same ratio. The win is that a columnar reader can skip. The footer has the schema and per-row-group statistics, so it reads only the column chunks the query names and only the row groups whose min/max can match. On a one-column aggregate over a 28-column table that's about a hundred times less data read, and the size on disk barely moved."

The follow-up that separates people: "when is Parquet the wrong choice?" Landing raw data you must not reject; whole-record reads; write-heavy streams; and any file a human or a partner has to open. A candidate with no answer here has read about Parquet rather than used it.

11.10 Summary

Two questions narrow nine formats to two: who reads it and how (human / whole records / few columns), and where the schema lives (nowhere / in each record / in the header / in a registry). The second question determines when you find out something is wrong — and only a registry moves that from the consumer, months later, to the producer, at write time.

Splittability is the property people forget. A whole-file-gzipped JSON or CSV is processed by exactly one core regardless of cluster size. Inside Parquet, compression is per column chunk, so the file stays splittable whatever the codec.

CSV is an interchange format and never an internal one: no types, no schema, no standard, ambiguous nulls, and embedded delimiters that break naive parsers. JSON Lines fixes JSON's disqualifying property — a JSON array is one value, so a 4 GB file must be parsed end to end before any record is available — and it is what makes a corrupt line cost one record rather than a file.

Avro on the wire, Parquet in storage. Avro's schema evolution is a first-class specification, which is what makes a schema registry possible; it is compact, splittable, and row-oriented, which is right for a stream and wrong for analytics. Converting between them is a normal pipeline step.

Three findings from the measurement, and none is the predictable one:

Gzipped CSV beat Parquet + snappy on size — 19.93 MB against 20.35 MB. General-purpose compressors are very good, and columnar's advantage is not primarily compression. The codec mattered more than the layout: Parquet + zstd was 1.6× smaller than Parquet + snappy.

Sorting made compression worse by 3.4%, because the events already arrived in timestamp order and sorting by session destroyed the delta encoding on event_ts. Sorting helps when the data has no useful order and hurts when it destroys one you already had. For append-only event data, time order is usually already the best layout — so sort by time, which usually means do nothing.

The ratio is a property of the data. The same code gave 19.3× on a low-cardinality generator and 13.3× when paths, referrers, and user agents were made realistically near-unique — against Kestrel's production 12.3×. A synthetic generator draws categorical fields from small lists, small vocabularies dictionary-encode almost perfectly, and every columnar format therefore looks better than it will on your data. Before quoting a compression ratio, say which data it was measured on.

Size is the wrong metric; scan cost is the argument. A one-column aggregate over one day reads ~1.08 GB of gzipped JSON and ~9.4 MB of Parquet — roughly 100× less, for the same answer. That gap is invisible in a compression comparison. On a per-byte meter it is $1,176 a year against $10 across twenty-two hourly dashboards; on a compute meter it is a minute of CPU against under a second. Same waste, different meter.

zstd for stored data (1.6× over snappy at comparable speed — the one codec choice this book is confident about), snappy or LZ4 for intermediate data where speed dominates, gzip only for interchange. Do not tune the level without measuring.

Three mistakes: CSV as internal storage, row formats for analytical data, and optimizing format before layout — because format is worth a factor of a few and layout is worth a factor of tens.

What's next

Chapter 12 closes Part II with the stores that are neither relational nor analytical: key-value, document, wide-column, time-series, search, and vector. Real platforms have several of these, acquired one emergency at a time, and the chapter is about knowing what each is genuinely good at — which is how you avoid acquiring a seventh you did not need.