Further Reading: Python Transformations

Sources are tagged Tier 1 (confident it exists, recommended without reservation) or Tier 2 (real and worth seeking, but confirm the current edition, version, or URL yourself).

This is the fastest-moving material in the book. pandas 3.0, Polars 1.x, and DuckDB 1.x all landed within a couple of years of each other, and anything written before 2024 describes a different landscape — often one in which the comparison this chapter makes came out the other way. Check dates and versions on everything, including this chapter.

The primary sources

• The DuckDB documentation. Unusually good, unusually short, and the pages worth reading end to end are the CSV import page (which documents the sniffer whose behaviour Case Study 2 measured), Parquet, and Performance Guide. The memory_limit and temp_directory settings from §22.8 are on the configuration page and are the difference between a job failing and a job being slower. Tier 1 — versioned; read the page for your DuckDB version.

• The Polars user guide, particularly the "Lazy API" and "Expressions" sections. §22.3's argument is theirs, made at more length. Read the "Streaming" page too, and read it for what it says is not covered — the list changes release to release, and an unsupported operation falls back silently. Tier 1 — Polars is pre-2.0 and the guide moves with it.

• The pandas documentation's "Scaling to large datasets" page, and the Copy-on-Write page. The first is pandas' own honest account of its limits and recommends other tools by name, which is worth reading precisely because it is the maintainers saying it. The second is the migration document for pandas 3.0's biggest behaviour change. Tier 1.

• The Apache Arrow specification and the Arrow Columnar Format page. §22.5's material. You do not need the whole specification; you need the columnar format page, once, so that "zero-copy" stops being a marketing word and becomes a fact about memory layout. Tier 1.

Books

• Wes McKinney, Python for Data Analysis (3rd ed., O'Reilly, 2022). By pandas' author, and the best systematic treatment of the library. The third edition is the first one worth buying for a data engineer — it covers Arrow-backed dtypes and the modern I/O paths. It is a book about doing analysis, not about running pipelines, and reading it that way is the right frame. Tier 1 — a 4th edition covering pandas 3.0 is plausible; check.

• Matt Harrison, Effective Pandas (2021, and a second edition). Idiomatic pandas: method chaining, avoiding the copy-versus-view trap, and using categorical dtypes — which is Exercise 22.13's material. The chaining style it advocates is also what makes pandas code readable enough to port, which is an unstated benefit. Tier 2 — self-published; confirm the current edition.

• Jake VanderPlas, Python Data Science Handbook (2nd ed., O'Reilly, 2022). Free online. The NumPy chapters are the reason to read it here: pandas' memory behaviour is NumPy's memory behaviour, and §22.7's multiplier stops being mysterious once you understand what a NumPy object array actually holds. Tier 1.

On the benchmark question

• Mark Raasveldt and Hannes Mühleisen's papers on DuckDB, particularly "DuckDB: an Embeddable Analytical Database" (SIGMOD 2019) and the fair-benchmarking work. The second strand is the relevant one: Mühleisen has written repeatedly and pointedly about how database benchmarks go wrong, and §22.6's process-isolation mistake is a small instance of exactly what that literature describes. Tier 1 for the SIGMOD paper; Tier 2 for the blog posts.

• The h2oai/db-benchmark project and its successors. A maintained cross-engine benchmark on group-by and join workloads at several scales. Read the methodology before the results — the question is always whether the workload resembles yours, and the group-by-heavy shape of that suite flatters exactly the engines §22.6 also found fastest, which should make you want a second measurement rather than a confirmation. Tier 2 — the project has changed hands; verify what is current.

• Chapter 11 §11.6 of this book. The honesty checklist for a format benchmark applies unchanged to an engine benchmark, and this chapter is where you should feel the pull to skip it.

On the failures

• The Linux kernel documentation on cgroup v2 memory control. Case Study 1's material from the other end: what memory.max actually does, what memory.peak records, and why the OOM killer sends a signal that cannot be caught. Read it once and exit code 137 stops being mysterious forever. Tier 1.

• Anything careful on CSV as an interchange format. RFC 4180 exists, describes what most people mean by CSV, and is honoured inconsistently by every tool — which is the root of Case Study 2. The useful reading is the RFC plus your own reader's documentation on where it diverges, and the divergences are where your incidents come from. Tier 1 for the RFC; Tier 2 for the divergence lists, which are per-tool.

• Chapter 17 of this book, on data contracts. Case Study 2 is a schema change with no schema, from a producer who did not know they had made one. The observed-contract pattern in §17.8 is the available response when the producer is external and will not sign anything.

Practice

• code/engine_benchmark.py in this chapter. Generates its own data, runs all four engines in isolated processes, measures peak RSS, and reports a per-step breakdown for pandas. Run it on your hardware and on your own data. Exercise 22.23(a) asks you to find a case where this chapter's ordering does not hold; there will be one.

• Rewrite one production pandas job in DuckDB. Not a large one. The exercise is worth doing for the reconciliation as much as the speedup: making the two produce byte-identical output, dtypes included, will teach you more about your data than the transformation did.

• Read sys/fs/cgroup/memory.peak for one job you own, today. Compare it to the limit. Case Study 1's audit found three of eleven jobs above 60%, none of which had failed.

A note on what to be skeptical of

Any table comparing these three that does not state the workload, the data size, the hardware, and the library versions. All four change the answer, and the ranking in this chapter reversed between two data sizes on one machine.

And a specific one: "pandas is slow" is not a useful claim and this chapter did not make it. pandas was 25× slower on a group-by over a million groups and 10× slower on the same workload at a tenth the size, and it is entirely adequate for the very large number of transformations that touch fewer than a few hundred thousand rows. The decision procedure in §22.10 ends with "leave it alone" for a reason.