Further Reading: Apache Spark

Sources are tagged Tier 1 (confident it exists, recommended without reservation) or Tier 2 (real and worth seeking, but confirm the current edition, version, or URL yourself).

Spark has an unusually large literature and an unusually large amount of it is out of date. The 2015– 2018 material describes a Spark without AQE, frequently uses RDDs where DataFrames are now correct, and contains tuning advice the engine has since automated. Check the version before the author.

Books

• Jules Damji, Brooke Wenig, Tathagata Das, and Denny Lee, Learning Spark (2nd ed., O'Reilly, 2020). Written by Spark committers, covers the DataFrame API and Spark 3.x, and the second edition is the first one worth reading — the first edition is an RDD book and will teach you a Spark that no longer exists. The chapters on the Catalyst optimizer and on tuning are the relevant ones here. It predates the AQE improvements in 3.2+, so read §21.8 alongside it. Tier 1.

• Holden Karau and Rachel Warren, High Performance Spark (2nd ed., O'Reilly, 2023). The book this chapter is closest in spirit to. Skew, shuffles, joins, partitioning, and the cost of Python UDFs, treated at length and with measurements. Chapter 4 on joins and Chapter 6 on working with key-value data are where the salting material properly lives, and Karau's treatment is more careful than §21.7's about when salting is and is not correct. Tier 1 — confirm the second edition; the first (2017) predates AQE entirely.

• Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), Chapter 10. The chapter on batch processing explains MapReduce and its descendants from first principles, including why a shuffle is expensive and why skew is the dominant failure mode of partitioned computation. It is not a Spark book and it will improve your Spark more than most Spark books. Tier 1.

The primary sources

• The Spark SQL Performance Tuning guide, in the official documentation. Short, and the single highest-value page: partition sizing, broadcast thresholds, AQE configuration, and the coalesce behaviour §21.4 warns about. Read the AQE section specifically — the defaults changed in 3.2 and a lot of blog advice predates it. Tier 1 — versioned; read the 3.5 page.

• The Spark Web UI documentation. Undersold and rarely read. It explains what every column on the stage page means, including Shuffle Spill (Disk) and the task-duration summary that constitutes this chapter's two most useful diagnostics. If you take one thing from this chapter's reading list, make it half an hour with the stage page and this document open together. Tier 1.

• The Spark configuration reference. Worth skimming once so you know what exists. Most of it you will never set; the fraction you will — shuffle.partitions, autoBroadcastJoinThreshold, the adaptive.* family, sql.files.maxPartitionBytes — is worth knowing by name. Tier 1.

• The Delta Lake documentation on OPTIMIZE and Z-ordering. Case Study 2's actual fix. Chapter 10 introduced these; this is where you go when a directory has 730,000 files in it. Tier 2 — the Delta docs move; confirm against your Delta version.

On the specific failures

• Any careful treatment of the small-files problem on object storage. The good sources are usually vendor engineering blogs rather than books, and the thing to look for is one that decomposes the cost — listing, per-file metadata round trips, task scheduling, and request throttling — rather than asserting that small files are bad. Case Study 2's arithmetic is that decomposition; do it for your own storage layer, because the constants differ by an order of magnitude between object stores and HDFS. Tier 2 — vendor blogs; verify the numbers against your own system.

• The AWS S3 documentation on request rate and performance. The reason Case Study 2's per-file overhead is superlinear rather than constant: request rates are limited per prefix, and past the limit you get 503s and exponential backoff. This is the single most surprising thing about small files at scale and it is documented plainly by the vendor. Tier 1.

• Anything on Zipf and power-law distributions in operational data. §21.7 draws a line between a genuine heavy tail (salt it) and a miscategorised entity (fix the model), and drawing that line well requires knowing what a natural heavy tail looks like. The practical version: plot your key's frequency rank against frequency on log-log axes. A straight line is Zipf; a single point far off a straight line is Case Study 1. Tier 2 — the statistics literature is large; you need one figure from it.

On the alternatives

• The DuckDB and Polars documentation, and Chapter 22. §21.1's argument depends on knowing what a single machine can do now, and the honest way to find out is to run your job both ways on the same data. Most people's mental model of "too big for one machine" was formed some years ago and has not been updated, which is the specific error §21.1 is trying to correct. Tier 1 for the docs; the measurement is on you.

• Any recent benchmark comparing single-node engines to distributed ones — read skeptically. Chapter 11 §11.6's honesty checklist applies unchanged: who ran it, on what data, with what cardinality, and does the workload resemble yours? The benchmarks in this space are unusually motivated, in both directions. Tier 2 by nature.

Practice

• code/spark_advisor.py in this chapter. Reads a plan and counts the four things §21.3 says to count, sizes partitions both ways, and runs the median-versus-max skew diagnostic — with no Spark installed, on text you already have. Point it at a plan from a job you own today.

• pyspark in local mode. SparkSession.builder.master("local[4]") reproduces skew, coalesce serialization, UDF costs, and shuffle counts faithfully on a laptop. Everything in Part B of the exercises runs locally, which removes the usual reason people never experiment with this material.

• Read one production plan a week. Not to fix anything — just to build the habit of counting exchanges and checking pushdowns. The two case studies here were each found in about fifteen seconds by someone who finally looked, and the skill being exercised is looking.