Exercises: Apache Spark

Several of these need a Spark session. pyspark in local mode (spark = SparkSession.builder.master("local[4]")) is enough for everything except the cluster-sizing questions, and local mode reproduces skew, coalesce, and UDF costs faithfully.

code/spark_advisor.py runs the plan reader, the partition sizer, and the skew diagnostic with no Spark installed at all — it works on the text explain() prints.

Difficulty: ⭐ warm-up · ⭐⭐ standard · ⭐⭐⭐ deeper. Solutions: daggered (†) and odd-numbered problems are in appendices/answers-to-selected.md.

Part A — Warm-ups ⭐

21.1 † Give the data-size bands from §21.1 and the three non-size reasons to use Spark. Which of the three do people apologize for, and should they?

21.2 Name the four nouns of the execution model. Which one is the number of tasks?

21.3 † Define narrow and wide. Give three examples of each, and say what a stage boundary is.

21.4 How do you count the shuffles in a job? What is the node to look for?

21.5 † State §21.3's four-step plan-reading order. Which step do most people start at, and why?

21.6 Give both partition-sizing rules of thumb and say which wins when. What is Spark's default, and what does it give you at 341 GB?

21.7 † What does coalesce do that repartition does not, and what does that cost you?

21.8 State the fifteen-second skew diagnostic and the ratio that separates skew from capacity.

21.9 † Name the three skew fixes in the order you should try them. What does salting cost?

21.10 Give the three things AQE does and the three it does not.

Part B — Standard ⭐⭐

21.11 Take any job you have and run df.explain(mode="formatted"). Count the Exchange nodes, the join strategies, and the empty PushedFilters lists. Then run spark_advisor.py --plan yourplan.txt and compare with what you found by hand.

21.12 † Reproduce §21.3's four-shuffle example, then reduce it to two. Report the plan before and after, and the elapsed time. State which of the two shuffles you removed and why it was removable.

21.13 Construct a skewed dataset: one key with 40% of the rows and a long tail of ordinary ones. Group by it. Report the median and max task duration, then double local[n] and report both again. Confirm the max does not move.

21.14 † Fix the skew from 21.13 three ways — filter the key, broadcast the other side, and salt — and report elapsed time for each. Then say which you would ship, and what you would write in the comment above it.

21.15 Take the salted aggregation from 21.14 and add an AVG. Compute it two ways: averaging the partial averages, and carrying sum and count. Report both numbers and the error. Then explain why nothing failed.

21.16 † Reproduce Case Study 2. Build a narrow pipeline — read, filter, write — and add .coalesce(1). Report the task count and the duration before and after. Then insert a groupBy before the coalesce and report again, and explain the difference.

21.17 Write the same transformation three ways: a built-in expression, a pandas UDF, and a plain Python UDF. Run each over at least a million rows and report the ratios. How close are they to §21.10's magnitudes?

21.18 † Write a job whose PushedFilters list is empty, then fix it so it is not. Report the input bytes read in both cases from the Spark UI, not from the plan.

Part C — Deeper ⭐⭐⭐

21.19 §21.1 argues Kestrel's dim_product job should not have been on Spark, and also that a team of two might reasonably keep it there. Price both sides for a system you know: the compute saved, and the cost of a second execution engine in your DAG. Which way does it come out, and what would change the answer?

21.20 † Case Study 1's skew was a distributor modelled as a customer. Write the query that finds candidates for the same problem in your own data — a key whose distribution suggests it is conflating two kinds of entity — and state honestly what its false-positive rate would be.

21.21 Design the check that would have caught Case Study 2 on day one. It must fire on a 4.2× regression that still finishes inside the SLA, and it must not fire on a legitimate backfill. State what it costs and what it will miss.

21.22 † §21.11 claims caching usually does not help. Falsify it: construct a job where caching produces a large, reproducible speedup, and identify exactly which of the section's conditions your job satisfies. Then construct one where caching makes a job fail, and explain the mechanism.

Part D — The Kestrel Platform ⭐⭐⭐

21.23 — Increment 21: the clickstream sessionizer on Spark.

(a) Write platform/spark/sessionize.py implementing Chapter 18 §18.9's sessionization over silver.events, including the overlap read and the emit-forward filter.

(b) Use one window specification for both window functions, and verify by counting Exchange nodes. Report the count.

(c) Set spark.sql.shuffle.partitions from the data, with the arithmetic in a comment. Enable all three AQE settings, each with a comment saying what it buys.

(d) Add a pre-aggregation skew check: median versus max partition size, failing above 10×.

(e) Run the job with spark.sql.shuffle.partitions at 200 and at your computed value. Report elapsed time, disk spill, and cost for both. This is the exercise that carries the chapter.

(f) Size the write separately from the compute: repartition to a file-size target computed from the day's volume, not a constant.

21.24 † Write platform/spark/tests/test_sessionize.py covering: a session that straddles the window boundary is emitted once and only once; a session starting in the overlap is not emitted; the gap parameter and the overlap satisfy overlap > gap; and the job is idempotent on re-run.

The boundary test is the one to write first. It is the assertion Chapter 18's Case Study 2 needed and did not have, and getting it right in Spark means constructing the fixture deliberately — a session with events at 23:50 and 00:05 will not appear by accident.

Reflection

A. Both case studies in this chapter involve a competent engineer applying correct knowledge and producing a bad outcome. In Case Study 1 the knowledge was "slow job, add capacity"; in Case Study 2 it was "coalesce is cheaper than repartition." What do those two have in common, and what would a book have to do differently to prevent them?

B. §21.1's honest answer to "should this be a Spark job?" is often no, and yet Spark's share of data engineering job postings does not reflect that. What forces keep a tool in use past the point where it is the best answer, and which of them are legitimate?