Self-Assessment Quiz: Apache Spark

Twenty questions. Aim for 16 or more. Questions 8, 12, and 16 are the three that separate people who have debugged a production Spark job from people who have written one.

Question 1

Spark's fixed overheads — cluster start, shuffle to disk, JVM serialization — mean:

• A. Bigger clusters always help
• B. A job that takes 8 seconds in DuckDB may take 90 in Spark, and the 90 does not shrink with cluster size
• C. Spark is always slower
• D. Overheads scale with data

Question 2

The number of partitions is:

• A. The number of executors
• B. The number of tasks
• C. Fixed at 200
• D. The number of files

Question 3

A stage boundary is:

• A. An action
• B. A shuffle
• C. A cached DataFrame
• D. A file read

Question 4

Which is a wide transformation?

• A. filter
• B. withColumn
• C. groupBy
• D. union

Question 5

Counting shuffles in a plan means counting:

• A. Project nodes
• B. Exchange nodes
• C. Scan nodes
• D. Stages, minus one

Question 6

§21.3's plan-reading order starts with:

• A. Partition counts
• B. Exchanges
• C. Memory settings
• D. Estimated rows

Question 7

The partition-sizing rules of thumb are ~128–200 MB each and 2–4× cores. You should:

• A. Take the smaller
• B. Take the larger
• C. Average them
• D. Always use cores

Question 8

.coalesce(1) before a write in a read → filter → write job:

• A. Writes one file, no other effect
• B. Makes the entire upstream stage run with one task
• C. Adds a shuffle
• D. Fails

Question 9

The same .coalesce(1) in a read → groupBy → write job is:

• A. Equally catastrophic
• B. Harmless for the pre-shuffle stage
• C. An error
• D. Faster

Question 10

A stage where the median task takes 1.1 s and the max takes 46 minutes indicates:

• A. Capacity shortage
• B. Skew — and a bigger cluster will not help
• C. Network problems
• D. A failed executor

Question 11

The three skew fixes, in the order to try them:

• A. Salt, broadcast, filter
• B. Filter the pathological key, broadcast the other side, salt
• C. Repartition, cache, salt
• D. Add executors, salt, broadcast

Question 12

Which aggregate is not safe to compute under salting without reformulation?

• A. SUM
• B. MIN
• C. AVG
• D. MAX

Question 13

Why is that one especially dangerous?

• A. It errors
• B. It runs and returns a plausibly wrong number that nothing will fail
• C. It is slow
• D. It requires a shuffle

Question 14

AQE does not fix:

• A. Small shuffle partitions
• B. A join strategy chosen from a bad estimate
• C. Skew in a groupBy, as opposed to a join
• D. Skewed joins

Question 15

AQE could not help Case Study 2's job because:

• A. It was disabled
• B. The job has no shuffle, so there are no shuffle statistics to act on
• C. The data was too small
• D. It only works on joins

Question 16

A plain Python UDF costs roughly, relative to a built-in expression:

• A. The same
• B. 2–5×
• C. 10–100×, and it blinds the optimizer
• D. 1,000×

Question 17

A Filter node appearing above a BatchEvalPython node means:

• A. The filter runs first
• B. The optimizer could not push the filter through the UDF
• C. The plan is misprinted
• D. The UDF is vectorized

Question 18

Caching helps when:

• A. Always
• B. A DataFrame is used more than once and recomputing costs more than storing
• C. The data is large
• D. Before every write

Question 19

An empty PushedFilters: [] list on a scan means:

• A. No filters were written
• B. You are reading everything and discarding it afterwards
• C. The scan is optimal
• D. Statistics are missing

Question 20

Raising executor memory is the wrong first response to:

• A. A driver collect()
• B. An oversized partition
• C. Skew
• D. All three, though for different reasons

Answer Key

1. B — §21.1. Most of the 90 is not compute, so it does not respond to cluster size.

2. B — §21.2. And it is the single most consequential number in any Spark job.

3. B — §21.2. Which is why counting stages tells you how many times the data crossed the network.

4. C — §21.3. So do join, distinct, orderBy, repartition, and any window with a PARTITION BY.

5. B — §21.3. Each one is a full write-to-disk and read-over-network.

6. B — §21.3, 🔎 callout. Most people start at partition counts, because that is the one with a knob attached.

7. B — §21.4. Size wins on large data; cores win on small.

8. B — §21.4 and Case Study 2. coalesce inserts no shuffle, so there is no stage boundary, so the reduced parallelism extends backwards through every narrow transformation.

9. B — Case Study 2. The same line is fine in one job and disastrous in the next, and the difference is whether a shuffle sits between the source and the coalesce.

10. B — §21.5 and Case Study 1. At 2,531× the healthy tasks were never the constraint.

11. B — §21.7. Filtering is cheapest and is frequently right; salting is last because it trades a skewed shuffle for a bigger one.

12. C — Case Study 1's ⚠️ callout. COUNT(DISTINCT) and percentiles also fail, but they fail visibly.

13. B — Averaging partial averages is wrong in proportion to how uneven the salt groups are, and it will be close enough to survive a spot check.

14. C — §21.8's 🧭 callout. skewJoin is named accurately.

15. B — Case Study 2's 📐 callout. AQE is a runtime optimizer over shuffle statistics; a job with no shuffle is a job AQE cannot help.

16. C — §21.10. A pandas UDF is 2–5×; the gap between those two rows is the one that decides jobs.

17. B — §21.10. That is the optimizer telling you it gave up.

18. B — §21.11. That is the entire condition, and it is not met by the overwhelming majority of cache() calls.

19. B — §21.12. The plan says so plainly and nobody looks.

20. D — §21.9. It buys a temporary reprieve for the oversized partition and broadcast, nothing at real scale for the driver collect(), and literally nothing for skew.

Topic map