Quiz: Statistical Foundations for Football Analytics

Target: 70% or higher to proceed.

Section 1: Multiple Choice (1 point each)

1. What does EPA measure?

  • A) Total points scored on a play
  • B) The change in expected points before and after a play
  • C) The probability of scoring on the next play
  • D) The point value of field position
Answer **B)** The change in expected points before and after a play *Explanation:* EPA = EP_after - EP_before, measuring how much a play changed the team's expected point outcome.

2. If a team has 60% win probability at halftime, what does this mean?

  • A) They will definitely win
  • B) Based on historical data, teams in similar situations won 60% of the time
  • C) They have scored 60% of the points
  • D) They are 60% better than the opponent
Answer **B)** Based on historical data, teams in similar situations won 60% of the time *Explanation:* Win probability is based on historical outcomes of games with similar game states (score, time, possession, etc.).

3. What is the relationship between statistical significance and practical significance?

  • A) They are the same thing
  • B) Statistical significance guarantees practical significance
  • C) A result can be statistically significant but not practically significant
  • D) Practical significance is not measurable
Answer **C)** A result can be statistically significant but not practically significant *Explanation:* With large samples, tiny differences can be statistically significant. Effect size measures practical importance.

4. What is Cohen's d?

  • A) A measure of statistical significance
  • B) A standardized effect size measure
  • C) A type of regression coefficient
  • D) A probability value
Answer **B)** A standardized effect size measure *Explanation:* Cohen's d measures the difference between means in standard deviation units, allowing comparison across studies.

5. Why is the average EPA for passes higher than for runs?

  • A) Passes are counted more
  • B) Passes have higher ceiling and successful passes gain more yards/EPA
  • C) NFL rules favor passing
  • D) Run plays are always negative
Answer **B)** Passes have higher ceiling and successful passes gain more yards/EPA *Explanation:* While passes are riskier, successful passes tend to gain more yards and create more value than runs.

6. What does a p-value of 0.03 mean?

  • A) There's a 3% chance the null hypothesis is true
  • B) If the null hypothesis is true, there's a 3% chance of seeing results this extreme
  • C) The effect size is 3%
  • D) The result is 97% reliable
Answer **B)** If the null hypothesis is true, there's a 3% chance of seeing results this extreme *Explanation:* P-values represent the probability of observing the data (or more extreme) assuming the null hypothesis is true.

7. What problem does the Bonferroni correction address?

  • A) Sample size issues
  • B) Inflation of Type I error when making multiple comparisons
  • C) Non-normal data
  • D) Heteroscedasticity
Answer **B)** Inflation of Type I error when making multiple comparisons *Explanation:* When running many tests, some will appear significant by chance. Bonferroni adjusts the significance threshold.

8. In logistic regression, what do the coefficients represent?

  • A) Change in the outcome variable
  • B) Change in log-odds per unit change in predictor
  • C) Probability of success
  • D) Correlation with the outcome
Answer **B)** Change in log-odds per unit change in predictor *Explanation:* Logistic regression models log-odds. Exponentiating coefficients gives odds ratios.

9. What is regression to the mean?

  • A) All values eventually become average
  • B) Extreme values tend to be followed by less extreme values
  • C) Regression coefficients equal the mean
  • D) Mean values don't change over time
Answer **B)** Extreme values tend to be followed by less extreme values *Explanation:* Due to random variation, extreme performances include some luck and tend to regress toward average.

10. What is survivorship bias?

  • A) Only analyzing successful outcomes, missing failures
  • B) Studying only players who survived injuries
  • C) Teams that survive to playoffs are better
  • D) Statistical bias in survival analysis
Answer **A)** Only analyzing successful outcomes, missing failures *Explanation:* If we only study successful draft picks (starters), we miss busts and overestimate success rates.

11. What does Win Probability Added (WPA) measure?

  • A) Probability of winning
  • B) How much a play changed the team's win probability
  • C) Average wins added per player
  • D) Expected wins for a team
Answer **B)** How much a play changed the team's win probability *Explanation:* WPA = WP_after - WP_before, measuring each play's contribution to win probability.

12. When comparing two proportions (like catch rates), which test is appropriate?

  • A) t-test
  • B) ANOVA
  • C) z-test for proportions
  • D) Chi-square test
Answer **C)** z-test for proportions *Explanation:* When comparing proportions (binary outcomes), the z-test for proportions is appropriate.

Section 2: True/False (1 point each)

13. A 95% confidence interval means there's a 95% probability the true parameter is in that interval.

Answer **False** *Explanation:* The true parameter is fixed. A 95% CI means if we repeated the experiment many times, 95% of calculated intervals would contain the true value.

14. With a large enough sample size, any difference will be statistically significant.

Answer **True** *Explanation:* As n increases, standard error decreases, making even tiny differences detectable. This is why effect size matters.

15. EPA can be negative for a play.

Answer **True** *Explanation:* Plays that reduce expected points (turnovers, sacks, failed third downs) have negative EPA.

16. Correlation between pass rate and winning proves that passing more causes wins.

Answer **False** *Explanation:* Correlation ≠ causation. Winning teams often pass more because they have leads, not because passing causes winning.

17. Ridge regression is preferred when you want to select a subset of important features.

Answer **False** *Explanation:* Lasso regression sets coefficients to zero, performing feature selection. Ridge shrinks but doesn't zero out.

18. Bayes' theorem allows us to update probabilities as new evidence arrives.

Answer **True** *Explanation:* Bayesian updating combines prior beliefs with new evidence to form posterior probabilities.

Section 3: Calculation (2 points each)

19. A kicker makes 42 of 50 field goal attempts. Calculate: - Success rate - 95% confidence interval for true success rate

Answer **Success rate:** 42/50 = 84% **95% CI:** - p = 0.84, n = 50 - SE = √(0.84 × 0.16 / 50) = 0.0518 - z = 1.96 for 95% - CI = 0.84 ± 1.96 × 0.0518 = (0.738, 0.942) The 95% confidence interval is approximately **73.8% to 94.2%**.

20. QB A has 0.15 EPA/play on 300 passes. QB B has 0.10 EPA/play on 350 passes. Assume pooled standard deviation is 1.2. Calculate Cohen's d.

Answer **Cohen's d = (Mean₁ - Mean₂) / Pooled SD** d = (0.15 - 0.10) / 1.2 = 0.05 / 1.2 = **0.042** This is a **negligible** effect size (< 0.2). Even if statistically significant due to large n, the practical difference is minimal.

21. A team wins 70% of games when leading at halftime. They lead at halftime in 55% of games. Calculate P(Lead at Half | Win).

Answer Using Bayes' theorem: - P(Win | Lead) = 0.70 - P(Lead) = 0.55 - P(Win) = P(Win|Lead)×P(Lead) + P(Win|Trail)×P(Trail) Assume P(Win | Trail) = 0.30 (complement roughly) - P(Win) = 0.70 × 0.55 + 0.30 × 0.45 = 0.385 + 0.135 = 0.52 P(Lead | Win) = P(Win|Lead) × P(Lead) / P(Win) P(Lead | Win) = 0.70 × 0.55 / 0.52 = 0.385 / 0.52 = **0.74 (74%)**

Section 4: Short Answer (2 points each)

22. Explain why small sample sizes are problematic for evaluating player performance in football.

Sample Answer Small samples are problematic because: 1. **High variance**: A player's observed performance may differ significantly from true ability due to random variation 2. **Wide confidence intervals**: With few plays, uncertainty around estimates is large, making comparisons unreliable 3. **Regression to the mean**: Extreme performances in small samples often contain luck and won't persist 4. **Underpowered tests**: Statistical tests lack power to detect real differences with small n **Example**: A RB with 5.5 YPC on 20 carries has a 95% CI of roughly 3.5-7.5 YPC—too wide for meaningful conclusions.

23. What is selection bias and give a football example.

Sample Answer **Selection bias** occurs when the sample analyzed is not representative of the population of interest, leading to biased conclusions. **Football Example: Fourth Down Analysis** When analyzing fourth-down conversion rates, we only observe plays where teams *chose* to go for it. Teams typically go for it when: - They're confident in success (good matchup) - Situation is desperate - Distance is very short This means the observed conversion rate (around 50-60%) doesn't represent what would happen if teams went for it *randomly*. The selection of when to attempt is biased toward favorable situations. **Consequence**: We can't simply say "teams should go for it more because conversion rate is 55%"—the rate would likely drop if teams went for it in situations they currently punt.

24. Describe the difference between EPA and WPA and when each is more useful.

Sample Answer **EPA (Expected Points Added)**: - Measures change in expected points - Context-neutral (same EPA for a TD in Week 1 vs Super Bowl) - Better for evaluating pure player skill and efficiency - Use for: player evaluation, team efficiency, play-calling analysis **WPA (Win Probability Added)**: - Measures change in win probability - Context-dependent (a TD in a close game adds more WPA than in a blowout) - Captures "clutch" value and situational importance - Use for: game narratives, identifying crucial plays, clutch performance **Key difference**: A long TD in garbage time has high EPA but low WPA. A short third-down conversion in a close playoff game has low EPA but potentially high WPA. EPA measures efficiency; WPA measures impact on winning.

Section 5: Application (3 points each)

25. You find that a QB has 0.25 EPA/play vs 0.18 league average with p = 0.04. The sample is 250 plays. What additional analyses would you perform before concluding this QB is significantly better?

Sample Answer Additional analyses needed: 1. **Effect size calculation**: Cohen's d to determine if the difference is practically meaningful, not just statistically significant 2. **Confidence interval**: Calculate 95% CI for the QB's EPA to understand uncertainty range 3. **Context adjustment**: - Check if performance differs home/away - Examine opponent strength - Control for supporting cast (receivers, O-line) 4. **Situation breakdown**: - EPA by down and distance - EPA in clean pocket vs pressure - Red zone vs rest of field 5. **Time series check**: Is performance consistent across weeks or driven by outlier games? 6. **Comparison group**: How does the QB rank among starters, not vs all QBs? 7. **Regression analysis**: Control for confounding variables (play-action, receiver separation, etc.) P = 0.04 alone is insufficient—practical significance and robustness checks are essential.

26. Design a study to test whether play-action passes are more efficient than non-play-action passes. Address potential confounders.

Sample Answer **Study Design:** **Hypothesis**: Play-action passes have higher EPA than non-play-action passes **Data**: Filter PBP to pass plays, split by play_action flag **Primary analysis**: Compare mean EPA between groups using two-sample t-test **Potential Confounders**: 1. **Down and distance**: Play-action used more on early downs - Control: Stratify by down or include in regression 2. **Score differential**: Play-action less common when trailing - Control: Include score differential as covariate 3. **Personnel**: Play-action requires different formations - Control: Analyze within similar personnel groupings 4. **Team effects**: Some teams use more play-action - Control: Include team random effects or fixed effects **Regression model**: 
EPA ~ play_action + down + ydstogo + score_diff + (1|team)
**Robustness checks**: - Subset to similar situations (early downs, tied games) - Propensity score matching - Separate analysis for each team **Causal limitation**: Teams choose when to use play-action, creating selection bias. True experiment impossible.

Section 6: Matching (1 point each)

Match the statistical concept with its definition:

Concept Definition
27a. Type I Error A. Using sample data to make conclusions about a population
27b. Type II Error B. Failing to reject a false null hypothesis
27c. Statistical Inference C. Rejecting a true null hypothesis
27d. Power D. Probability of correctly rejecting a false null hypothesis
Answers **27a. C** - Type I Error: Rejecting a true null hypothesis (false positive) **27b. B** - Type II Error: Failing to reject a false null hypothesis (false negative) **27c. A** - Statistical Inference: Using sample data to make conclusions about a population **27d. D** - Power: Probability of correctly rejecting a false null hypothesis

Section 7: True Understanding (2 points each)

28. A analyst claims: "This QB's EPA of 0.22 is in the top 5, proving he's elite." What questions would you ask?

Sample Answer Questions to ask: 1. **Sample size**: How many plays? CI width? 2. **Statistical significance**: Is 0.22 significantly above average? 3. **Time period**: Full season or cherry-picked stretch? 4. **Supporting cast**: Adjusted for receiver/O-line quality? 5. **Opponents**: Quality of defenses faced? 6. **Situation mix**: Easy situations (3rd and short) vs tough? 7. **Consistency**: Steady performance or outlier games? 8. **Sustainability**: Historical EPA or one-season spike?

29. Why might observing that "teams that pass more tend to win" not mean "teams should pass more"?

Sample Answer This is a classic case of **reverse causation** and **confounding**: 1. **Reverse causation**: Teams that are *winning* tend to pass more - When trailing, teams pass to catch up - When leading, teams can pass comfortably - The relationship is: winning → passing, not passing → winning 2. **Situational confounding**: - Better teams both pass more AND win more - The correlation is due to team quality, not pass rate causing wins 3. **Game script effects**: - Trailing teams face prevent defense (easier passing) - Leading teams face stacked boxes (easier passing) 4. **Selection on success**: - Successful passes (completions, big gains) are counted - Failed passes (incompletions, sacks) lead to running - We observe pass rate conditional on passing working **Bottom line**: Correlation between pass rate and winning is largely spurious—caused by game situation, not a causal effect of passing more.

30. What's wrong with saying "Player X had a 90% catch rate, better than Player Y's 85%, so X is the better receiver"?

Sample Answer Problems with this comparison: 1. **Sample size**: How many targets? 9/10 vs 85/100 is very different - Small samples have high variance 2. **Target quality**: - X might get easy short targets - Y might run deep routes with harder catches 3. **Context differences**: - X might be a slot receiver (shorter routes) - Y might be a perimeter receiver (contested catches) 4. **QB differences**: Different QBs throwing to them 5. **Situation mix**: 3rd down catches vs garbage time 6. **Statistical significance**: Is 90% vs 85% significantly different? - With 50 targets each: CI overlap likely 7. **Value added**: Catch rate doesn't capture: - Yards after catch - EPA per target - Separation creation **Better approach**: Compare EPA per target, YPRR, or catch rate above expected (CAE) based on route difficulty.

Scoring

Section Points Your Score
Multiple Choice (1-12) 12 ___
True/False (13-18) 6 ___
Calculation (19-21) 6 ___
Short Answer (22-24) 6 ___
Application (25-26) 6 ___
Matching (27) 4 ___
True Understanding (28-30) 6 ___
Total 46 ___

Passing Score: 32/46 (70%)