Quiz: Quarterback Evaluation

Target: 70% or higher to proceed.

Section 1: Multiple Choice (1 point each)

1. What does CPOE measure?

  • A) Career pass attempts over expected
  • B) Completion percentage over expected
  • C) Consistent performance over expectations
  • D) Clean pocket offensive efficiency
Answer **B)** Completion percentage over expected *Explanation:* CPOE compares a QB's actual completion percentage to what would be expected given target depth, separation, and other factors.

2. Why is raw completion percentage a limited metric?

  • A) It doesn't count attempts
  • B) It's too complicated to calculate
  • C) It's heavily influenced by target depth and scheme
  • D) It only counts completions to receivers
Answer **C)** It's heavily influenced by target depth and scheme *Explanation:* A QB throwing mostly short passes will have a higher completion percentage than one throwing deep, regardless of accuracy.

3. What is Average Depth of Target (ADOT)?

  • A) Average yards gained per catch
  • B) Average air yards on all pass attempts
  • C) Average distance to the end zone
  • D) Average yards after catch
Answer **B)** Average air yards on all pass attempts *Explanation:* ADOT measures how far downfield a QB typically throws, including incomplete passes.

4. If a QB has high EPA but low CPOE, what might explain this?

  • A) Poor accuracy compensated by good receivers (YAC)
  • B) The QB is inaccurate but lucky
  • C) EPA is calculated incorrectly
  • D) The QB only throws deep
Answer **A)** Poor accuracy compensated by good receivers (YAC) *Explanation:* Low CPOE means below-expected accuracy, but receivers gaining YAC can still produce high EPA.

5. What is the primary advantage of EPA over traditional passer rating?

  • A) EPA is easier to calculate
  • B) EPA accounts for game context (down, distance, field position)
  • C) EPA was invented more recently
  • D) Passer rating includes sacks, EPA doesn't
Answer **B)** EPA accounts for game context (down, distance, field position) *Explanation:* EPA values plays based on their contribution to scoring, considering the full game situation.

6. Why should QB evaluation adjust for opponent strength?

  • A) To make stats look better
  • B) Because QBs facing tough defenses may appear worse than they are
  • C) NFL requires it
  • D) Opponent strength doesn't affect QB play
Answer **B)** Because QBs facing tough defenses may appear worse than they are *Explanation:* A QB with a tough schedule may have lower raw stats but be equally skilled to one with an easy schedule.

7. What does a high "sack rate" indicate about a quarterback?

  • A) The QB is definitely bad
  • B) The offensive line is definitely bad
  • C) Either the QB holds the ball too long OR the line is poor
  • D) The QB throws too many interceptions
Answer **C)** Either the QB holds the ball too long OR the line is poor *Explanation:* Sacks can result from QB indecision, poor protection, or both. Context is needed to interpret.

8. Why is sample size particularly important for interception rate?

  • A) Interceptions are counted differently
  • B) Interceptions are rare events requiring many attempts to stabilize
  • C) INTs are always the QB's fault
  • D) Sample size doesn't matter for INT rate
Answer **B)** Interceptions are rare events requiring many attempts to stabilize *Explanation:* With only 2-3% of passes intercepted, random variation is high in small samples.

9. What is YAC (Yards After Catch)?

  • A) Yards the QB runs after a catch
  • B) Yards gained by the receiver after catching the ball
  • C) Yards allowed by defense after completion
  • D) Yards to first down after a catch
Answer **B)** Yards gained by the receiver after catching the ball *Explanation:* YAC measures what the receiver contributes after the catch, independent of the throw.

10. When comparing two QBs, why is it important to check confidence intervals?

  • A) Because one QB might have more attempts
  • B) To determine if differences are meaningful given sample uncertainty
  • C) Confidence intervals look professional
  • D) Confidence intervals are required by the NFL
Answer **B)** To determine if differences are meaningful given sample uncertainty *Explanation:* Overlapping CIs suggest the difference might be due to random variation, not true skill difference.

Section 2: True/False (1 point each)

11. A QB with 0.15 EPA per play is definitively better than one with 0.10 EPA per play.

Answer **False** *Explanation:* The difference might not be statistically significant. Need to check sample sizes and confidence intervals.

12. High ADOT indicates an aggressive passing style.

Answer **True** *Explanation:* A high Average Depth of Target means the QB throws deeper passes on average, indicating aggressiveness.

13. EPA fully isolates quarterback skill from supporting cast performance.

Answer **False** *Explanation:* EPA includes receiver YAC, blocking effects, and play-calling. It's team passing efficiency, not pure QB skill.

14. Completion percentage over 70% always indicates elite accuracy.

Answer **False** *Explanation:* High completion percentage can result from scheme (short passes), receivers, or accuracy. CPOE better isolates accuracy.

15. Third-down performance is typically worse than early-down performance.

Answer **True** *Explanation:* Third downs often have longer distances and more predictable passing situations, making them harder.

Section 3: Code Analysis (2 points each)

16. What does this code calculate?

pbp.query("pass == 1").groupby('passer_player_name')['cpoe'].mean()
Answer The average CPOE for each quarterback across all their pass attempts. This aggregates at the QB level, giving each QB's mean Completion Percentage Over Expected.

17. What's the issue with this QB comparison?

qb_a = pbp.query("passer_player_name == 'QB_A'")['epa'].mean()
qb_b = pbp.query("passer_player_name == 'QB_B'")['epa'].mean()
print(f"QB_A is better: {qb_a > qb_b}")
Answer **Issues:** 1. No sample size check - one QB might have few attempts 2. No statistical test for significance 3. No context adjustment (opponents, supporting cast) 4. No confidence intervals **Better approach:** 
# Check sample sizes
# Calculate confidence intervals
# Apply statistical test
# Consider context

18. What does this code accomplish?

passes['depth_bucket'] = pd.cut(passes['air_yards'],
    bins=[-20, 0, 10, 20, 100],
    labels=['Behind', 'Short', 'Medium', 'Deep'])

depth_epa = passes.groupby(['passer_player_name', 'depth_bucket'])['epa'].mean()
Answer Creates a breakdown of EPA by passing depth for each quarterback: 1. Categorizes passes into 4 depth buckets (Behind LOS, Short, Medium, Deep) 2. Calculates average EPA within each bucket for each QB 3. Allows analysis of which QBs excel at different depths This reveals passing style effectiveness.

Section 4: Short Answer (2 points each)

19. Explain why a QB might have high EPA but low CPOE (or vice versa).

Sample Answer **High EPA, Low CPOE:** - Receivers create YAC that compensates for inaccuracy - QB throws deep successfully despite low completion rate - Run-after-catch scheme generates EPA without precise throws **Low EPA, High CPOE:** - Accurate on short, low-value throws - Poor decision-making (accurate but wrong decisions) - Bad luck on turnover-worthy plays that got intercepted - Poor supporting cast (accurate throws into tight coverage)

20. What are three factors that influence QB performance but aren't captured well by standard statistics?

Sample Answer Three factors not well captured: 1. **Pre-snap reads**: Identifying coverage, making adjustments, audibling—none visible in play-by-play data 2. **Ball placement**: Throw location quality (high vs low, leading vs trailing) isn't in standard data 3. **Pocket presence**: How QB navigates pressure, extends plays, and creates throwing windows isn't quantified Others: Leadership, clutch mentality, game preparation, play-calling influence

21. Why might a QB's performance look different in the first half vs second half of a game?

Sample Answer Several reasons for half-by-half variation: 1. **Game script**: Leading teams run more; trailing teams pass more (affecting efficiency) 2. **Adjustments**: Defenses adjust at halftime; some QBs adapt better 3. **Fatigue**: Physical and mental fatigue affect late-game performance 4. **Prevent defense**: Late leads can inflate passing stats (garbage time) 5. **Time pressure**: Two-minute drills require different skills This is why situational analysis is important for QB evaluation.

Section 5: Application (3 points each)

22. Design an analysis to determine if a QB's performance decline is due to the QB or the supporting cast.

Sample Answer **Analysis Design:** 1. **Compare YAC trends**: If YAC dropped, receivers may be the issue - Calculate team YAC before/after decline - Compare to league trends 2. **Analyze pressure rate**: If sacks increased, O-line may be issue - Track sack rate and time to throw - Compare to previous seasons 3. **CPOE tracking**: If CPOE dropped, QB accuracy declined - CPOE isolates QB from supporting cast - Track CPOE trend independent of EPA 4. **Receiver separation**: If available (NGS data) - Less separation = receiver issue - Same separation + worse results = QB issue 5. **Control analysis**: Compare same QB with different personnel - Games with/without key receiver - Before/after O-line injury **Limitations**: Can't fully isolate—confounding factors always exist.

23. A front office asks: "Should we sign this free agent QB who has 0.05 EPA/play higher than our current QB?" What additional analysis would you provide?

Sample Answer **Additional Analysis Required:** 1. **Statistical significance** - Calculate confidence intervals for both QBs - Do they overlap? Is 0.05 significant? 2. **Context comparison** - Opponent-adjusted EPA - Supporting cast quality - Scheme fit 3. **Sample and stability** - How many seasons of data? - Is performance consistent or variable? 4. **Situational breakdown** - Performance in pressure situations - Third down, red zone, clutch 5. **Age and trajectory** - Career arc position - Injury history 6. **Cost-benefit** - Salary difference - How many wins does 0.05 EPA/play add? - ~0.05 × 500 passes = 25 EPA ≈ 2.5 points/season **Recommendation**: Provide full analysis showing that 0.05 difference likely isn't large enough to justify typical FA QB premium.

Section 6: Matching (1 point each)

Match the metric with what it primarily measures:

Metric Measures
24a. EPA per play A. Passing depth tendency
24b. CPOE B. Overall passing efficiency
24c. ADOT C. Accuracy above expectation
24d. Success rate D. Consistency of positive outcomes
Answers **24a. B** - EPA per play: Overall passing efficiency (points added per attempt) **24b. C** - CPOE: Accuracy above expectation (completion % vs expected) **24c. A** - ADOT: Passing depth tendency (average air yards) **24d. D** - Success rate: Consistency of positive outcomes (% of plays with positive EPA)

Section 7: Critical Thinking (2 points)

25. A QB has the highest EPA in the league but also the highest interception rate. How would you evaluate this player?

Sample Answer **Evaluation approach:** 1. **Acknowledge the tradeoff**: High-risk/high-reward style produces both outcomes 2. **Net impact analysis**: - Does EPA already account for INTs? (Yes—negative EPA for turnovers) - If still highest EPA, the positives outweigh negatives 3. **Context investigation**: - Are INTs on aggressive deep shots or bad decisions? - In what situations do INTs occur? (Score, time) - Are some "arm punts"? 4. **Stability analysis**: - Is this style sustainable? - Do high-INT QBs regress? - Injury risk from aggressive style? 5. **Game impact**: - WPA analysis: Are INTs at high-leverage moments? - Win contribution despite turnovers **Conclusion**: If EPA is highest even with INT penalty, player is net-positive. But investigate sustainability and situational context before making long-term decisions.

Scoring

Section Points Your Score
Multiple Choice (1-10) 10 ___
True/False (11-15) 5 ___
Code Analysis (16-18) 6 ___
Short Answer (19-21) 6 ___
Application (22-23) 6 ___
Matching (24) 4 ___
Critical Thinking (25) 2 ___
Total 39 ___

Passing Score: 27/39 (70%)