Case Study: Building Your First NFL Prediction Model

A step-by-step walkthrough of creating, evaluating, and improving a real prediction model

Introduction

This case study walks through building a complete NFL prediction model from scratch. We'll start with the simplest possible approach, evaluate its performance, identify weaknesses, and iteratively improve it. By the end, you'll have a working model and understand the key decisions that affect prediction quality.

The Goal

Build a model that: 1. Predicts NFL game winners 2. Generates point spreads 3. Produces calibrated win probabilities 4. Outperforms random guessing

Phase 1: The Simplest Model

Starting Point: Pick the Home Team

The simplest "model" is to always pick the home team:

import pandas as pd
import numpy as np

def evaluate_home_team_model(games: pd.DataFrame) -> Dict:
    """
    Baseline model: always pick the home team.

    This sets the floor for any model to beat.
    """
    games = games[games['home_score'].notna()].copy()

    home_wins = (games['home_score'] > games['away_score']).sum()
    total_games = len(games)

    return {
        'model': 'Always Home',
        'correct': home_wins,
        'total': total_games,
        'accuracy': home_wins / total_games,
        'note': 'Absolute baseline - any model should beat this'
    }

# Load 2022 season data
# results = evaluate_home_team_model(games_2022)
# Typical result: ~52% accuracy

# Analysis:
# - Home teams win about 52% of games
# - This is our floor - any model must beat this
# - But this ignores team quality entirely

Results

Lesson: Simply picking home teams gives us ~52% accuracy. Any useful model must do better.

Phase 2: Adding Team Quality

The Key Insight

The biggest factor in who wins is... who's better. Let's add team ratings.

def calculate_simple_ratings(games: pd.DataFrame) -> Dict[str, float]:
    """
    Calculate team ratings from point differential.

    Rating = Average margin of victory/defeat
    """
    teams = set(games['home_team'].tolist() + games['away_team'].tolist())
    ratings = {}

    for team in teams:
        # Get all games
        home = games[games['home_team'] == team]
        away = games[games['away_team'] == team]

        # Calculate margins (adjusted for HFA of ~2.5)
        home_margins = (home['home_score'] - home['away_score'] - 2.5).tolist()
        away_margins = (away['away_score'] - away['home_score'] + 2.5).tolist()

        all_margins = home_margins + away_margins
        ratings[team] = np.mean(all_margins) if all_margins else 0

    # Normalize to mean 0
    avg = np.mean(list(ratings.values()))
    return {team: r - avg for team, r in ratings.items()}


def rating_based_prediction(home_team: str, away_team: str,
                            ratings: Dict, hfa: float = 2.5) -> Dict:
    """
    Predict game outcome using team ratings.

    Spread = Away Rating - Home Rating - HFA
    """
    home_rating = ratings.get(home_team, 0)
    away_rating = ratings.get(away_team, 0)

    spread = away_rating - home_rating - hfa
    home_win_prob = 1 / (1 + 10 ** (spread / 8))

    return {
        'predicted_winner': home_team if spread < 0 else away_team,
        'spread': round(spread, 1),
        'home_win_prob': round(home_win_prob, 3)
    }


def evaluate_rating_model(games: pd.DataFrame,
                          training_weeks: int = 8) -> Dict:
    """
    Evaluate rating-based model with proper temporal split.

    First N weeks for training, remaining weeks for testing.
    """
    # Split data
    train = games[games['week'] <= training_weeks]
    test = games[games['week'] > training_weeks]

    # Calculate ratings from training data
    ratings = calculate_simple_ratings(train)

    # Evaluate on test data
    correct = 0
    total = 0
    brier_sum = 0

    for _, game in test.iterrows():
        if pd.isna(game['home_score']):
            continue

        pred = rating_based_prediction(game['home_team'], game['away_team'], ratings)

        actual_winner = game['home_team'] if game['home_score'] > game['away_score'] else game['away_team']

        if pred['predicted_winner'] == actual_winner:
            correct += 1

        actual_home_win = 1 if actual_winner == game['home_team'] else 0
        brier_sum += (pred['home_win_prob'] - actual_home_win) ** 2

        total += 1

    return {
        'model': 'Simple Ratings',
        'training_weeks': training_weeks,
        'test_games': total,
        'correct': correct,
        'accuracy': correct / total if total > 0 else 0,
        'brier_score': brier_sum / total if total > 0 else 0
    }

Results

Improvement: +6.3 percentage points by adding team quality!

Phase 3: Identifying Problems

Problem 1: Early Season Instability

Ratings are unreliable early in the season:

def analyze_early_season_performance(games: pd.DataFrame) -> pd.DataFrame:
    """
    Analyze model performance by week.
    """
    results = []

    for test_week in range(4, 18):
        # Use all prior weeks for training
        train = games[games['week'] < test_week]
        test = games[games['week'] == test_week]

        if len(train) < 50 or len(test) < 10:
            continue

        ratings = calculate_simple_ratings(train)

        correct = 0
        for _, game in test.iterrows():
            if pd.isna(game['home_score']):
                continue

            pred = rating_based_prediction(game['home_team'], game['away_team'], ratings)
            actual = game['home_team'] if game['home_score'] > game['away_score'] else game['away_team']

            if pred['predicted_winner'] == actual:
                correct += 1

        results.append({
            'week': test_week,
            'test_games': len(test[test['home_score'].notna()]),
            'training_games': len(train),
            'accuracy': correct / len(test[test['home_score'].notna()])
        })

    return pd.DataFrame(results)

# Typical results:
# Week 4-6: ~53% accuracy (limited data)
# Week 7-12: ~57% accuracy (ratings stabilizing)
# Week 13-17: ~60% accuracy (stable ratings)

Problem 2: Not Accounting for Recent Form

Teams change during the season. A team's recent games matter more than early-season results:

def calculate_weighted_ratings(games: pd.DataFrame,
                                decay: float = 0.9) -> Dict[str, float]:
    """
    Calculate ratings with exponential decay for older games.

    More recent games weighted more heavily.
    """
    games = games.sort_values(['season', 'week'])
    teams = set(games['home_team'].tolist() + games['away_team'].tolist())
    ratings = {}

    for team in teams:
        team_games = games[
            (games['home_team'] == team) | (games['away_team'] == team)
        ].copy()

        if len(team_games) == 0:
            ratings[team] = 0
            continue

        margins = []
        weights = []

        for i, (_, game) in enumerate(team_games.iterrows()):
            if game['home_team'] == team:
                margin = game['home_score'] - game['away_score'] - 2.5
            else:
                margin = game['away_score'] - game['home_score'] + 2.5

            margins.append(margin)
            # More recent games weighted higher
            weight = decay ** (len(team_games) - i - 1)
            weights.append(weight)

        ratings[team] = np.average(margins, weights=weights)

    # Normalize
    avg = np.mean(list(ratings.values()))
    return {team: r - avg for team, r in ratings.items()}

Problem 3: Ignoring Margin of Victory Limits

A 35-point blowout doesn't mean a team is 35 points better:

def calculate_capped_ratings(games: pd.DataFrame,
                              margin_cap: int = 14) -> Dict[str, float]:
    """
    Calculate ratings with capped margins.

    Blowouts beyond cap points don't provide additional information.
    """
    teams = set(games['home_team'].tolist() + games['away_team'].tolist())
    ratings = {}

    for team in teams:
        home = games[games['home_team'] == team]
        away = games[games['away_team'] == team]

        # Cap margins
        home_margins = (home['home_score'] - home['away_score'] - 2.5).clip(-margin_cap, margin_cap)
        away_margins = (away['away_score'] - away['home_score'] + 2.5).clip(-margin_cap, margin_cap)

        all_margins = home_margins.tolist() + away_margins.tolist()
        ratings[team] = np.mean(all_margins) if all_margins else 0

    avg = np.mean(list(ratings.values()))
    return {team: r - avg for team, r in ratings.items()}

Phase 4: The Improved Model

Combining Improvements

class ImprovedNFLPredictor:
    """
    Improved prediction model incorporating lessons learned.

    Improvements over baseline:
    1. Recency weighting (decay = 0.92)
    2. Margin capping (14 points)
    3. Prior season regression
    4. Proper uncertainty handling
    """

    def __init__(self, decay: float = 0.92, margin_cap: int = 14,
                 regression_factor: float = 0.3, hfa: float = 2.5):
        self.decay = decay
        self.margin_cap = margin_cap
        self.regression_factor = regression_factor
        self.hfa = hfa
        self.ratings = {}

    def fit(self, games: pd.DataFrame,
            prior_ratings: Dict[str, float] = None) -> 'ImprovedNFLPredictor':
        """
        Fit model to game data.

        Args:
            games: Historical games
            prior_ratings: Previous season's final ratings (for regression)
        """
        games = games.sort_values(['season', 'week']).copy()
        teams = set(games['home_team'].tolist() + games['away_team'].tolist())

        for team in teams:
            team_games = games[
                (games['home_team'] == team) | (games['away_team'] == team)
            ]

            if len(team_games) == 0:
                # Use prior rating with regression
                if prior_ratings and team in prior_ratings:
                    self.ratings[team] = prior_ratings[team] * (1 - self.regression_factor)
                else:
                    self.ratings[team] = 0
                continue

            margins = []
            weights = []

            for i, (_, game) in enumerate(team_games.iterrows()):
                if game['home_team'] == team:
                    margin = game['home_score'] - game['away_score'] - self.hfa
                else:
                    margin = game['away_score'] - game['home_score'] + self.hfa

                # Cap margin
                margin = np.clip(margin, -self.margin_cap, self.margin_cap)
                margins.append(margin)

                # Recency weight
                weight = self.decay ** (len(team_games) - i - 1)
                weights.append(weight)

            current_rating = np.average(margins, weights=weights)

            # Regress to prior (if available)
            if prior_ratings and team in prior_ratings:
                prior = prior_ratings[team]
                self.ratings[team] = (
                    current_rating * (1 - self.regression_factor * 0.5) +
                    prior * self.regression_factor * 0.5
                )
            else:
                self.ratings[team] = current_rating

        # Normalize
        avg = np.mean(list(self.ratings.values()))
        self.ratings = {team: r - avg for team, r in self.ratings.items()}

        return self

    def predict(self, home_team: str, away_team: str) -> Dict:
        """Make a prediction."""
        home_r = self.ratings.get(home_team, 0)
        away_r = self.ratings.get(away_team, 0)

        spread = away_r - home_r - self.hfa
        home_wp = 1 / (1 + 10 ** (spread / 8))

        # Confidence based on rating difference
        rating_diff = abs(home_r - away_r)
        confidence = min(0.95, 0.5 + rating_diff / 20)

        return {
            'home_team': home_team,
            'away_team': away_team,
            'predicted_winner': home_team if spread < 0 else away_team,
            'spread': round(spread, 1),
            'home_win_prob': round(home_wp, 3),
            'confidence': round(confidence, 2),
            'home_rating': round(home_r, 1),
            'away_rating': round(away_r, 1)
        }

Evaluation

def comprehensive_evaluation(model, test_games: pd.DataFrame) -> Dict:
    """
    Comprehensive model evaluation.
    """
    predictions = []
    actuals = []

    for _, game in test_games.iterrows():
        if pd.isna(game['home_score']):
            continue

        pred = model.predict(game['home_team'], game['away_team'])
        actual_winner = game['home_team'] if game['home_score'] > game['away_score'] else game['away_team']
        actual_spread = game['away_score'] - game['home_score']

        predictions.append({
            'pred_winner': pred['predicted_winner'],
            'pred_spread': pred['spread'],
            'pred_prob': pred['home_win_prob']
        })

        actuals.append({
            'actual_winner': actual_winner,
            'actual_spread': actual_spread,
            'home_won': 1 if actual_winner == game['home_team'] else 0
        })

    # Calculate metrics
    correct_su = sum(1 for p, a in zip(predictions, actuals)
                     if p['pred_winner'] == a['actual_winner'])

    brier = np.mean([(p['pred_prob'] - a['home_won'])**2
                     for p, a in zip(predictions, actuals)])

    mae = np.mean([abs(p['pred_spread'] - a['actual_spread'])
                   for p, a in zip(predictions, actuals)])

    return {
        'n_games': len(predictions),
        'straight_up_accuracy': correct_su / len(predictions),
        'brier_score': round(brier, 4),
        'mae_spread': round(mae, 2),
        'vs_baseline': correct_su / len(predictions) - 0.52
    }

Phase 5: Final Results

Model Comparison

Improvement Breakdown

Key Learnings

1. Simple Models Work

A basic rating system with a few thoughtful adjustments achieves ~60% accuracy—competitive with sophisticated approaches.

2. Data Quality > Model Complexity

Proper temporal handling and feature engineering matter more than complex algorithms.

3. Evaluation Is Critical

Without rigorous evaluation, you can't distinguish real improvement from noise.

4. Marginal Gains Compound

Each small improvement (weighting, capping, regression) adds 0.5-1.5% accuracy. Combined, they make a significant difference.

5. Uncertainty Is Real

Even our best model is wrong ~40% of the time. Understanding this uncertainty is crucial.

Your Turn

Extend this model with: 1. Team-specific home field advantage 2. Rest day adjustments 3. Injury information 4. Weather factors

Track how each addition affects evaluation metrics.

Complete Code

# Full implementation available at:
# chapter-18/code/example-01-first-model.py

# Key classes:
# - SimpleRatingModel
# - ImprovedNFLPredictor
# - ModelEvaluator
# - PredictionPipeline