Chapter 17: Exercises - Introduction to Predictive Analytics

Overview

These exercises build fundamental skills in predictive modeling for football analytics. Progress from basic classification to complete ML pipelines.

Level 1: Fundamentals (Recall and Basic Application)

Exercise 1.1: Classification vs Regression

Categorize each prediction problem as classification or regression.

Problems: 1. Predicting whether a team will win their next game 2. Predicting how many points a team will score 3. Predicting the defensive coverage on a play 4. Predicting a player's rushing yards per game 5. Predicting whether a fourth down conversion will succeed 6. Predicting a quarterback's passer rating 7. Predicting which team will win the national championship 8. Predicting the point spread margin

Starter Table: | Problem | Type | Reasoning | |---------|------|-----------| | 1. Win next game | | | | 2. Points scored | | | | ... | | |

Exercise 1.2: Train-Test Split

Implement a proper train-test split for football game data.

Task: Split game data while avoiding data leakage.

Starter Code:

import pandas as pd
import numpy as np

def temporal_train_test_split(games_df: pd.DataFrame,
                              test_year: int = 2023,
                              target_col: str = 'home_win'):
    """
    Split games into train/test sets based on season.

    Parameters:
    -----------
    games_df : pd.DataFrame
        Game data with 'season' column
    test_year : int
        First year to include in test set
    target_col : str
        Name of target column

    Returns:
    --------
    tuple : (X_train, X_test, y_train, y_test)
    """
    # TODO: Split data by season
    # TODO: Separate features from target
    # TODO: Return split data

    pass

# Test with sample data
np.random.seed(42)
sample_games = pd.DataFrame({
    'season': np.random.choice([2020, 2021, 2022, 2023], 200),
    'home_epa': np.random.normal(0.1, 0.1, 200),
    'away_epa': np.random.normal(0.05, 0.1, 200),
    'home_win': np.random.choice([0, 1], 200)
})

X_train, X_test, y_train, y_test = temporal_train_test_split(sample_games)
print(f"Training games: {len(X_train)}")
print(f"Test games: {len(X_test)}")

Exercise 1.3: Basic Logistic Regression

Train a logistic regression model to predict game outcomes.

Task: Build a simple win prediction model.

Starter Code:

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

def train_win_predictor(X_train, y_train, X_test, y_test):
    """
    Train logistic regression for win prediction.

    Returns:
    --------
    dict : Model and evaluation metrics
    """
    # TODO: Initialize model

    # TODO: Fit model

    # TODO: Make predictions

    # TODO: Calculate accuracy

    pass

Exercise 1.4: Evaluate Model Performance

Calculate multiple evaluation metrics for a classifier.

Task: Implement evaluation function.

Starter Code:

from sklearn.metrics import (accuracy_score, precision_score,
                            recall_score, f1_score, confusion_matrix)

def evaluate_classifier(y_true, y_pred):
    """
    Calculate classification metrics.

    Returns:
    --------
    dict : All metrics
    """
    # TODO: Calculate accuracy
    # TODO: Calculate precision
    # TODO: Calculate recall
    # TODO: Calculate F1 score
    # TODO: Generate confusion matrix

    pass

Exercise 1.5: Feature Scaling

Implement feature scaling for model input.

Task: Use StandardScaler appropriately.

from sklearn.preprocessing import StandardScaler

def scale_features(X_train, X_test):
    """
    Scale features using StandardScaler.

    Important: Fit scaler on training data only!

    Returns:
    --------
    tuple : (X_train_scaled, X_test_scaled, scaler)
    """
    # TODO: Initialize scaler
    # TODO: Fit on training data
    # TODO: Transform both train and test

    pass

Level 2: Application (Apply Concepts to New Situations)

Exercise 2.1: Cross-Validation Pipeline

Implement k-fold cross-validation for model evaluation.

Task: Build a complete CV pipeline.

from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.linear_model import LogisticRegression

def cross_validate_model(X, y, n_folds=5):
    """
    Perform stratified k-fold cross-validation.

    Parameters:
    -----------
    X : np.ndarray
        Feature matrix
    y : np.ndarray
        Target vector
    n_folds : int
        Number of folds

    Returns:
    --------
    dict : CV results with mean, std, and all fold scores
    """
    # TODO: Create StratifiedKFold splitter
    # TODO: Initialize model
    # TODO: Run cross_val_score
    # TODO: Return results dictionary

    pass

Exercise 2.2: Feature Engineering for Games

Create predictive features from raw game statistics.

Task: Engineer features that capture team strength.

def engineer_game_features(team_stats: pd.DataFrame,
                          schedule: pd.DataFrame) -> pd.DataFrame:
    """
    Create features for game prediction.

    Required features:
    1. Differential features (home - away)
    2. Rolling averages (last 5 games)
    3. Opponent-adjusted metrics
    4. Home field indicator

    Returns:
    --------
    pd.DataFrame : Games with engineered features
    """
    # TODO: Merge team stats for home teams
    # TODO: Merge team stats for away teams
    # TODO: Calculate differentials
    # TODO: Add home advantage indicator

    pass

Exercise 2.3: Model Comparison

Compare multiple models on the same dataset.

Task: Train and compare logistic regression vs. random forest.

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

def compare_models(X, y):
    """
    Compare multiple classification models.

    Models to compare:
    1. Logistic Regression
    2. Random Forest (n_estimators=100)

    Returns:
    --------
    pd.DataFrame : Comparison of model performance
    """
    models = {
        'Logistic Regression': LogisticRegression(max_iter=1000),
        'Random Forest': RandomForestClassifier(n_estimators=100, random_state=42)
    }

    results = []

    for name, model in models.items():
        # TODO: Cross-validate each model
        # TODO: Store results

        pass

    return pd.DataFrame(results)

Exercise 2.4: Probability Calibration Assessment

Evaluate how well-calibrated model probabilities are.

Task: Create a calibration assessment function.

from sklearn.calibration import calibration_curve
import matplotlib.pyplot as plt

def assess_calibration(y_true, y_prob, n_bins=10):
    """
    Assess probability calibration.

    Returns:
    --------
    dict : Brier score, ECE, and calibration data
    """
    # TODO: Calculate Brier score
    # TODO: Calculate calibration curve
    # TODO: Calculate Expected Calibration Error (ECE)
    # TODO: Create calibration plot

    pass

Exercise 2.5: Baseline Comparison

Compare model performance to baselines.

Task: Implement baseline models for comparison.

def create_baselines(y_train, y_test):
    """
    Create baseline predictions.

    Baselines:
    1. Random guess (50%)
    2. Majority class
    3. Historical home team win rate

    Returns:
    --------
    dict : Baseline accuracies
    """
    # TODO: Random baseline
    # TODO: Majority class baseline
    # TODO: Historical rate baseline

    pass

Level 3: Analysis (Break Down and Examine)

Exercise 3.1: Feature Importance Analysis

Analyze which features drive predictions.

Task: Extract and visualize feature importance.

import matplotlib.pyplot as plt

def analyze_feature_importance(model, feature_names, top_n=10):
    """
    Analyze and visualize feature importance.

    Works with:
    - Logistic Regression (coefficients)
    - Random Forest (feature_importances_)

    Returns:
    --------
    pd.DataFrame : Ranked feature importance
    """
    # TODO: Extract importance scores
    # TODO: Create ranking DataFrame
    # TODO: Create visualization
    # TODO: Return top features

    pass

Exercise 3.2: Error Analysis

Analyze where the model makes mistakes.

Task: Identify patterns in prediction errors.

def analyze_errors(X_test, y_test, y_pred, feature_names):
    """
    Analyze prediction errors.

    Analysis:
    1. Characteristics of correctly predicted games
    2. Characteristics of incorrectly predicted games
    3. Features that differ between correct/incorrect

    Returns:
    --------
    dict : Error analysis results
    """
    # TODO: Split into correct and incorrect
    # TODO: Compare feature distributions
    # TODO: Identify error patterns

    pass

Exercise 3.3: Temporal Performance Analysis

Analyze how model performance changes over time.

Task: Track accuracy by season and week.

def temporal_performance(games_df, predictions, actuals):
    """
    Analyze performance over time.

    Questions:
    1. Does accuracy vary by season?
    2. Does accuracy improve as season progresses?
    3. Are there systematic biases early vs late season?

    Returns:
    --------
    pd.DataFrame : Performance by time period
    """
    # TODO: Calculate accuracy by season
    # TODO: Calculate accuracy by week
    # TODO: Identify trends

    pass

Exercise 3.4: Overfitting Detection

Detect and diagnose overfitting.

Task: Create functions to detect overfitting.

def detect_overfitting(model, X_train, y_train, X_test, y_test):
    """
    Detect overfitting by comparing train/test performance.

    Indicators:
    1. Large train-test accuracy gap
    2. Perfect training accuracy
    3. Learning curves

    Returns:
    --------
    dict : Overfitting diagnosis
    """
    # TODO: Calculate training accuracy
    # TODO: Calculate test accuracy
    # TODO: Assess gap
    # TODO: Generate learning curves

    pass

Exercise 3.5: Sample Size Analysis

Analyze impact of sample size on model performance.

Task: Create learning curves to understand data requirements.

from sklearn.model_selection import learning_curve

def analyze_sample_size(model, X, y, train_sizes):
    """
    Analyze how sample size affects performance.

    Returns:
    --------
    dict : Learning curve data and visualization
    """
    # TODO: Generate learning curve data
    # TODO: Plot training and validation scores
    # TODO: Identify minimum viable sample size

    pass

Level 4: Synthesis (Combine Elements in New Ways)

Exercise 4.1: Complete Prediction Pipeline

Build an end-to-end prediction pipeline.

Task: Create a class that handles the entire ML workflow.

class GamePredictionPipeline:
    """
    Complete pipeline for game outcome prediction.

    Components:
    1. Data preparation
    2. Feature engineering
    3. Model training with CV
    4. Evaluation
    5. Prediction on new games
    """

    def __init__(self, model_type='logistic'):
        # TODO: Initialize components
        pass

    def prepare_data(self, raw_data: pd.DataFrame) -> pd.DataFrame:
        # TODO: Clean and prepare data
        pass

    def engineer_features(self, data: pd.DataFrame) -> pd.DataFrame:
        # TODO: Create features
        pass

    def train(self, X: pd.DataFrame, y: pd.Series) -> dict:
        # TODO: Train with cross-validation
        pass

    def evaluate(self, X_test: pd.DataFrame, y_test: pd.Series) -> dict:
        # TODO: Generate evaluation report
        pass

    def predict(self, new_games: pd.DataFrame) -> pd.DataFrame:
        # TODO: Make predictions on new data
        pass

Exercise 4.2: Ensemble Model Builder

Create an ensemble combining multiple models.

Task: Build a voting classifier with multiple base models.

from sklearn.ensemble import VotingClassifier

def build_ensemble(X_train, y_train, X_test, y_test):
    """
    Build ensemble model combining multiple classifiers.

    Ensemble components:
    1. Logistic Regression
    2. Random Forest
    3. Gradient Boosting

    Returns:
    --------
    dict : Ensemble model and performance
    """
    # TODO: Define base models
    # TODO: Create VotingClassifier
    # TODO: Train ensemble
    # TODO: Compare to individual models

    pass

Exercise 4.3: Hyperparameter Optimization

Optimize model hyperparameters.

Task: Implement grid search for hyperparameter tuning.

from sklearn.model_selection import GridSearchCV

def optimize_hyperparameters(X, y, model_type='random_forest'):
    """
    Find optimal hyperparameters using grid search.

    Returns:
    --------
    dict : Best parameters and performance
    """
    param_grids = {
        'random_forest': {
            'n_estimators': [50, 100, 200],
            'max_depth': [5, 10, 20, None],
            'min_samples_split': [2, 5, 10]
        },
        'logistic': {
            'C': [0.001, 0.01, 0.1, 1.0, 10.0],
            'penalty': ['l1', 'l2']
        }
    }

    # TODO: Set up GridSearchCV
    # TODO: Fit grid search
    # TODO: Extract best parameters
    # TODO: Evaluate best model

    pass

Exercise 4.4: Model Monitoring System

Build a system to monitor model performance over time.

Task: Create monitoring infrastructure.

class ModelMonitor:
    """
    Monitor prediction model performance.

    Features:
    1. Log predictions and outcomes
    2. Calculate rolling accuracy
    3. Detect performance degradation
    4. Alert when retraining needed
    """

    def __init__(self, model_name: str, baseline_accuracy: float):
        # TODO: Initialize monitoring state
        pass

    def log_prediction(self, game_id: str, prediction: dict):
        # TODO: Store prediction
        pass

    def log_outcome(self, game_id: str, actual: int):
        # TODO: Store actual outcome
        pass

    def get_performance_report(self, window: int = 50) -> dict:
        # TODO: Calculate recent performance
        pass

    def check_for_drift(self, threshold: float = 0.05) -> dict:
        # TODO: Compare recent vs historical performance
        pass

Exercise 4.5: Interactive Prediction Dashboard

Build a simple dashboard for predictions.

Task: Create functions for a prediction interface.

def create_prediction_interface(model, team_stats: pd.DataFrame):
    """
    Create interface for making predictions.

    Features:
    1. Select home team
    2. Select away team
    3. Display prediction with confidence
    4. Show feature contributions
    """
    # TODO: Create team selection
    # TODO: Generate prediction
    # TODO: Display results

    pass

Level 5: Evaluation (Judge and Justify)

Exercise 5.1: Model Selection Justification

Choose and justify the best model for a scenario.

Scenario: A college athletic department wants to predict game outcomes to set expectations with boosters.

Task: 1. Train multiple models 2. Evaluate each on relevant criteria 3. Write a recommendation memo explaining your choice

Criteria to consider: - Accuracy - Calibration of probabilities - Interpretability for stakeholders - Computational requirements - Maintenance burden

Exercise 5.2: Ethical Considerations

Evaluate ethical implications of football predictions.

Task: Write an analysis addressing: 1. Impact on gambling and integrity concerns 2. Privacy of player data used in models 3. Transparency of predictions to affected parties 4. Potential for biased predictions

Exercise 5.3: Feature Selection Debate

Evaluate different approaches to feature selection.

Task: Compare three approaches and recommend one: 1. Domain expertise only (manually selected features) 2. Statistical selection (correlation-based) 3. Model-based selection (importance from random forest)

Include: - Pros and cons of each approach - When each is appropriate - Your recommendation with justification

Exercise 5.4: Validation Strategy Critique

Evaluate a proposed validation strategy.

Proposed Strategy: "We'll use 5-fold cross-validation on all games from 2018-2023 to evaluate our model."

Task: 1. Identify problems with this approach 2. Propose an improved strategy 3. Explain why your approach is better

Exercise 5.5: Production Readiness Assessment

Evaluate whether a model is ready for production deployment.

Task: Create a production readiness checklist and evaluate a model against it.

Model Details: - Logistic regression for game predictions - 65% accuracy on test set (2023 season) - Trained on 2018-2022 data - Uses 12 features - No monitoring implemented

Deliverable: - Readiness checklist (10+ items) - Assessment of current state - Recommendations for production deployment

Submission Guidelines

Code Requirements

• All Python code must be executable
• Include docstrings for all functions
• Follow PEP 8 style guidelines
• Include type hints

Analysis Requirements

• Show your reasoning
• Compare to baselines
• Discuss limitations

Report Requirements

• Clear structure
• Data-driven conclusions
• Actionable recommendations

Grading Rubric