Chapter 26 Exercises: Linear Regression — Your First Predictive Model

How to use these exercises: Part A focuses on understanding the intuition and interpretation of linear regression without code. Part B applies regression concepts to real-world scenarios. Part C is hands-on coding with scikit-learn. Part D challenges you to think critically about when linear regression works, when it doesn't, and what the results actually mean.

Difficulty key: ⭐ Foundational | ⭐⭐ Intermediate | ⭐⭐⭐ Advanced | ⭐⭐⭐⭐ Extension

Part A: Conceptual Understanding

Exercise 26.1 — Interpreting slope and intercept ⭐

A linear regression model predicts apartment rent (in dollars per month) from apartment size (in square feet):

Rent = 450 + 1.25 * SquareFeet

1. What is the slope? Interpret it in plain English.
2. What is the intercept? Interpret it. Is this interpretation realistic?
3. What rent does the model predict for a 900 sq ft apartment?
4. What rent does the model predict for a 1,500 sq ft apartment?

Exercise 26.2 — Understanding residuals ⭐

Using the model from Exercise 26.1, compute the residual for each apartment:

1. Fill in the predicted rent and residual columns.
2. Which apartment's rent is most underestimated by the model?
3. Which apartment's rent is most overestimated?
4. Do the residuals sum to approximately zero? Why might that be expected?

Exercise 26.3 — Interpreting R-squared ⭐

For each scenario, interpret the R² value in plain English:

1. A model predicting house prices from square footage: R² = 0.72
2. A model predicting daily stock returns from yesterday's returns: R² = 0.02
3. A model predicting student test scores from hours studied: R² = 0.45
4. A model predicting a person's height from their arm span: R² = 0.95

Exercise 26.4 — Least squares intuition ⭐

Explain in your own words why linear regression minimizes the sum of squared residuals rather than the sum of absolute residuals. What practical difference does squaring make?

Exercise 26.5 — Multiple regression coefficients ⭐⭐

A multiple regression model predicts a car's fuel efficiency (MPG) from three features:

MPG = 45.0 - 0.005 * Weight_lbs - 1.2 * Engine_liters + 3.5 * Hybrid_flag

Interpret each coefficient: 1. What does -0.005 mean for Weight_lbs? 2. What does -1.2 mean for Engine_liters? 3. What does 3.5 mean for Hybrid_flag (where 1 = hybrid, 0 = not)? 4. Predict MPG for a 3,000 lb car with a 2.0L engine that is not a hybrid. 5. Predict MPG for the same car if it were a hybrid.

Exercise 26.6 — When linear regression fails ⭐⭐

For each scenario, predict whether linear regression will work well or poorly, and explain why:

1. Predicting temperature from altitude (temperature generally decreases linearly with altitude)
2. Predicting population growth from time (growth is exponential)
3. Predicting crop yield from rainfall (too little rain is bad, too much rain is bad)
4. Predicting a person's weight from their height (roughly linear in adults)

Exercise 26.7 — Multicollinearity ⭐⭐

You're building a model to predict house prices using these features: total square footage, number of bedrooms, number of bathrooms, and lot size.

1. Which pairs of features are likely to be highly correlated? Why?
2. How would multicollinearity affect your coefficient for "number of bedrooms"?
3. Would multicollinearity affect the model's overall prediction accuracy?
4. If your goal is prediction, should you worry about multicollinearity? What if your goal is explanation?

Part B: Applied Scenarios ⭐⭐

Exercise 26.8 — Choosing features for a regression model

You want to predict a restaurant's monthly revenue. You have these potential features:

1. Number of seats
2. Average Yelp rating
3. Neighborhood median income
4. Restaurant's Instagram follower count
5. Number of menu items
6. Distance to nearest subway station
7. The owner's zodiac sign
8. Average temperature that month

For each feature, state whether you'd include it in your model and why. Classify each as: "definitely include," "maybe include," or "definitely exclude."

Exercise 26.9 — Reading a regression report ⭐⭐

A data science team presents the following regression results:

Target: Employee annual salary ($)
Features: years_experience, education_level (1-5), department_code

Training R²: 0.89
Test R²: 0.42

Coefficients:
  years_experience: $2,150
  education_level: $8,300
  department_code: $1,050
  intercept: $28,000

1. What does the years_experience coefficient mean?
2. Is this model overfitting? How can you tell?
3. What's problematic about using department_code as a numerical feature?
4. What would you recommend the team do before trusting these results?

Exercise 26.10 — Baseline comparison ⭐⭐

Your linear regression model predicts daily ice cream sales with the following results:

• Baseline MAE (predict the mean): $245
• Model MAE: $210
• R²: 0.28

Your manager says: "R² of 0.28 is terrible — the model only explains 28% of the variance. We should scrap it."

Write a response defending or challenging this assessment. Consider: Is R² the right metric to focus on? Is the improvement over baseline meaningful? What domain context matters?

Exercise 26.11 — Feature engineering ⭐⭐

You're predicting house prices and have a feature "year_built" (e.g., 1965, 2003, 2020). Using year_built directly in linear regression has a problem: the model assumes a linear relationship between year and price, but very old houses might actually be worth more than moderately old houses (if they're "historic").

Suggest three ways to engineer better features from year_built:

Exercise 26.12 — Extrapolation danger ⭐⭐

Your model was trained on countries with GDP per capita between $1,000 and $65,000, and it predicts:

Vaccination Rate = 55 + 0.0004 * GDP_per_capita

1. What does the model predict for a country with GDP per capita of $50,000? Is this reasonable?
2. What does the model predict for GDP per capita of $200,000? Is this reasonable?
3. What does the model predict for GDP per capita of $300? Is this reasonable?
4. What general lesson does this illustrate about extrapolation?

Part C: Coding Exercises ⭐⭐

Exercise 26.13 — Simple linear regression from scratch

Create a scatter plot showing the relationship between study hours and exam scores, then fit a linear regression model:

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression

np.random.seed(42)
study_hours = np.random.uniform(1, 12, 50)
exam_scores = 40 + 4.5 * study_hours + np.random.normal(0, 8, 50)

Tasks: 1. Create a scatter plot of the data 2. Fit a LinearRegression model 3. Add the regression line to the scatter plot 4. Print and interpret the slope and intercept 5. What exam score does the model predict for 8 hours of studying?

import matplotlib.pyplot as plt

X = study_hours.reshape(-1, 1)
y = exam_scores

model = LinearRegression().fit(X, y)

plt.scatter(study_hours, exam_scores, alpha=0.6)
x_line = np.linspace(1, 12, 100).reshape(-1, 1)
plt.plot(x_line, model.predict(x_line), color='coral', lw=2)
plt.xlabel('Study Hours')
plt.ylabel('Exam Score')
plt.show()

print(f"Slope: {model.coef_[0]:.2f}")
print(f"Intercept: {model.intercept_:.2f}")
print(f"Predicted score for 8 hours: "
      f"{model.predict([[8]])[0]:.1f}")

Exercise 26.14 — Full workflow with train-test split

Using the study hours data from Exercise 26.13, implement the complete modeling workflow:

1. Split into 80/20 train/test sets
2. Calculate the baseline MAE (mean prediction)
3. Fit a LinearRegression on the training set
4. Evaluate on the test set (R² and MAE)
5. Compare training R² to test R² (check for overfitting)
6. Compare model MAE to baseline MAE (check for usefulness)

from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42)

baseline_mae = mean_absolute_error(
    y_test, [y_train.mean()] * len(y_test))

model = LinearRegression().fit(X_train, y_train)
y_pred = model.predict(X_test)

print(f"Training R²:  {model.score(X_train, y_train):.3f}")
print(f"Test R²:      {model.score(X_test, y_test):.3f}")
print(f"Model MAE:    {mean_absolute_error(y_test, y_pred):.2f}")
print(f"Baseline MAE: {baseline_mae:.2f}")

Exercise 26.15 — Multiple regression comparison ⭐⭐

Extend the model from Exercise 26.14 by adding features. Generate additional features and compare simple vs. multiple regression:

np.random.seed(42)
n = 200
data = pd.DataFrame({
    'study_hours': np.random.uniform(1, 12, n),
    'sleep_hours': np.random.uniform(4, 10, n),
    'previous_score': np.random.uniform(50, 100, n),
})
data['exam_score'] = (
    20 + 4 * data['study_hours'] +
    3 * data['sleep_hours'] +
    0.4 * data['previous_score'] +
    np.random.normal(0, 6, n)
)

Tasks: 1. Fit a model using only study_hours. Report test R². 2. Fit a model using all three features. Report test R². 3. How much did adding features improve R²? 4. Print the coefficients. Which feature is the strongest predictor? 5. Are any features redundant?

X1 = data[['study_hours']]
X3 = data[['study_hours', 'sleep_hours', 'previous_score']]
y = data['exam_score']

X1_tr, X1_te, y_tr, y_te = train_test_split(
    X1, y, test_size=0.2, random_state=42)
X3_tr, X3_te, _, _ = train_test_split(
    X3, y, test_size=0.2, random_state=42)

m1 = LinearRegression().fit(X1_tr, y_tr)
m3 = LinearRegression().fit(X3_tr, y_tr)

print(f"Simple R²: {m1.score(X1_te, y_te):.3f}")
print(f"Multiple R²: {m3.score(X3_te, y_te):.3f}")

for f, c in zip(X3.columns, m3.coef_):
    print(f"  {f}: {c:.3f}")

Exercise 26.16 — Residual analysis ⭐⭐

Fit a linear regression model and create diagnostic residual plots:

np.random.seed(42)
x = np.random.uniform(0, 10, 100)
y = 2 * x**2 + np.random.normal(0, 10, 100)

Tasks: 1. Fit a linear regression model to this data 2. Create a scatter plot with the regression line 3. Create a residual plot (residuals vs. predicted values) 4. What pattern do you see in the residuals? 5. What does this pattern tell you about the appropriateness of linear regression for this data?

X = x.reshape(-1, 1)
model = LinearRegression().fit(X, y)
y_pred = model.predict(X)
residuals = y - y_pred

fig, axes = plt.subplots(1, 2, figsize=(14, 5))

axes[0].scatter(x, y, alpha=0.6)
axes[0].plot(sorted(x), model.predict(
    np.sort(x).reshape(-1,1)), 'r-', lw=2)
axes[0].set_title('Data with Linear Fit')

axes[1].scatter(y_pred, residuals, alpha=0.6)
axes[1].axhline(0, color='red', linestyle='--')
axes[1].set_xlabel('Predicted')
axes[1].set_ylabel('Residuals')
axes[1].set_title('Residual Plot')

plt.tight_layout()
plt.show()

Exercise 26.17 — Log transformation ⭐⭐

When the relationship between feature and target is logarithmic (rapid change at low values, diminishing returns at high values), transforming the feature can help:

np.random.seed(42)
gdp = np.random.lognormal(9, 1.2, 120)
vacc = 30 + 8 * np.log(gdp / 1000) + np.random.normal(0, 5, 120)
vacc = np.clip(vacc, 10, 100)

Tasks: 1. Fit linear regression using raw GDP. Report R². 2. Fit linear regression using log(GDP). Report R². 3. Which model fits better? Why? 4. Create scatter plots for both models to visualize the difference.

X_raw = gdp.reshape(-1, 1)
X_log = np.log(gdp).reshape(-1, 1)

m_raw = LinearRegression().fit(X_raw, vacc)
m_log = LinearRegression().fit(X_log, vacc)

print(f"Raw GDP R²: {m_raw.score(X_raw, vacc):.3f}")
print(f"Log GDP R²: {m_log.score(X_log, vacc):.3f}")

Exercise 26.18 — Feature scaling and coefficient comparison ⭐⭐

Using the multiple regression model from Exercise 26.15, standardize the features and compare coefficients:

Tasks: 1. Fit the model on unscaled data. Print coefficients. 2. Fit the model on StandardScaler-transformed data. Print coefficients. 3. Which feature has the largest standardized coefficient? 4. Why are standardized coefficients more useful for comparing feature importance? 5. Do the model's predictions change after scaling? Verify.

from sklearn.preprocessing import StandardScaler

# Unscaled
m_raw = LinearRegression().fit(X3_tr, y_tr)
print("Unscaled coefficients:", dict(zip(X3.columns, m_raw.coef_.round(3))))

# Scaled
scaler = StandardScaler()
X3_tr_s = scaler.fit_transform(X3_tr)
X3_te_s = scaler.transform(X3_te)
m_scaled = LinearRegression().fit(X3_tr_s, y_tr)
print("Scaled coefficients:", dict(zip(X3.columns, m_scaled.coef_.round(3))))

# Compare predictions
pred_raw = m_raw.predict(X3_te)
pred_scaled = m_scaled.predict(X3_te_s)
print(f"Max prediction difference: "
      f"{np.abs(pred_raw - pred_scaled).max():.10f}")

Exercise 26.19 — Overfitting with too many features ⭐⭐⭐

Demonstrate overfitting by adding random noise features to a regression model:

np.random.seed(42)
n = 50
X_real = np.random.uniform(0, 10, (n, 2))
y = 3 * X_real[:, 0] + 2 * X_real[:, 1] + np.random.normal(0, 3, n)

Tasks: 1. Fit a model using only the 2 real features. Report train and test R². 2. Add 20 random noise features (np.random.normal columns). Fit again. Report train and test R². 3. Add 45 random noise features (almost as many as observations). Report train and test R². 4. Plot training R² and test R² as a function of number of features. 5. At what point does overfitting become severe? Why?

results = []
for n_noise in [0, 5, 10, 20, 30, 45]:
    X_noise = np.random.normal(0, 1, (n, n_noise))
    X_all = np.hstack([X_real, X_noise]) if n_noise > 0 else X_real
    Xtr, Xte, ytr, yte = train_test_split(
        X_all, y, test_size=0.2, random_state=42)
    m = LinearRegression().fit(Xtr, ytr)
    results.append({
        'n_features': 2 + n_noise,
        'train_r2': m.score(Xtr, ytr),
        'test_r2': m.score(Xte, yte)
    })

res_df = pd.DataFrame(results)
print(res_df)

Exercise 26.20 — Predicted vs. actual plot ⭐⭐

Create a "predicted vs. actual" diagnostic plot for the vaccination rate model:

np.random.seed(42)
n = 150
X = pd.DataFrame({
    'gdp': np.random.lognormal(9, 1.2, n),
    'health': np.random.uniform(2, 12, n),
    'education': np.random.uniform(0.3, 0.95, n)
})
y = 30 + 5*np.log(X['gdp']/1000) + 2*X['health'] + 25*X['education'] + np.random.normal(0, 5, n)
y = y.clip(15, 100)

Tasks: 1. Split, train, and predict 2. Create a predicted vs. actual scatter plot 3. Add the diagonal "perfect prediction" line 4. Calculate and display R² on the plot 5. Are there any obvious patterns in the errors?

Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, random_state=42)
model = LinearRegression().fit(Xtr, ytr)
ypred = model.predict(Xte)

plt.figure(figsize=(8, 8))
plt.scatter(yte, ypred, alpha=0.6)
lims = [min(yte.min(), ypred.min()), max(yte.max(), ypred.max())]
plt.plot(lims, lims, 'r--', lw=2)
plt.xlabel('Actual')
plt.ylabel('Predicted')
plt.title(f'Predicted vs Actual (R² = {r2_score(yte, ypred):.3f})')
plt.show()

Part D: Synthesis and Critical Thinking ⭐⭐⭐

Exercise 26.21 — The interpretation trap

A researcher builds a regression model predicting country life expectancy from GDP per capita. The coefficient is positive and significant. They write: "Increasing GDP per capita by $10,000 will increase life expectancy by 2.3 years."

Identify at least three problems with this conclusion. Connect your answer to concepts from both Chapter 24 (correlation/causation) and Chapter 26 (regression interpretation).

Exercise 26.22 — When to stop adding features ⭐⭐⭐

You're building a model to predict employee performance ratings. You have access to 200 features. A colleague suggests using all of them "to capture as much information as possible."

Write a paragraph explaining why this is likely a bad idea. Reference specific concepts from Chapters 25 and 26 (overfitting, bias-variance tradeoff, multicollinearity, interpretability).

Exercise 26.23 — Model comparison report ⭐⭐⭐

Build three models of increasing complexity and write a short report comparing them:

1. Model A: Simple regression with 1 feature
2. Model B: Multiple regression with 3-4 features
3. Model C: Multiple regression with 3-4 features plus polynomial terms (degree 2)

For each, report training R², test R², MAE, and whether the model is overfitting, underfitting, or well-fit. Which model would you deploy and why?

Exercise 26.24 — Real-world regression critique ⭐⭐⭐

Find a news article or blog post that presents results from a regression analysis (search for "regression analysis shows that..." or "our model predicts that..."). Critically evaluate the claims:

1. What is the target variable? What features are used?
2. Is this a prediction or explanation task?
3. Do the authors confuse correlation with causation?
4. Is R² or any evaluation metric reported? If so, is it the training or test score?
5. Could there be important confounders not included in the model?
6. Are the results being extrapolated beyond the range of the data?

Exercise 26.25 — Teaching linear regression ⭐⭐⭐⭐

Write a 200-word explanation of linear regression for someone who has never taken a statistics course. Use an analogy. Do not use any mathematical notation. Your explanation should cover: what the model does, what "best fit" means, and why we need to test on data the model hasn't seen.