Chapter 27 Exercises: Logistic Regression and Classification — Predicting Categories

How to use these exercises: Part A tests your understanding of classification concepts, the sigmoid function, and error types. Part B applies these concepts to real-world scenarios where the costs of errors differ. Part C involves coding logistic regression models and evaluating them. Part D challenges you with threshold selection, class imbalance, and ethical considerations.

Difficulty key: ⭐ Foundational | ⭐⭐ Intermediate | ⭐⭐⭐ Advanced | ⭐⭐⭐⭐ Extension

Part A: Conceptual Understanding

Exercise 27.1 — Regression vs. classification ⭐

Classify each problem as regression or classification. Explain your reasoning.

1. Predicting the temperature tomorrow in degrees Fahrenheit
2. Predicting whether it will rain tomorrow (yes/no)
3. Predicting a student's letter grade (A, B, C, D, F)
4. Predicting how many minutes a flight will be delayed
5. Predicting whether a tumor is malignant or benign
6. Predicting a restaurant's star rating (1-5 stars)

Exercise 27.2 — The sigmoid function ⭐

Without running code, determine the approximate output of the sigmoid function for each input:

1. sigmoid(0)
2. sigmoid(5)
3. sigmoid(-5)
4. sigmoid(100)
5. sigmoid(-100)

Then explain in plain language what the sigmoid function does and why it's useful for classification.

Exercise 27.3 — Reading a confusion matrix ⭐

A spam filter produces the following confusion matrix:

                Predicted
              Spam    Not Spam
Actual Spam    180       20
   Not Spam     30      770

1. Identify the TP, FP, TN, and FN values
2. Calculate the accuracy
3. Calculate the precision for spam detection
4. Calculate the recall for spam detection
5. What type of error is more harmful here: false positives or false negatives?

Exercise 27.4 — Precision vs. recall ⭐⭐

For each scenario, state whether you would prioritize precision or recall, and explain why:

1. A fire alarm system in a hospital
2. A recommendation engine suggesting movies
3. A test for detecting COVID-19 during a pandemic
4. An automated hiring tool that screens resumes
5. A self-driving car's pedestrian detection system

Exercise 27.5 — The threshold effect ⭐⭐

A model produces the following probability predictions for 10 patients being tested for a disease:

1. Using a threshold of 0.5, classify each patient and compute the confusion matrix
2. Using a threshold of 0.3, classify each patient and compute the confusion matrix
3. How did recall change between the two thresholds? Precision?
4. Which threshold is more appropriate for medical screening and why?

Exercise 27.6 — Class imbalance awareness ⭐⭐

A credit card fraud detection model achieves 99.8% accuracy. The dataset contains 0.2% fraudulent transactions.

1. What accuracy would a model that always predicts "not fraud" achieve?
2. Is the 99.8% accuracy meaningful? Why or why not?
3. What metrics would you use instead of accuracy?
4. If the model catches 70% of fraud but falsely flags 0.5% of legitimate transactions, is it useful? Calculate the numbers for a day with 100,000 transactions, 200 of which are fraudulent.

Exercise 27.7 — Why linear regression fails for classification ⭐

In your own words, explain two specific problems with using linear regression (instead of logistic regression) for a binary classification problem. Use concrete examples.

Part B: Applied Scenarios ⭐⭐

Exercise 27.8 — Designing a classification system

You're building a system to predict whether online reviews are genuine or fake. Fake reviews can be either overly positive (fake 5-star reviews) or overly negative (fake 1-star reviews from competitors).

1. Define the target and features for this classification problem
2. What would the baseline accuracy be if 15% of reviews are fake?
3. Should you prioritize precision or recall? Consider both the platform's perspective and the reviewer's perspective.
4. How might false positives (flagging genuine reviews as fake) harm the platform?
5. How might false negatives (letting fake reviews through) harm the platform?

Exercise 27.9 — Interpreting logistic regression output ⭐⭐

A logistic regression model predicts whether a loan applicant will default. The model outputs:

Coefficients:
  income:        -0.0002
  credit_score:  -0.015
  loan_amount:    0.00005
  prior_defaults:  1.8

Intercept: 5.2

1. Interpret the sign of each coefficient (which features increase default probability?)
2. Which feature has the strongest association with default? (Be careful about scale.)
3. A customer has income=$80,000, credit_score=720, loan_amount=$25,000, prior_defaults=0. Calculate the logit score and the predicted probability.
4. Same customer but with 2 prior defaults. How does the probability change?

Exercise 27.10 — The cost of errors in healthcare ⭐⭐

A classification model screens mammograms for breast cancer. Results on a test set of 10,000 mammograms (200 have cancer):

Confusion Matrix:
                Predicted
              Cancer    No Cancer
Actual Cancer   170        30
   No Cancer    800      9,000

1. Calculate accuracy, precision, and recall
2. This model has low precision. What does that mean in practice?
3. This model has high recall. What does that mean in practice?
4. Is this tradeoff acceptable for a screening test? Why or why not?
5. What happens if you raise the threshold to improve precision?

Exercise 27.11 — Comparing models with different error profiles ⭐⭐

Two models predict customer churn:

1. Which model catches more actual churners?
2. Which model has fewer false alarms?
3. If it costs $50 to contact a customer for retention but a lost customer costs $500 in lost revenue, which model saves more money?
4. Explain why accuracy alone is insufficient for choosing between these models.

Exercise 27.12 — Probability calibration ⭐⭐

Your logistic regression model says a customer has a 0.72 probability of churning. Your manager asks: "So there's a 72% chance they'll leave?"

1. Is this interpretation correct? What does the 0.72 actually mean?
2. How would you test whether the model's probabilities are well-calibrated?
3. Why are calibrated probabilities important for business decisions?

Part C: Coding Exercises ⭐⭐

Exercise 27.13 — Basic logistic regression

Fit a logistic regression model to predict whether a student passes a course:

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

np.random.seed(42)
n = 200
students = pd.DataFrame({
    'study_hours': np.random.uniform(0, 15, n),
    'attendance_pct': np.random.uniform(40, 100, n),
})
logit = -4 + 0.3 * students['study_hours'] + 0.04 * students['attendance_pct']
students['passed'] = (np.random.random(n) < 1/(1+np.exp(-logit))).astype(int)

Tasks: 1. Split into 80/20 train/test sets 2. Calculate the baseline accuracy (most frequent class) 3. Fit a LogisticRegression model 4. Report accuracy on train and test sets 5. Print the confusion matrix 6. Calculate precision and recall for "passed"

from sklearn.metrics import confusion_matrix, precision_score, recall_score

X = students[['study_hours', 'attendance_pct']]
y = students['passed']

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42)

baseline = y_train.value_counts().max() / len(y_train)
print(f"Baseline accuracy: {baseline:.3f}")

model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

print(f"Train accuracy: {model.score(X_train, y_train):.3f}")
print(f"Test accuracy:  {model.score(X_test, y_test):.3f}")
print(f"Confusion matrix:\n{confusion_matrix(y_test, y_pred)}")
print(f"Precision: {precision_score(y_test, y_pred):.3f}")
print(f"Recall:    {recall_score(y_test, y_pred):.3f}")

Exercise 27.14 — Using predict_proba

Using the model from Exercise 27.13:

1. Get probability predictions for the test set
2. Print the probabilities for the first 10 test observations alongside actual values
3. Sort test observations by probability and identify the 5 most uncertain predictions (probabilities closest to 0.5)
4. Create a histogram of predicted probabilities

proba = model.predict_proba(X_test)[:, 1]

print("First 10 predictions:")
for i in range(10):
    print(f"  P(pass)={proba[i]:.3f}, Actual={y_test.iloc[i]}")

# Most uncertain
uncertainty = np.abs(proba - 0.5)
uncertain_idx = np.argsort(uncertainty)[:5]
print("\nMost uncertain:")
for i in uncertain_idx:
    print(f"  P(pass)={proba[i]:.3f}, Actual={y_test.iloc[i]}")

plt.hist(proba, bins=20, edgecolor='black', alpha=0.7)
plt.xlabel('P(Pass)')
plt.ylabel('Count')
plt.title('Distribution of Predicted Probabilities')
plt.show()

Exercise 27.15 — Threshold exploration ⭐⭐

Using the model from Exercise 27.13, explore how different thresholds affect classification performance:

Tasks: 1. Calculate precision, recall, and F1-score for thresholds from 0.2 to 0.8 (step 0.1) 2. Create a plot showing precision and recall vs. threshold 3. At what threshold are precision and recall approximately equal? 4. If "missing a student who needs help" (false negative) is worse than "over-flagging students for support" (false positive), what threshold would you recommend?

from sklearn.metrics import f1_score

thresholds = np.arange(0.2, 0.85, 0.1)
precisions, recalls, f1s = [], [], []

for t in thresholds:
    preds = (proba >= t).astype(int)
    precisions.append(precision_score(y_test, preds, zero_division=0))
    recalls.append(recall_score(y_test, preds, zero_division=0))
    f1s.append(f1_score(y_test, preds, zero_division=0))

plt.plot(thresholds, precisions, 'o-', label='Precision')
plt.plot(thresholds, recalls, 's-', label='Recall')
plt.plot(thresholds, f1s, '^--', label='F1')
plt.xlabel('Threshold')
plt.ylabel('Score')
plt.legend()
plt.title('Precision-Recall Tradeoff')
plt.show()

Exercise 27.16 — Confusion matrix visualization ⭐⭐

Create a visually clear confusion matrix heatmap with labels:

Tasks: 1. Fit a logistic regression model on the student data 2. Create a confusion matrix using sklearn 3. Visualize it as a heatmap with counts and labels 4. Add a title showing the overall accuracy

from sklearn.metrics import confusion_matrix

y_pred = model.predict(X_test)
cm = confusion_matrix(y_test, y_pred)

fig, ax = plt.subplots(figsize=(6, 5))
im = ax.imshow(cm, cmap='Blues')
plt.colorbar(im)

labels = ['Failed', 'Passed']
ax.set_xticks([0, 1])
ax.set_yticks([0, 1])
ax.set_xticklabels(labels)
ax.set_yticklabels(labels)
ax.set_xlabel('Predicted')
ax.set_ylabel('Actual')

for i in range(2):
    for j in range(2):
        color = 'white' if cm[i,j] > cm.max()/2 else 'black'
        ax.text(j, i, str(cm[i,j]), ha='center',
                va='center', fontsize=20, color=color)

acc = (cm[0,0]+cm[1,1])/cm.sum()
plt.title(f'Confusion Matrix (Accuracy: {acc:.1%})')
plt.tight_layout()
plt.show()

Exercise 27.17 — Handling class imbalance ⭐⭐

Create an imbalanced dataset and compare model performance with and without class weighting:

np.random.seed(42)
n = 1000
X_imb = pd.DataFrame({
    'feature1': np.random.normal(0, 1, n),
    'feature2': np.random.normal(0, 1, n),
})
# Only 5% positive class
logit = -3 + 1.5 * X_imb['feature1'] + X_imb['feature2']
y_imb = (np.random.random(n) < 1/(1+np.exp(-logit))).astype(int)

Tasks: 1. Check the class distribution 2. Split and fit a standard LogisticRegression 3. Fit a LogisticRegression with class_weight='balanced' 4. Compare confusion matrices, precision, and recall for both models 5. Which model is better at detecting the minority class?

print(f"Class distribution: {pd.Series(y_imb).value_counts().to_dict()}")

Xtr, Xte, ytr, yte = train_test_split(
    X_imb, y_imb, test_size=0.2, random_state=42)

m_std = LogisticRegression(max_iter=1000).fit(Xtr, ytr)
m_bal = LogisticRegression(class_weight='balanced',
                           max_iter=1000).fit(Xtr, ytr)

for name, m in [('Standard', m_std), ('Balanced', m_bal)]:
    pred = m.predict(Xte)
    print(f"\n{name}:")
    print(confusion_matrix(yte, pred))
    print(f"Precision: {precision_score(yte, pred, zero_division=0):.3f}")
    print(f"Recall: {recall_score(yte, pred, zero_division=0):.3f}")

Exercise 27.18 — Complete vaccination classification ⭐⭐

Build a complete classification pipeline for the vaccination project:

np.random.seed(42)
n = 150
countries = pd.DataFrame({
    'gdp_per_capita': np.random.lognormal(9, 1.2, n),
    'health_spending': np.random.uniform(2, 12, n),
    'education_index': np.random.uniform(0.3, 0.95, n),
})
vacc = 30 + 5*np.log(countries['gdp_per_capita']/1000) + 2*countries['health_spending'] + 25*countries['education_index'] + np.random.normal(0,6,n)
countries['high_vacc'] = (vacc.clip(15,100) >= 80).astype(int)

Tasks: 1. Report the class distribution 2. Split 80/20, fit a logistic regression 3. Print the classification report 4. Identify the most important feature (largest absolute coefficient) 5. Print probability predictions for 5 countries with actual labels 6. Write a 2-sentence summary of the model's performance

from sklearn.metrics import classification_report

X = countries[['gdp_per_capita', 'health_spending', 'education_index']]
y = countries['high_vacc']

Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, random_state=42)

model = LogisticRegression(max_iter=1000).fit(Xtr, ytr)
ypred = model.predict(Xte)

print(f"Class distribution: {y.value_counts().to_dict()}")
print(classification_report(yte, ypred, target_names=['Low', 'High']))

for f, c in sorted(zip(X.columns, model.coef_[0]),
                   key=lambda x: abs(x[1]), reverse=True):
    print(f"  {f}: {c:.4f}")

Exercise 27.19 — Comparing regression and classification approaches ⭐⭐⭐

Using the vaccination data from Exercise 27.18, build both a linear regression model (predicting vaccination rate as a number) and a logistic regression model (predicting high/low). Compare:

1. Which approach gives more information?
2. Can you convert the regression predictions into classifications?
3. Which model's errors are more interpretable?

from sklearn.linear_model import LinearRegression

# Regression model
y_reg = vacc.clip(15, 100)
Xtr_r, Xte_r, ytr_r, yte_r = train_test_split(
    X, y_reg, test_size=0.2, random_state=42)
reg_model = LinearRegression().fit(Xtr_r, ytr_r)
reg_pred = reg_model.predict(Xte_r)

# Convert regression to classification
reg_class = (reg_pred >= 80).astype(int)
actual_class = (yte_r >= 80).astype(int)

print("Regression -> Classification:")
print(confusion_matrix(actual_class, reg_class))

# Direct classification
print("\nDirect Classification:")
print(confusion_matrix(yte, ypred))

Exercise 27.20 — Feature scaling effect ⭐⭐⭐

Investigate whether feature scaling affects logistic regression results:

Tasks: 1. Fit a logistic regression on unscaled vaccination data 2. Scale features using StandardScaler and fit again 3. Compare the coefficients (they should change) 4. Compare the predictions (they might differ slightly) 5. Explain why scaling can matter more for logistic regression than for linear regression

from sklearn.preprocessing import StandardScaler

# Unscaled
m1 = LogisticRegression(max_iter=1000).fit(Xtr, ytr)

# Scaled
scaler = StandardScaler()
Xtr_s = scaler.fit_transform(Xtr)
Xte_s = scaler.transform(Xte)
m2 = LogisticRegression(max_iter=1000).fit(Xtr_s, ytr)

print("Unscaled coefficients:", m1.coef_[0].round(5))
print("Scaled coefficients:", m2.coef_[0].round(5))

pred1 = m1.predict(Xte)
pred2 = m2.predict(Xte_s)
print(f"Prediction differences: {(pred1 != pred2).sum()}")

Part D: Synthesis and Critical Thinking ⭐⭐⭐

Exercise 27.21 — The threshold as a policy decision

You build a model to flag potentially fraudulent insurance claims. The model outputs probabilities, and you need to choose a threshold. For each stakeholder, explain what threshold they would prefer and why:

1. The insurance company's fraud investigation team (limited staff)
2. The company's customer satisfaction department
3. The company's CEO (concerned about both fraud losses and customer retention)
4. A consumer protection advocate

Exercise 27.22 — Ethical implications of classification in criminal justice ⭐⭐⭐

A classification model predicts whether a person is "high risk" or "low risk" for recidivism (committing another crime) to inform parole decisions.

1. What features might such a model use?
2. Why might the training data be biased?
3. If the model has different false positive rates for different racial groups, what are the implications?
4. Should such a model be used? Under what conditions, if any?
5. How does the choice of threshold directly affect people's liberty?

Exercise 27.23 — Model comparison essay ⭐⭐⭐

Write a short essay (200-300 words) comparing linear regression and logistic regression. Address: - When to use each - How they're similar - How they're different - What a beginner is most likely to confuse about them

Exercise 27.24 — When probabilities beat predictions ⭐⭐⭐

Give three real-world scenarios where probability outputs from logistic regression are more useful than binary yes/no predictions. For each, explain what decision the probability enables that a binary prediction wouldn't.

Exercise 27.25 — Building a complete classification report ⭐⭐⭐⭐

Choose a classification problem that interests you (spam detection, disease screening, credit approval, student success, etc.) and design a complete analysis:

1. Define the problem (target, features, data source)
2. State the baseline accuracy
3. Identify whether precision or recall is more important and explain why
4. Describe what the confusion matrix entries mean in your specific context
5. Recommend a threshold and justify it based on error costs
6. Identify one potential source of bias in the training data
7. State one ethical concern about deploying this model