Key Takeaways: Logistic Regression

One-Sentence Summary

Logistic regression models the probability of a binary outcome by applying the sigmoid function to a linear combination of predictors, producing coefficients interpretable as odds ratios, and evaluated using confusion matrices, ROC curves, and AUC — making it the foundational classification technique that underpins AI/machine learning classification systems.

Core Concepts at a Glance

The Logistic Regression Procedure

Step by Step

1. Confirm binary outcome. Ensure the response variable is binary (0/1, yes/no, success/failure).

2. Explore the data: - Compare predictor distributions between the two outcome groups (box plots, bar charts) - Check outcome rate (base rate) — imbalanced classes require special attention

3. Fit the model using sm.Logit(y, X).fit() (statsmodels) or LogisticRegression().fit(X, y) (sklearn).

4. Interpret coefficients: - Exponentiate each coefficient: $e^{b_i}$ = odds ratio - "For each one-unit increase in $x_i$, the odds of the outcome are multiplied by $e^{b_i}$, holding all other variables constant" - Check p-values and 95% CIs for the odds ratios

5. Evaluate the model: - Confusion matrix at the chosen threshold - Accuracy, sensitivity, specificity, precision, F1 score - ROC curve and AUC

6. Choose the threshold: - Consider the relative costs of false positives vs. false negatives - The threshold is a values decision, not just a statistical one

7. Check fairness: - Evaluate error rates separately for each relevant group - Equal accuracy does not guarantee equal error rates

Key Python Code

import statsmodels.api as sm
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (confusion_matrix, classification_report,
                             roc_curve, auc)
import numpy as np

# --- STATSMODELS (inference: coefficients, p-values, CIs) ---
X_sm = sm.add_constant(df[['x1', 'x2', 'x3']])
model = sm.Logit(df['y'], X_sm).fit()
print(model.summary())

# Odds ratios
print("Odds Ratios:")
print(np.exp(model.params).round(3))
print("95% CI for Odds Ratios:")
print(np.exp(model.conf_int()).round(3))

# --- SKLEARN (prediction and evaluation) ---
clf = LogisticRegression(max_iter=1000)
clf.fit(X_train, y_train)

y_pred = clf.predict(X_test)
y_prob = clf.predict_proba(X_test)[:, 1]

# Confusion matrix
cm = confusion_matrix(y_test, y_pred)
print(cm)
print(classification_report(y_test, y_pred))

# ROC and AUC
fpr, tpr, thresholds = roc_curve(y_test, y_prob)
print(f"AUC: {auc(fpr, tpr):.3f}")

Excel Procedure

The Threshold Concept: Thinking in Odds

Probability, odds, and log-odds are three representations of the same quantity. Logistic regression models log-odds because they range from $-\infty$ to $+\infty$, making them suitable for a linear model. The odds ratio ($e^{b}$) is the most interpretable form of the coefficient.

Key Formulas

Confusion Matrix Reference

Metric Quick Reference

ROC and AUC Interpretation

AUC interpretation: The probability that a randomly chosen positive case gets a higher predicted probability than a randomly chosen negative case.

Odds Ratio Interpretation

Common Mistakes

Connections