Appendix E: Dataset Catalog

This appendix catalogs every dataset used in exercises, case studies, and the progressive project throughout this textbook. For each dataset, you will find what it contains, how large it is, where to get it, what license governs its use, which chapters reference it, and any preprocessing notes specific to our exercises.

E.1 Tabular / Structured Datasets

MovieLens 25M

UCI Adult (Census Income)

UCI Wine Quality

California Housing

Kaggle Ames Housing

Lending Club Loan Data

IBM HR Analytics (Synthetic)

E.2 Time Series and Financial Datasets

Yahoo Finance (via yfinance)

FRED Economic Data

ERA5 Climate Reanalysis

CMIP6 Climate Projections

E.3 Natural Language Processing Datasets

IMDb Movie Reviews

SQuAD 2.0

AG News

Amazon Reviews (Multilingual)

E.4 Health and Biomedical Datasets

MIMIC-IV

Access note: MIMIC-IV requires approximately 1-2 weeks for access approval. Begin the credentialing process before starting Chapter 20. If access is not available, the MIMIC-IV demo dataset (100 patients) is available without credentialing for code testing, and all exercises include synthetic data alternatives.

Framingham Heart Study (Teaching Dataset)

E.5 Graph and Network Datasets

Cora Citation Network

OGB (Open Graph Benchmark)

E.6 Image Datasets

CIFAR-10 / CIFAR-100

ImageNet (ILSVRC 2012 subset)

E.7 Synthetic Datasets

Several exercises use synthetic data to isolate specific statistical concepts. These are generated programmatically and do not require downloads.

Synthetic Causal Data

import numpy as np
import pandas as pd

def generate_causal_dataset(n=5000, seed=42):
    """Generate data with known causal structure for validation.

    DAG: Z -> X -> Y, Z -> Y (confounder)
         U -> X (instrument)
    True ATE of X on Y: 2.0
    """
    rng = np.random.default_rng(seed)

    Z = rng.normal(0, 1, n)                          # Confounder
    U = rng.normal(0, 1, n)                          # Instrument
    X = 0.5 * Z + 0.8 * U + rng.normal(0, 0.5, n)  # Treatment
    Y = 2.0 * X + 1.5 * Z + rng.normal(0, 1, n)    # Outcome (true ATE = 2.0)

    return pd.DataFrame({"Z": Z, "U": U, "X": X, "Y": Y})

Chapters: Ch. 21 (causal identification), Ch. 22 (backdoor adjustment), Ch. 23 (instrumental variables)

Synthetic Fairness Data

def generate_fairness_dataset(n=10000, seed=42):
    """Generate hiring data with known demographic disparity.

    True qualification rate is equal across groups.
    Selection rate differs due to biased threshold.
    """
    rng = np.random.default_rng(seed)

    group = rng.choice(["A", "B"], size=n, p=[0.6, 0.4])
    qualification = rng.normal(50, 10, n)
    noise = rng.normal(0, 5, n)

    # Biased scoring: Group B gets a penalty
    score = qualification + noise - 5.0 * (group == "B").astype(float)
    hired = (score > 50).astype(int)

    return pd.DataFrame({
        "group": group,
        "qualification": qualification,
        "score": score,
        "hired": hired,
    })

Chapters: Ch. 25 (fairness metrics), Ch. 26 (mitigation)

Synthetic Time Series

def generate_arima_with_intervention(n=500, intervention_point=300, seed=42):
    """Generate time series with a known structural break for
    difference-in-differences and interrupted time series exercises.
    """
    rng = np.random.default_rng(seed)

    trend = np.linspace(0, 10, n)
    seasonality = 3 * np.sin(2 * np.pi * np.arange(n) / 52)
    noise = rng.normal(0, 1, n)

    treatment_effect = np.zeros(n)
    treatment_effect[intervention_point:] = 5.0  # true effect = 5.0

    y = trend + seasonality + noise + treatment_effect

    return pd.DataFrame({
        "t": np.arange(n),
        "y": y,
        "post_intervention": (np.arange(n) >= intervention_point).astype(int),
    })

Chapters: Ch. 22 (difference-in-differences), Ch. 18 (Bayesian structural time series)

E.8 Download and Storage Recommendations

Directory Structure

Organize datasets under a single root to keep paths consistent across chapters:

~/ads-data/
├── movielens/
│   └── ml-25m/
├── lending-club/
├── mimic-iv/          # credentialed access
├── era5/
│   ├── temperature/
│   └── precipitation/
├── imagenet/          # if using full dataset
├── huggingface/       # HuggingFace datasets cache
└── synthetic/         # generated by code in exercises

Set the environment variable in your shell profile:

export ADS_DATA_DIR="$HOME/ads-data"
export HF_HOME="$HOME/ads-data/huggingface"

Storage Estimates

Download Script

"""download_datasets.py — Download all freely available datasets."""

import os
from pathlib import Path

DATA_DIR = Path(os.environ.get("ADS_DATA_DIR", Path.home() / "ads-data"))
DATA_DIR.mkdir(parents=True, exist_ok=True)

def download_sklearn_datasets():
    from sklearn.datasets import (
        fetch_california_housing,
        fetch_openml,
    )
    print("Downloading scikit-learn datasets...")
    fetch_california_housing(data_home=DATA_DIR / "sklearn")
    fetch_openml("adult", version=2, data_home=DATA_DIR / "sklearn")
    fetch_openml("wine-quality-red", data_home=DATA_DIR / "sklearn")
    print("  Done.")

def download_huggingface_datasets():
    from datasets import load_dataset
    print("Downloading HuggingFace datasets...")
    load_dataset("imdb", cache_dir=DATA_DIR / "huggingface")
    load_dataset("rajpurkar/squad_v2", cache_dir=DATA_DIR / "huggingface")
    load_dataset("ag_news", cache_dir=DATA_DIR / "huggingface")
    print("  Done.")

def download_torchvision_datasets():
    import torchvision
    print("Downloading torchvision datasets...")
    torchvision.datasets.CIFAR10(root=DATA_DIR / "cifar", download=True)
    torchvision.datasets.CIFAR100(root=DATA_DIR / "cifar", download=True)
    print("  Done.")

def download_movielens():
    import urllib.request
    import zipfile
    print("Downloading MovieLens 25M...")
    url = "https://files.grouplens.org/datasets/movielens/ml-25m.zip"
    dest = DATA_DIR / "movielens" / "ml-25m.zip"
    dest.parent.mkdir(parents=True, exist_ok=True)
    if not dest.exists():
        urllib.request.urlretrieve(url, dest)
        with zipfile.ZipFile(dest, "r") as z:
            z.extractall(DATA_DIR / "movielens")
    print("  Done.")

if __name__ == "__main__":
    download_sklearn_datasets()
    download_huggingface_datasets()
    download_torchvision_datasets()
    download_movielens()
    print(f"\nAll datasets saved to {DATA_DIR}")
    print("Note: MIMIC-IV and ERA5 require separate credentialed access.")
    print("Note: Kaggle datasets require `kaggle` CLI and API key.")

Cross-references: Appendix C documents the libraries used to load and process these datasets. Appendix D covers the environment setup required to run the download script. Individual chapters provide detailed loading and preprocessing code for each dataset they use.