what triggered the alert and what it means in plain language. **(2) Impact assessment** — who is affected, how severely, and what business impact is expected. **(3) Diagnostic steps** — step-by-step commands, queries, and dashboard links to identify the root cause. **(4) Mitigation options** — immed → Chapter 30: Quiz
(1) Clean data
benchmark datasets are curated; production data has missing values, label noise, and inconsistencies. **(2) Static evaluation** — benchmarks are fixed; production data undergoes distribution shift over time. **(3) Unlimited tuning** — papers tune hyperparameters extensively on the benchmark; product → Chapter 37: Quiz
(1) Community and ecosystem
size and activity of contributors, tutorials, Stack Overflow answers, and third-party integrations. **(2) Hiring signal** — whether you can hire people with this technology on their resume. **(3) Migration path** — how difficult it will be to move away when the platform is eventually replaced; open → Chapter 38: Quiz
(1) Concept drift
the relationship between inputs and outcomes has changed (e.g., a pandemic changes user behavior), but the system infrastructure is functioning correctly. In traditional software, the relationship between inputs and outputs is determined by code, not learned from data, so it does not drift. **(2) Fe → Chapter 30: Quiz
(1) Credibility
a track record of technical decisions that led to good outcomes, earned slowly over years and lost quickly through a single catastrophic recommendation. Protected by honesty about uncertainty, early admission of mistakes, and never overpromising. **(2) Reciprocity** — helping other teams (reviewing → Chapter 38: Quiz
(1) Data gap
papers use clean, curated datasets; production data has missing values, label noise, and distribution shift. Example: a model trained on curated movie ratings underperforms on production data where 15% of ratings are missing. **(2) Scale gap** — papers evaluate at a single scale; production systems → Chapter 37: Quiz
(1) Data quality
feature null rates, freshness, distribution drift, training-serving skew, schema compliance. This has no software analogue because software inputs are deterministic; ML inputs are statistical. **(2) Model performance** — prediction distribution, accuracy proxies, calibration, business metric alignme → Chapter 30: Quiz
(1) Detection
the incident is identified through automated alerts, business metric anomalies, or user reports. **(2) Triage** — the incident is classified by severity (SEV-1 through SEV-4) and type (system, data, model, integration), determining the response urgency and the team that responds. **(3) Mitigation** → Chapter 30: Quiz
(1) Judgment
the ability to make good technical decisions under uncertainty, demonstrated through design reviews, RFCs, and architectural decisions that stand the test of time. Not about being right every time, but about being right more often than not, admitting errors, and learning from outcomes. **(2) Scope** → Chapter 38: Quiz
(1) Tech Lead
steers a specific team or project; in data science, owns the modeling approach for a product area (e.g., defining evaluation metrics, selecting architectures, reviewing experiment designs). **(2) Architect** — sets technical direction across teams; in data science, defines organizational standards f → Chapter 38: Quiz
(1) Technical Vision
a 1-2 page narrative describing the desired end state in terms of outcomes (not technologies), providing direction for 2-3 years. **(2) Key Bets** — the 3-5 major technical investments that move toward the vision, each justified with a business case and a technical case, sequenced by dependencies, r → Chapter 38: Quiz
(1) Temporal leakage
training on data that is chronologically after the test data. This is most common in time-series forecasting, recommendation, and fraud detection, where the model can effectively memorize future events. **(2) Preprocessing leakage** — fitting preprocessing steps (normalization, imputation, feature s → Chapter 37: Quiz
(1) Tuned
the baseline must be tuned with the same care as the proposed method, not run with default hyperparameters. **(2) Current** — the baseline must represent the current state of the art, not a historical method that has been superseded. **(3) Equivalent** — the baseline must use the same data, features → Chapter 37: Quiz
(b) Loss computation
reductions (sums/means) over large tensors accumulate rounding errors in low precision. → Chapter 7: Quiz
(d) Softmax
involves exponentials that can overflow in fp16 (max fp16 value is 65,504; $e^{11} \approx 60,000$ already approaches the limit). → Chapter 7: Quiz
(e) Batch normalization variance computation
the variance formula involves squared differences and averaging, which is numerically sensitive in low precision. → Chapter 7: Quiz
22-26%
roughly 5 times the intended rate. The more frequently the analyst checks, the higher the inflation. The root cause is that the fixed-horizon p-value is valid only at the pre-specified sample size; using it at multiple sample sizes violates the assumptions of the test. → Chapter 33: Quiz
it has learned meaningful sequential patterns in user browsing behavior (e.g., within-category browsing, transition patterns between categories). This is meaningful because it demonstrates that the order of item interactions carries predictive information that a non-sequential model would miss. The → Chapter 9: Quiz
All "Advanced Sidebar" sections read and annotated - Challenge problems (marked in exercises as "Challenge") attempted for every chapter - Two case study write-ups per part (14 total) - 5 paper reviews using the Ch. 37 framework, spaced across the 30 weeks - All 13 appendices read (especially Append → Syllabus: Self-Paced Independent Study
not numerical differentiation (finite differences) and not symbolic differentiation (formula manipulation). It is the algorithmic application of the chain rule to a recorded sequence of operations. → Chapter 6: Neural Networks from Scratch
B
backfill safety checklist
a procedure that the team follows before executing any backfill. The checklist should include at least 8 items covering: scope verification, resource planning, output isolation, comparison testing, rollback planning, communication, and monitoring. → Chapter 27: Exercises
Scoring predictions for a large set of inputs. Each input is processed independently; there are no dependencies between predictions. 2. **Hyperparameter search** — Evaluating different hyperparameter configurations. Each trial trains an independent model; no information flows between trials (in grid → Chapter 5: Quiz
batch validation
it checks a complete dataset after it has been produced, typically at a pipeline checkpoint. Pandera performs **inline validation** — it checks DataFrames at Python function boundaries, raising an exception the moment invalid data enters a function. Great Expectations integrates with orchestration s → Chapter 28: Quiz
Better approaches:
**Contextual bandits** with product embeddings as the context, so that information about one product generalizes to similar products. - **Collaborative filtering bandits** that model the reward as a function of user and product embeddings, exploiting the low-rank structure of the user-product intera → Chapter 22: Quiz
a directed acyclic graph specifying the causal relationships between the protected attribute, all features, and the outcome. This is a much stronger requirement, because the causal graph determines which features are causally downstream of the protected attribute (and must be adjusted in the counter → Chapter 31: Quiz
the FAISS index and the model share a version tag. The deployment pipeline rebuilds the FAISS index whenever a new model is promoted, and both artifacts are deployed atomically. The model registry (MLflow, Chapter 29) tracks the correspondence. → Case Study 1: StreamRec Track B Implementation — Standard Integration
coincidental
a statistical artifact of testing thousands of pairwise correlations across many time series. With enough variables, some will be correlated by chance. However, it could also reflect **confounding** by a shared time trend (both variables increased over the same period due to unrelated underlying cau → Chapter 15: Quiz
only the columns used in the query are read from disk, reducing I/O by the fraction of unused columns (e.g., reading 5 of 100 columns reduces I/O by ~95%); (2) **predicate pushdown** — min/max statistics per row group allow entire row groups to be skipped when they cannot match a filter condition; a → Chapter 25: Quiz
combining domain depth with consistency
spokes provide domain context while the hub ensures standards, shared infrastructure, and career development. → Chapter 39: Quiz
Common diagnostic patterns:
*Sigmoid-shaped*: over-confident at extremes, reasonable in the middle — common for uncalibrated neural networks - *Flat*: model outputs cluster in a narrow probability range regardless of true label frequency — the model has not learned to distinguish confidence levels - *Shifted*: systematically a → Appendix G: Evaluation Metrics Reference
Both Qini and AUUC require randomized data (or credible causal estimation) to compute. You cannot evaluate an uplift model using observational data without addressing confounding. - Confidence intervals for Qini/AUUC require bootstrap or permutation methods. They are wider than for predictive metric → Appendix G: Evaluation Metrics Reference
Common pitfalls:
The *accuracy paradox*: In a dataset with 99% negatives, a model that always predicts "negative" achieves 99% accuracy while being completely useless. This is the single most common metric mistake in applied ML. - Accuracy treats all errors as equally costly. In credit scoring (Chapter 31), denying → Appendix G: Evaluation Metrics Reference
the attributions sum to $f(x) - f(x')$. Like Shapley's efficiency axiom, every unit of prediction difference is accounted for. (2) **Sensitivity** — if changing feature $j$ from the baseline value to the input value changes the output, feature $j$ receives nonzero attribution. A feature that matters → Chapter 35: Quiz
they are limited by the GPU's arithmetic throughput. Operations below the ridge point are **memory-bound** — they are limited by the speed of reading/writing data from HBM. For an A100 (312 TFLOP/s, 2.0 TB/s), the ridge point is 156 FLOP/byte. Large matrix multiplications (arithmetic intensity ~4096 → Chapter 26: Quiz
conditional edges
dependencies that should be followed only if a specific condition is met (e.g., "run `register_models` only if at least one model passed evaluation"). → Chapter 27: Exercises
Conditional independence
$X \perp\!\!\!\perp Y \mid Z$ — means $p(x, y \mid z) = p(x \mid z) \cdot p(y \mid z)$. Conditional independence is the structural assumption behind graphical models, naive Bayes classifiers, and the causal DAGs we will study in Part III. → Chapter 3: Probability Theory and Statistical Inference
Configuration after optimization:
Parallelism: FSDP (ZeRO-2 equivalent), NCCL backend - Local batch size per GPU: 12 - Gradient accumulation: 4 - Effective global batch size: $12 \times 4 \times 64 = 3{,}072$ - Optimizer: LAMB with warmup cosine schedule - Learning rate: $3 \times 10^{-4} \times (3072/8) = 0.1152$ (capped at 0.01 wi → Case Study 1: Climate DL — Distributed Training for a Global Weather Prediction Model
Configuration:
Parallelism: DDP (NCCL backend, NVLink) - Local batch size per GPU: 8 - Global batch size: 64 - Learning rate: $3 \times 10^{-4} \times (64/8) = 2.4 \times 10^{-3}$ (linear scaling, 8x base) - Warmup: 2,000 steps (approximately 5% of total steps) - Optimizer: LAMB (required for global batch size > 2 → Case Study 1: Climate DL — Distributed Training for a Global Weather Prediction Model
Confounder
good control. - **Item popularity** ($P$): Causes recommendation (popular items are recommended more) and engagement (popular items get more engagement). **Confounder** — good control. - **Content quality** ($Q$): Causes engagement but not recommendation (the algorithm does not observe quality direc → Chapter 17: Graphical Causal Models — DAGs, d-Separation, and Structural Causal Models
Confounders:
User Preference: causes both Recommendation (through the algorithm's prediction) and Engagement (organic behavior). This is the primary confounder. - Item Features: may cause both Recommendation (algorithm input) and Engagement (if features like genre directly affect engagement). → Chapter 17: Graphical Causal Models — DAGs, d-Separation, and Structural Causal Models
predictions are in the expected range and format. (2) **Latency** — the new model meets the latency budget. (3) **Consistency** — predictions are correlated with the production model's predictions (large discrepancies suggest a bug, not a model improvement). (4) **Error handling** — the model handle → Chapter 24: Quiz
$n$ is so large that $O(n)$ space is infeasible (e.g., billions of events per day). 2. **Data arrives continuously** — There is no "end" to the dataset; it is an unbounded stream. 3. **Single-pass requirement** — Data can only be read once (e.g., network packets, sensor readings). 4. **Approximate a → Chapter 5: Quiz
the DS sits with the domain team, understands the business problem intimately, and has a fast feedback loop with the decision-maker. → Chapter 39: Quiz
**Dual-store feature architecture** (H.6.1): online Redis for serving, offline Parquet for training - **Hybrid serving** (H.1.1 + H.1.2): batch candidate generation + real-time re-ranking - **Graceful degradation** (H.5.2): four levels from full model to popularity baseline - **Circuit breaker** (H. → Appendix H: ML System Design Patterns
Difficulty ratings:
**Introductory** — Accessible to a reader who has completed the relevant chapters. Good first papers in a subfield. - **Intermediate** — Requires comfort with the mathematical foundations (Part I) and familiarity with the subfield. The bulk of the papers listed here. - **Advanced** — Assumes deep fl → Appendix I: Key Papers and Reading Lists
Directional (6 tests):
Higher income → lower predicted default probability - Higher FICO score → lower predicted default probability - Higher debt-to-income ratio → higher predicted default probability - Longer employment tenure → lower predicted default probability - More delinquencies → higher predicted default probabil → Case Study 2: Meridian Financial — Regulatory Model Validation for Credit Scoring
organizations running ETL, ML, analytics, dbt, and data engineering pipelines on a shared platform. Its 80+ provider packages, 10+ years of production track record, and massive community mean that operators exist for almost every external system. Teams with existing Airflow expertise and a mix of wo → Chapter 27: Quiz
domain ownership
the team that produces data owns it as a product; (2) **data as a product** — each data asset has a discoverable interface, documented schema, quality guarantees, and an owner; (3) **self-serve data platform** — shared infrastructure enables domain teams to publish data products without building the → Chapter 25: Quiz
it consumes less than 1% of an A100's 80 GB memory. DDP is simpler than FSDP (no sharding complexity, no all-gather overhead per layer), and the model is small enough that replicating it on every GPU wastes negligible memory. FSDP would become necessary if the model grew to the point where **paramet → Chapter 26: Quiz
effect modifiers
covariates that may cause the treatment effect to vary. The CATE $\tau(x)$ is estimated as a function of $X$. Including a variable in $X$ means "I believe the treatment effect may differ at different values of this variable." → Chapter 19: Quiz
the X-ray of a square matrix, revealing its fundamental modes of action 2. **Singular Value Decomposition** — the fundamental theorem of data science, applicable to *any* matrix 3. **Matrix calculus** — the language that backpropagation speaks, and the mathematical apparatus behind every gradient yo → Chapter 1: Linear Algebra for Machine Learning
applied to different symmetry groups. Weight sharing in CNNs (translation equivariance) and permutation-invariant aggregation in GNNs (permutation equivariance) are both consequences of imposing the data's symmetry on the architecture. The transformer, which has no built-in symmetry and uses positio → Chapter 14: Quiz
**Mathematical derivation exercises (Parts I, III, IV):** Award full credit only when the student shows every application of the chain rule, every conditional expectation expansion, and every assumption invoked. A correct final answer without derivation earns at most 5/10. A correct setup with a sig → Grading Rubrics
Exercises
implementation problems designed to take 1-4 hours each - **Quiz** — self-assessment questions - **Two case studies** — applied scenarios from industry and research - **Key takeaways** — summary card - **Further reading** — annotated paper recommendations → Advanced Data Science
multi-source retrieval, deep ranking, re-ranking; (2) **Real-time features unavailable** — fall back to batch features (stale but personalized); (3) **Ranking model unavailable** — use retrieval scores directly (lower quality); (4) **Feature store unavailable** — return globally popular items (not p → Chapter 24: Quiz
fundamental
it is a mathematical consequence, not an engineering limitation. Information-theoretically, if a mechanism reveals zero information about any individual (perfect privacy), it also reveals zero information about the population (zero utility). Useful analysis requires extracting some information from → Chapter 32: Quiz
G
global modeling
training across many related series to leverage cross-series patterns — which is how TFT outperforms per-series classical methods on StreamRec's multi-category engagement data. The key risk is overfitting on short series. → Chapter 23: Key Takeaways
Start with no features; iteratively add the feature that improves the objective most. $O(d^2)$ model evaluations. 2. **Greedy backward elimination** — Start with all features; iteratively remove the least important. $O(d^2)$ model evaluations. 3. **L1 regularization (Lasso)** — Adds an $\ell_1$ pena → Chapter 5: Quiz
where connected nodes tend to have *different* labels (e.g., in dating networks, bipartite buyer-seller networks, or amino acid interaction networks) — aggregation mixes features from dissimilar nodes, which can destroy the discriminative signal. In extreme cases, a GCN on a heterophilous graph perf → Chapter 14: Quiz
the baseline probability of the adverse outcome, regardless of whether the treatment changes it. The quantity that optimal targeting should use is the **Conditional Average Treatment Effect (CATE)**: $E[Y(1) - Y(0) \mid X = x]$, which measures how much the treatment changes the outcome for individua → Chapter 15: Quiz
homophily
the assumption that connected nodes tend to have similar labels or features (e.g., friends share political views, papers in the same field cite each other). Aggregation effectively "borrows" useful information from similar nodes. → Chapter 14: Quiz
How many neurons are needed
the required width can be exponentially large in the input dimension. (2) **That gradient descent can find the right weights** — the theorem is an existence result and says nothing about optimization. (3) **Sample efficiency** — it says nothing about how much training data is needed to learn the app → Chapter 6: Quiz
no weight matrix $\mathbf{W}_{hh}$ appears in this gradient path. The gradient can flow through the cell state for hundreds of time steps without multiplicative attenuation, creating a "gradient highway." → Chapter 9: Quiz
some features cannot be changed (age, race, sex). The optimization must hold these fixed. (2) **Actionability** — the changes should represent actions the person can actually take. "Increase your age by 10 years" is mathematically valid but useless. (3) **Plausibility** — the counterfactual should l → Chapter 35: Quiz
Implementation considerations:
Shadow predictions must not affect user experience (asynchronous evaluation) - Log both champion and challenger predictions with identical features to ensure a fair comparison - Monitor challenger latency independently — a slow challenger should not affect champion performance - Run shadow mode for → Appendix H: ML System Design Patterns
inconsistent predictions
the same input can produce different outputs depending on what other inputs are in the batch. This is one of the most common production bugs in deep learning. → Chapter 7: Quiz
Increase dropout
start with 0.3 and increase to 0.5 if needed. 3. **Increase weight decay** — try $10^{-3}$ to $10^{-1}$ with AdamW. 4. **Data augmentation** — if applicable to the domain. 5. **Reduce model size** — fewer layers or smaller hidden dimensions. 6. **Get more training data** — the most reliable fix but → Chapter 7: Quiz
Increase local batch size
processes more data per GPU before each all-reduce, amortizing the fixed communication cost. Trade-off: more GPU memory consumed by activations; may require gradient checkpointing. (2) **Gradient accumulation** — accumulate gradients over $K$ micro-batches before the all-reduce, reducing communicati → Chapter 26: Quiz
Inference speed
1D CNNs are much faster than transformers for short sequences because they avoid the $O(n^2)$ self-attention computation and are highly parallelizable. In a recommendation system serving millions of requests per second, latency matters. (2) **Model size** — 1D CNNs have far fewer parameters than tra → Chapter 8: Quiz
it routes information between positions, allowing each position to aggregate information from other positions. The FFN performs **intra-token computation** — it applies an independent nonlinear transformation at each position, processing the information that attention has gathered. Research on trans → Chapter 10: Quiz
interaction effect
the amount by which the treatment effect of A differs when B is active vs. inactive. Test $H_0: \beta_{AB} = 0$. A significant interaction means the combined effect of A and B is not the sum of their individual effects. In practice, the interaction is compared to the main effect of A as a relative m → Chapter 33: Quiz
**Reliability** measures calibration error. Lower is better. A perfectly calibrated model has reliability = 0. - **Resolution** measures discrimination — how much the model's predictions vary with the true outcome. Higher is better. - **Uncertainty** is a property of the data, not the model. It is m → Appendix G: Evaluation Metrics Reference
Intrinsic dimensionality
the fine-tuning loss landscape has a much lower effective dimension than the full parameter space (Aghajanyan et al., 2021). (2) **The weight update $\Delta W$ is empirically low-rank** — most of the model's knowledge is preserved from pretraining, and only a small task-specific adjustment is needed → Chapter 11: Quiz
Introductory Statistics
Making Sense of Data in the Age of AI 2. **Intermediate Data Science** — Machine Learning, Experimentation, and the Craft of Data-Driven Decisions 3. **Advanced Data Science** — Deep Learning, Causal Inference, and Production Systems at Scale *(this book)* → Advanced Data Science
Intuition
a plain-language explanation of what the math means 2. **Formal notation** — the mathematical expression 3. **Code** — a numpy or PyTorch implementation you can run → How to Use This Book
Invalid adjustment sets:
$\{$Biomarker$\}$: Biomarker is a descendant of Drug X (Drug X $\to$ Biomarker). Violates condition 1. Conditioning on it blocks the causal pathway. - $\{$Insurance Status$\}$: Insurance Status is a collider (Age $\to$ Insurance Status $\leftarrow$ Comorbidities). Conditioning on it opens a spurious → Chapter 17: Graphical Causal Models — DAGs, d-Separation, and Structural Causal Models
invariance tolerances are much higher
credit scoring invariance tests require 0.98-0.99 overlap (near-perfect invariance to protected attributes like gender and zip code), while recommendation invariance tests tolerate 0.70-0.95 because some platform-specific effects are legitimate. Second, **directional tests encode economic relationsh → Chapter 28: Quiz
Invariance — Fair Lending (6 tests):
Gender invariance: changing gender must not change the score by more than 0.01 - Marital status invariance: same constraint - Race/ethnicity invariance: same constraint (tested via constructed matched pairs) - National origin invariance: applicant born in the U.S. vs. not must not change the score b → Case Study 2: Meridian Financial — Regulatory Model Validation for Credit Scoring
investigate the SRM root cause
check for platform-specific differences, redirect chains, performance discrepancies, and bot filtering; (3) **fix the root cause** and re-run the experiment; (4) only trust the engagement result if the re-run passes SRM. An SRM-contaminated experiment cannot be salvaged by statistical adjustment — t → Chapter 33: Quiz
5.1% accuracy degradation and 14.2% Recall@20 degradation is borderline acceptable - Casual and Japanese market segments require quality mitigation before deployment - $\varepsilon = 3$ is within the range used by Apple (1-8 per data type per day) and represents a defensible privacy standard → Case Study 2: StreamRec DP Training — Privacy-Utility Tradeoff at Scale
Justification for $\varepsilon = 8$ (initial):
Only 1.9% accuracy degradation and 7.0% Recall@20 degradation from baseline - All user segments except new_users (already underperforming) meet the 0.18 threshold - Formal DP guarantee satisfies the EDPB's "quantifiable privacy protection" guidance - 3.5x training time overhead is manageable within → Case Study 2: StreamRec DP Training — Privacy-Utility Tradeoff at Scale
K
KD-tree
Exact (in low dimensions); partitions space with axis-aligned hyperplanes. Degrades to $O(n)$ in high dimensions. 2. **Locality-Sensitive Hashing (LSH)** — Approximate; hashes similar points to the same bucket with high probability. Query time is $O(n^\rho)$ where $\rho < 1$. 3. **HNSW (Hierarchical → Chapter 5: Quiz
Key ablation findings:
Reducing the number of heads hurts performance — multi-head is better than single-head. - Reducing $d_k$ hurts performance — attention dimension matters. - Bigger models are better (up to the tested range). - Dropout is important — without it, performance degrades. - Learned positional embeddings pe → Case Study 1: Paper Walkthrough — "Attention Is All You Need" Using the Three-Pass Method
Key design decisions:
**Multi-stage ranking:** The funnel architecture (1000 -> 100 -> 10) allows using progressively more expensive models at each stage, keeping total latency within budget. - **Hybrid retrieval:** Combine lexical (BM25) and semantic (dense retrieval with bi-encoders) to capture both exact-match and mea → Appendix H: ML System Design Patterns
Key design principles:
**Two-stage architecture:** Rules handle obvious cases (known fraud patterns, velocity checks); ML handles the ambiguous middle. Rules provide explainability and fast response; ML provides generalization. - **Latency constraint:** The entire pipeline must complete within 100-200ms to avoid degrading → Appendix H: ML System Design Patterns
Key properties:
$\mathbf{H}$ is symmetric (by Schwarz's theorem, when second partials are continuous) - If $\mathbf{H}$ is positive definite at $\mathbf{x}$, then $\mathbf{x}$ is a local minimum - If $\mathbf{H}$ is negative definite, then $\mathbf{x}$ is a local maximum - If $\mathbf{H}$ has both positive and nega → Chapter 2: Multivariate Calculus and Optimization
we observe proxies (user history, demographics, past behavior) but not the latent preference itself. (2) The algorithm is specifically designed to exploit preference signals, creating strong confounding. (3) Because the confounder is unobserved, the backdoor criterion cannot be straightforwardly sat → Chapter 17: Quiz
Learning objectives
what you will be able to do after completing the chapter - **Motivating example** — why this topic matters, grounded in the anchor examples - **Main content** — rigorous exposition with mathematical derivations paired with code implementations - **Progressive project milestone** — the next step in b → How to Use This Book
with its real distribution of users, items, contexts, and edge cases — rather than on a static, retrospective holdout set. This catches issues that holdout data misses: traffic pattern changes, seasonal effects, new user segments, and the interaction between model predictions and the serving infrast → Chapter 28: Quiz
$O(Bs)$ instead of $O(Bs^2)$, because the attention matrix is never written to HBM; and (2) **Higher throughput** — by fusing multiple HBM read/write operations into a single kernel, FlashAttention eliminates the round trips that make standard attention memory-bound. The speedup comes not from reduc → Chapter 26: Quiz
M
malicious or curious server
a server that honestly computes the aggregate but also inspects individual client updates to infer information about their local data. Differential privacy addresses a different threat: an adversary who sees the *output* (the trained model) and tries to infer information about the training data. DP → Chapter 32: Quiz
one team sets standards for experimentation, validation, fairness, and deployment, ensuring that all DS work meets the same quality bar. Also enables knowledge sharing and career development. → Chapter 39: Quiz
Milestone-specific notes:
**M0-M1 (mathematical foundations):** Emphasis on correctness of SVD implementation and profiling methodology. Do not penalize for naive implementations — the point is understanding, not optimization. - **M2-M6 (deep learning):** Emphasis on training curves, proper evaluation, and comparison across → Grading Rubrics
**Regression:** Assuming Gaussian noise on the target variable leads to the mean squared error loss. Specifically, if $y = f(x) + \epsilon$ with $\epsilon \sim \mathcal{N}(0, \sigma^2)$, then the MLE for $f$ minimizes $\sum_i (y_i - f(x_i))^2$. - **Latent spaces:** Variational autoencoders enforce $ → Chapter 3: Probability Theory and Statistical Inference
The fallback for a recommendation model might be a popularity-based ranker (no personalization, but always available and fast) - The fallback for a fraud model might be a rule-based system (higher false positive rate, but never misses known patterns) - Monitor fallback activation rate as a system he → Appendix H: ML System Design Patterns
instrumental variables, interrupted time series, or synthetic control methods that exploit the policy's timing or geographic variation. → Chapter 15: Quiz
natural parameters
$\mathbf{T}(x)$ is the **sufficient statistic** — the function of the data that contains all information about $\boldsymbol{\eta}$ - $A(\boldsymbol{\eta})$ is the **log-partition function** (ensures normalization) - $h(x)$ is the **base measure** → Chapter 3: Probability Theory and Statistical Inference
Neither model should be deployed
the validation gate should block both. The first model passes behavioral tests but does not meet the absolute performance floor, meaning it provides worse recommendations than the minimum acceptable standard. The second model has good aggregate performance but fails behavioral tests, meaning it has → Chapter 28: Quiz
the distribution is stable. A PSI between 0.1 and 0.25 indicates **moderate shift** — the distribution has changed enough to warrant investigation but may not require immediate action. A PSI above 0.25 indicates **significant shift** — the distribution has changed substantially and action is require → Chapter 28: Quiz
non-numerical
selecting the best feature, choosing a hyperparameter, picking the most popular category — where adding continuous noise to the output is not meaningful. The Laplace and Gaussian mechanisms add noise to numerical answers; the exponential mechanism randomizes the *selection* of a discrete answer, bia → Chapter 32: Quiz
Non-obvious dependencies
an ML decision like choosing real-time serving implies downstream requirements (online feature store, streaming pipeline, GPU infrastructure) that should be documented explicitly. (2) **Evolution through experimentation** — decisions based on A/B test results (e.g., "+12% engagement for session-awar → Chapter 24: Quiz
holdout metrics and sliced metrics, taking minutes; (2) **Behavioral tests** — MFT, INV, DIR suites, taking minutes; (3) **Shadow evaluation** — live traffic comparison, taking 3-7 days; (4) **Canary deployment** — 5-10% real traffic, taking 3 days. The ordering is important because each stage is pr → Chapter 28: Quiz
Operation ordering
numpy and PyTorch may compute matrix multiplications using different BLAS implementations with different accumulation orders. Floating-point addition is not associative, so $(a + b) + c \neq a + (b + c)$ in general. (2) **Fused operations** — PyTorch may use fused multiply-add (FMA) instructions tha → Chapter 6: Quiz
git SHA pinning the exact code that ran 2. **Configuration version** — hyperparameters, feature lists, and thresholds used 3. **Data versions** — Delta Lake table versions or partition identifiers for each input data source 4. **Feature store version** — Feast feature set version and the specific fe → Chapter 27: Quiz
placebo tests
the key inferential tool for synthetic control. By applying the method to each control state as if it were treated, the authors construct a distribution of "placebo effects" and show that California's effect is an extreme outlier, providing a permutation-based p-value. For the extension to multiple → Chapter 33: Further Reading
beliefs about $\theta$ before observing data - $p(D \mid \theta)$: **likelihood** — probability of the data given parameter value $\theta$ - $p(D) = \int p(D \mid \theta) p(\theta) d\theta$: **marginal likelihood** (evidence) — normalizing constant - $p(\theta \mid D)$: **posterior** — updated belie → Chapter 20: Quiz
Problem formulation
translating a vague business question into a well-defined data science problem — is the highest-value activity a data scientist performs. A candidate who selects the perfect algorithm but frames the wrong problem produces negative value. **Communication** — explaining results to non-technical stakeh → Chapter 39: Quiz
Estimate $e(x) = P(W=1 \mid X=x)$ using logistic regression, gradient boosting, or a neural network - Match each treated unit to the nearest control unit(s) in propensity score space - Estimate ATE from the matched sample → Appendix J: Causal Inference Identification Guide
Points above the diagonal: the model is *under-confident* (it predicts 0.6 but the actual rate is 0.8) - Points below the diagonal: the model is *over-confident* (it predicts 0.8 but the actual rate is 0.6) - Horizontal gap between the point and the diagonal is the calibration error for that bin → Appendix G: Evaluation Metrics Reference
add dropout, weight decay (L2 regularization), or both. (2) **More data** — if available, more training data reduces overfitting. (3) **Simpler model** — reduce the number of layers or neurons. (4) **Early stopping** — stop training when validation loss starts increasing. (5) **Data augmentation** — → Chapter 6: Quiz
provide a direct gradient highway from the loss to earlier layers, bypassing the vanishing gradient path. 2. **Normalization layers** (batch norm or layer norm) — keep activations and gradients in a healthy range at every layer. 3. **Proper initialization** — He initialization for ReLU networks ensu → Chapter 7: Quiz
that message passing is inherently a low-pass filtering operation. They delay over-smoothing but do not eliminate it. **Graph transformers**, which replace local message passing with global attention, fundamentally avoid the depth-smoothing tradeoff but sacrifice the sparsity advantage and local ind → Chapter 14: Quiz
$P(Y \mid X)$. Questions about observed joint distributions. Example: "What is the click-through rate for users who were shown item X?" → Chapter 15: Quiz
Rung 2: Intervention (Doing)
$P(Y \mid \text{do}(X))$. Questions about the effects of active interventions. Example: "If we recommend item X to a random user, what is the expected click probability?" → Chapter 15: Quiz
Rung 3: Counterfactual (Imagining)
$P(Y_{x'} \mid X=x, Y=y)$. Questions about specific individuals under alternative conditions. Example: "This user clicked on item X after we recommended it. Would they have clicked if we had NOT recommended it?" → Chapter 15: Quiz
S
sample space
the set of all possible outcomes - $\mathcal{F}$ is a **$\sigma$-algebra** — a collection of subsets of $\Omega$ (the "events" we can assign probabilities to) - $P: \mathcal{F} \to [0, 1]$ is a **probability measure** satisfying the Kolmogorov axioms → Chapter 3: Probability Theory and Statistical Inference
sampling variability
they do not account for **systematic bias** from confounding, model misspecification, or SUTVA violations. → Chapter 16: Quiz
Scale
foundation models are trained on orders of magnitude more data (billions of tokens/images vs. millions), (2) **Generality** — they transfer to a wide range of tasks without architectural modification, not just tasks similar to the pretraining task, and (3) **Emergence** — they exhibit capabilities t → Chapter 13: Quiz
Scope
transformer attention is global (all positions attend to all), while GAT attention is local (only neighbors attend). (2) **Score function** — GAT uses an additive scoring function with a single attention vector $\mathbf{a}$, while transformers use separate query/key projections with dot-product scor → Chapter 14: Quiz
batch, real-time, and near-real-time — differ in latency, freshness, cost, and infrastructure complexity. The right choice depends on the product requirements, not the model architecture. Hybrid patterns (real-time with batch fallback) are common and effective. → Chapter 24: ML System Design — Architecture Patterns for Real-World Machine Learning
set operation
it computes pairwise similarities and weighted averages without any notion of position or order. If the input sequence is permuted, the outputs are the same permutation of the original outputs (permutation equivariance). Without positional encoding, the transformer treats "the dog bit the man" ident → Chapter 10: Quiz
Single feature computation path
the same code (or pipeline) writes to both the online and offline stores, eliminating implementation mismatches. (2) **Backfill** — when a new feature is added, it is retroactively computed for historical data in the offline store. (3) **Point-in-time joins** — the training pipeline retrieves featur → Chapter 24: Quiz
small fraction
often cited as roughly 5% — of the total system. The rest consists of data collection and verification, feature extraction and management, serving infrastructure, monitoring and alerting, configuration management, process management, machine resource management, and analysis tools. The surrounding i → Chapter 24: Quiz
Smooth everywhere
GELU is differentiable at $z = 0$, avoiding the sharp corner of ReLU. This smoothness can benefit optimization. (2) **No dead neurons** — GELU has a non-zero gradient for slightly negative inputs, so neurons are not permanently killed. (3) **Probabilistic interpretation** — GELU can be viewed as a s → Chapter 6: Quiz
specifically, PSI-based distribution shift detection or a statistical test on the feature's mean. Schema validation (Great Expectations or Pandera) would not catch this because the schema is unchanged: the column exists, has the correct type, is non-null, and values may still be within the valid ran → Chapter 28: Quiz
they reveal gaps in the team's mental model of the system, gaps that may cause future surprises if not addressed. In the StreamRec retrospective, the "surprised_by" item (cold-start bias is fairness bias) revealed a gap in the team's mental model: they understood cold-start as a model quality proble → Chapter 36: Quiz
the inclusion of future information in the training data. Without a point-in-time join, a naive join retrieves the latest feature value, which may include data generated after the training example's event. The model then learns from signals that are not available at serving time, producing inflated → Chapter 25: Quiz
Tests for features and data
schema validation, feature importance correlation, data pipeline unit tests, feature coverage, and data monitoring; (2) **Tests for model development** — training reproducibility, holdout quality, sliced quality, staleness detection, and proxy metrics; (3) **Tests for ML infrastructure** — training → Chapter 28: Quiz
training RMSE decreases monotonically with rank, but held-out RMSE reaches a minimum around the true latent dimension and increases thereafter. 4. **Simple SVD on the imputed matrix is a reasonable baseline** but has fundamental limitations: the imputation step biases the decomposition. In Chapter 2 → Chapter 1: Linear Algebra for Machine Learning
They are equal when there is no confounding
specifically, when $X$ has no common causes with $Y$, meaning all paths between $X$ and $Y$ are directed from $X$ to $Y$ (no backdoor paths). In this case, the observational conditional distribution already reflects the causal effect. Formally: if $\{Y(0), Y(1)\} \perp\!\!\!\perp X$ (unconditional i → Chapter 17: Quiz
fork, chain, collider — determine how information flows through a graph. Forks and chains are open by default and blocked by conditioning; colliders are blocked by default and opened by conditioning. This asymmetry is the fundamental rule of graphical causal reasoning. → Chapter 17: Graphical Causal Models — DAGs, d-Separation, and Structural Causal Models
Training loop
forward pass, loss computation, backward pass, optimizer step, gradient accumulation, (2) **Evaluation** — periodic evaluation on a validation set with custom metrics, (3) **Logging** — integration with TensorBoard, Weights & Biases, and other logging frameworks, (4) **Checkpointing** — automatic mo → Chapter 13: Quiz
it does not capture the full variability in the data. Common causes: (1) the model assumes a distribution with too little variance (e.g., Poisson when the data are overdispersed — a negative binomial would be more appropriate), (2) important sources of variation are missing from the model (e.g., a h → Chapter 21: Quiz
Understanding Why
Understanding WHY an algorithm works (not just how to call it) is what separates a senior data scientist from someone who copies Stack Overflow. The math serves understanding, not credentials. 2. **Prediction ≠ Causation** — The most important and most commonly confused distinction in applied data s → Advanced Data Science — Master Outline
the latent variable representing a user's underlying interest in specific content. User Preference is a common cause (fork) of both the Recommendation (the algorithm recommends items it predicts the user will like, which is driven by preferences) and Engagement (users engage more with content that m → Chapter 17: Quiz
V
Valid adjustment sets:
$\{$Severity, Age$\}$: Blocks all three backdoor paths. Does not include descendants of Drug X. Valid. - $\{$Severity$\}$: Blocks paths 1 and 2. Path 3 (through Age) remains open. Valid only if there is no path Drug X $\leftarrow$ Age $\to$ Hospitalization that is unblocked — not valid in this graph → Chapter 17: Graphical Causal Models — DAGs, d-Separation, and Structural Causal Models
inserting untreated buffer periods between alternations; (2) **longer periods** — reducing the fraction of time contaminated by carryover; and (3) **regression adjustment** — explicitly modeling the carryover as a function of the previous period's assignment, e.g., $Y_t = \mu + \tau D_t + \gamma D_{ → Chapter 33: Quiz
No confidence intervals or multi-run statistics. Single-run numbers only. - No evaluation beyond translation (the English constituency parsing result in Table 4 is a limited generalization test). - No analysis of failure modes — what kinds of sentences does the Transformer get wrong? - No scaling an → Case Study 1: Paper Walkthrough — "Attention Is All You Need" Using the Three-Pass Method
The query space is too large to enumerate (e.g., arbitrary text queries in search) - Real-time context matters (e.g., current session behavior, time-of-day) - The application requires sub-second freshness → Appendix H: ML System Design Patterns
Use the **potential outcomes framework** when the treatment is well-defined, the estimand (ATE, ATT, LATE) is clear, and you are primarily concerned with estimation and inference. - Use the **graphical framework** when the causal structure is complex (many variables, potential mediators, colliders), → Appendix J: Causal Inference Identification Guide
The entity space is finite and enumerable (all users, all products) - Prediction freshness of hours is acceptable - Real-time features do not significantly improve quality - The team lacks real-time serving infrastructure - StreamRec example: pre-compute the top-100 recommendations for each user dai → Appendix H: ML System Design Patterns
Z
zero inference overhead
the merged model has identical architecture and latency to the original, and (4) multiple LoRA adapters can be swapped in and out of a single base model for different tasks, enabling multi-task serving with a single GPU. → Chapter 13: Quiz