A function that determines the next point to evaluate in Bayesian optimization by balancing exploration (regions of high uncertainty) and exploitation (regions of high predicted value). Common acquisition functions include Expected Improvement (EI), Upper Confidence Bound (UCB), and Probability of Improvement (PI).
Activation Function
A nonlinear function applied element-wise to the output of a neural network layer. Enables the network to learn nonlinear mappings. Common choices include ReLU, sigmoid, tanh, GELU, and Swish.
AdamW
An optimizer that combines Adam's adaptive learning rates with decoupled weight decay regularization. Unlike the original Adam with L2 regularization, AdamW applies weight decay directly to the parameters rather than to the gradient, which is more correct for adaptive methods.
Adapter
A small trainable module inserted into a frozen pretrained model to enable parameter-efficient fine-tuning. Adapters add a small number of parameters per layer while keeping the original model weights fixed.
Adjacency Matrix
A square matrix A where entry A[i,j] indicates whether nodes i and j are connected by an edge in a graph. For weighted graphs, entries represent edge weights.
All-Reduce
A distributed communication primitive where each worker contributes a tensor and all workers receive the sum (or other reduction) of all tensors. In data-parallel training, all-reduce synchronizes gradients across GPUs. Ring all-reduce is the most common communication-efficient variant.
Amortized Analysis
A technique for analyzing the average time complexity of a sequence of operations, even when individual operations may occasionally be expensive. Used to analyze data structures like dynamic arrays and hash tables.
Approximate Nearest Neighbors (ANN)
Algorithms that find approximately (not exactly) the nearest neighbors of a query point, trading accuracy for speed. Common methods include locality-sensitive hashing (LSH), HNSW, and IVF. Critical for recommendation serving at scale.
Architecture Decision Record (ADR)
A structured document that records a significant design decision, its context, the alternatives considered, and the rationale for the chosen option. Used to maintain institutional knowledge about ML system design choices.
Arithmetic Intensity
The ratio of floating-point operations to memory accesses in a computation. Determines whether a computation is compute-bound (high arithmetic intensity) or memory-bound (low arithmetic intensity).
Attention Mechanism
A mechanism that allows a neural network to focus on relevant parts of its input when producing each element of its output. Originally introduced for sequence-to-sequence models (Bahdanau, Luong), later generalized into self-attention in transformers.
Augmented IPW (AIPW)
A doubly robust causal estimator that combines outcome modeling with inverse probability weighting. Consistent if either the propensity score model or the outcome model is correctly specified, but not necessarily both.
Autoregressive Generation
A generation strategy where each output token is produced one at a time, conditioned on all previously generated tokens. The standard generation approach for decoder-only language models.
Average Treatment Effect (ATE)
The expected difference in potential outcomes across the entire population: E[Y(1) - Y(0)]. The most commonly targeted causal estimand.
Average Treatment Effect on the Treated (ATT)
The expected treatment effect among those who actually received treatment: E[Y(1) - Y(0) | T=1]. Differs from the ATE when treatment effects vary and treatment assignment is non-random.
Average Treatment Effect on the Untreated (ATU)
The expected treatment effect among those who did not receive treatment: E[Y(1) - Y(0) | T=0]. Important for policy decisions about expanding treatment to new populations.
B
Backdoor Criterion
A graphical condition for identifying valid adjustment sets in causal DAGs. A set of variables Z satisfies the backdoor criterion relative to (X, Y) if Z blocks all backdoor paths from X to Y and no variable in Z is a descendant of X.
Backdoor Path
A non-causal path between treatment and outcome that begins with an arrow into the treatment variable. Backdoor paths transmit confounding bias and must be blocked for causal identification.
Backpropagation
An algorithm for computing gradients of a scalar loss function with respect to all parameters in a computational graph, using the chain rule applied in reverse (output to input). Equivalent to reverse-mode automatic differentiation.
Backpropagation Through Time (BPTT)
The application of backpropagation to recurrent neural networks by "unrolling" the network across time steps. Truncated BPTT limits the number of time steps through which gradients flow.
Bandwidth Selection
In regression discontinuity designs, the choice of how wide a window around the cutoff to use for estimation. Smaller bandwidths reduce bias but increase variance.
Batch Normalization
A technique that normalizes layer activations across the mini-batch dimension during training. Stabilizes training, allows higher learning rates, and acts as a regularizer. During inference, uses running statistics instead of batch statistics.
Bayes Factor
The ratio of marginal likelihoods under two competing models. Quantifies the evidence provided by data in favor of one model over another.
Bayesian Credible Interval
An interval [a, b] such that the posterior probability of the parameter lying within [a, b] equals a specified level (e.g., 95%). Unlike frequentist confidence intervals, credible intervals have a direct probability interpretation conditional on the observed data.
Bayesian Optimization
A sequential strategy for optimizing expensive-to-evaluate black-box functions. Builds a probabilistic surrogate model (typically a Gaussian process) of the objective function and uses an acquisition function to decide where to evaluate next.
Behavioral Testing
A testing paradigm for ML models based on expected model behavior rather than aggregate accuracy metrics. Includes invariance tests (perturbations that should not change output), directional tests (perturbations with a known effect direction), and minimum functionality tests (basic cases the model must handle).
Beta-Binomial
A conjugate Bayesian model where the prior on a Bernoulli success probability is a Beta distribution. After observing successes and failures, the posterior is also Beta-distributed.
Big-O Notation
A mathematical notation describing the asymptotic upper bound of an algorithm's time or space complexity as input size grows. O(n log n) means the algorithm's resource usage grows no faster than n log n.
Blue-Green Deployment
A deployment strategy that maintains two identical production environments. Traffic is switched from the old (blue) environment to the new (green) environment instantaneously, with the ability to roll back by switching traffic back.
BPR Loss
Bayesian Personalized Ranking loss, an objective function for learning-to-rank in recommendation systems. Optimizes the pairwise ranking between observed positive items and sampled negative items.
C
Calibration
The property that a model's predicted probabilities match empirical frequencies. A well-calibrated model that predicts 70% probability should be correct approximately 70% of the time. Measured by Expected Calibration Error (ECE).
Canary Deployment
A deployment strategy that gradually shifts a small percentage of traffic (e.g., 5-10%) to a new model version while monitoring metrics. If metrics degrade, traffic is rolled back.
Causal DAG
A directed acyclic graph where directed edges represent direct causal relationships between variables. Encodes assumptions about the data-generating process and enables identification of causal effects.
Causal Forest
A modification of random forests designed to estimate heterogeneous treatment effects. Splits are optimized to maximize treatment effect heterogeneity (not prediction accuracy) across the resulting subgroups. Part of the Generalized Random Forest (GRF) framework.
Causal Masking
An attention mask that prevents each position from attending to future positions in a sequence. Used in decoder-only transformer models to maintain the autoregressive property during training.
Chain Rule
The calculus rule for computing derivatives of composite functions: d/dx f(g(x)) = f'(g(x)) g'(x). The mathematical foundation of backpropagation, extended to multivariate functions via Jacobians.
Changepoint Detection
Methods for identifying points in a time series where the statistical properties (mean, variance, distribution) change abruptly. Important for detecting regime changes, anomalies, and concept drift.
CheckList
A behavioral testing methodology for NLP models (Ribeiro et al., 2020) that defines test types: Minimum Functionality Tests (MFT), Invariance Tests (INV), and Directional Expectation Tests (DIR). Adapted broadly for ML model testing.
Classifier-Free Guidance
A technique for controllable generation in diffusion models that interpolates between conditional and unconditional generation during the reverse process. Higher guidance scales increase fidelity to the conditioning signal at the cost of diversity.
CLIP
Contrastive Language-Image Pretraining. A foundation model trained to align image and text representations in a shared embedding space using contrastive learning on internet-scale image-caption pairs.
Collider
A variable that is a common effect of two or more other variables. Conditioning on a collider opens a spurious association between its causes — a common source of selection bias.
Computational Graph
A directed graph representing a mathematical computation, where nodes are operations or variables and edges represent data flow. The structure that automatic differentiation traverses to compute gradients.
Concept Drift
A change in the relationship between model inputs and the target variable over time, causing model performance to degrade. Distinct from data drift (change in input distribution alone).
Conditional Average Treatment Effect (CATE)
The expected treatment effect conditional on covariates X: E[Y(1) - Y(0) | X=x]. Captures how treatment effects vary across subpopulations.
Conformal Prediction
A distribution-free framework for constructing prediction sets with guaranteed finite-sample coverage. If the exchangeability assumption holds, a conformal prediction set at level 1-α contains the true outcome with probability at least 1-α.
Confounding
A situation where an observed association between treatment and outcome is distorted by a common cause (confounder) of both. The central challenge of observational causal inference.
Conjugate Prior
A prior distribution that, combined with a particular likelihood function, yields a posterior distribution in the same family. Enables closed-form Bayesian updating.
Contrastive Learning
A self-supervised learning approach that trains representations by pulling similar (positive) pairs close together and pushing dissimilar (negative) pairs apart in embedding space. InfoNCE is the standard contrastive loss.
Convolution
A mathematical operation that applies a learned kernel (filter) across the spatial dimensions of an input, computing weighted sums at each position. In CNNs, technically cross-correlation rather than convolution, but the terms are used interchangeably.
Cosine Annealing
A learning rate schedule that decreases the learning rate following a cosine curve from its initial value to near zero over a training cycle. Often combined with warm restarts.
Cramér-Rao Bound
A lower bound on the variance of any unbiased estimator. The variance of an unbiased estimator is at least 1/I(θ), where I(θ) is the Fisher information.
Cross-Entropy
The expected number of bits needed to encode data from distribution p using the optimal code for distribution q: H(p, q) = -Ep[log q]. Minimizing cross-entropy is equivalent to minimizing KL divergence from p and to maximum likelihood estimation.
Cross-Fitting
A sample-splitting technique used in double/debiased machine learning where nuisance parameters are estimated on one fold and causal effects estimated on the other, then roles are swapped. Prevents overfitting bias.
CUPED
Controlled-experiment Using Pre-Experiment Data. A variance reduction technique for A/B testing that uses pre-experiment covariates to reduce the variance of the treatment effect estimator. Achieves variance reduction proportional to the squared correlation between pre- and post-experiment metrics.
D
d-Separation
A graphical criterion for reading conditional independence relationships from a causal DAG. Two sets of variables X and Y are d-separated given Z if every path between X and Y is blocked by Z.
Dagster
A modern data orchestration framework based on software-defined assets. Each pipeline step defines the asset it produces, with dependencies inferred from asset relationships.
Data Contract
A formal agreement between data producers and consumers specifying schema, quality expectations, freshness guarantees, and SLAs. Prevents silent data quality issues that cause production ML failures.
Data Drift
A change in the distribution of model input features over time, without necessarily implying a change in the target relationship. Measured by PSI, KS statistic, or JS divergence.
Data Lakehouse
A data architecture that combines the flexibility of data lakes (storing raw, unstructured data) with the management features of data warehouses (ACID transactions, schema enforcement, indexing). Implemented by frameworks like Delta Lake, Apache Iceberg, and Apache Hudi.
Data Parallelism
A distributed training strategy where each GPU holds a complete copy of the model and processes a different shard of the data. Gradients are synchronized across GPUs via all-reduce.
Data Processing Inequality
An information-theoretic result stating that no processing of data can create information that was not already present: if X → Y → Z forms a Markov chain, then I(X; Z) ≤ I(X; Y).
Dead Neuron
A neuron whose activation is permanently zero for all inputs, typically caused by large negative biases with ReLU activation. Dead neurons consume parameters but contribute nothing to the network output.
DeepAR
An autoregressive deep learning model for probabilistic time series forecasting that outputs the parameters of a probability distribution at each time step, enabling uncertainty quantification.
Delta Lake
An open-source storage layer that brings ACID transactions, schema enforcement, and time-travel to data lakes. Built on Parquet files with a transaction log.
Denoising Diffusion Probabilistic Model (DDPM)
A generative model that learns to reverse a gradual noising process. The forward process adds Gaussian noise over many steps until the data becomes pure noise; the reverse process learns to denoise step by step to generate samples.
Depthwise Separable Convolution
A factorization of a standard convolution into a depthwise convolution (one filter per input channel) and a pointwise 1x1 convolution (combining channels). Reduces parameters and computation by a factor of approximately k², where k is the kernel size.
Difference-in-Differences (DiD)
A causal estimation method that compares the change in outcomes over time between a treated group and a control group. Requires the parallel trends assumption — that both groups would have followed the same trajectory absent treatment.
Diffusion Model
A class of generative models that learn to reverse a gradual noising process. The training objective is to predict the noise added at each step, and generation proceeds by iteratively denoising random noise into data.
Direct Preference Optimization (DPO)
An alignment method for LLMs that eliminates the need for an explicit reward model by directly optimizing the policy on preference pairs. Simpler than RLHF while achieving comparable results.
Directed Acyclic Graph (DAG)
A graph with directed edges and no cycles. In causal inference, DAGs encode assumptions about causal structure. In pipeline orchestration, DAGs represent task dependencies.
Divergence (MCMC)
A diagnostic indicating that the Hamiltonian Monte Carlo sampler encountered a region of the posterior where the curvature was too high for the step size. Divergences indicate unreliable posterior estimates and typically require reparameterization.
Do-Operator
Pearl's notation for interventions: P(Y | do(X=x)) denotes the distribution of Y when X is set to value x by external intervention, as opposed to P(Y | X=x) which conditions on observing X=x. The most important inequality in causal inference: P(Y|do(X)) ≠ P(Y|X) in general.
Double/Debiased Machine Learning (DML)
A framework for causal estimation that uses ML models for nuisance parameter estimation (propensity scores, outcome models) while maintaining valid statistical inference for the causal parameter. Based on Neyman orthogonality and cross-fitting.
Doubly Robust Estimation
A causal estimation strategy that combines outcome modeling and propensity score weighting. Provides consistent estimates if either the outcome model or the propensity score model is correctly specified.
Dropout
A regularization technique that randomly sets a fraction of neuron activations to zero during training. Prevents co-adaptation of features and can be interpreted as training an implicit ensemble of sub-networks. Inverted dropout scales surviving activations to maintain expected values.
E
Early Stopping
A regularization technique that stops training when validation performance begins to degrade, preventing overfitting. The model from the best validation epoch is selected as the final model.
Effective Sample Size (ESS)
A diagnostic for MCMC that estimates how many independent samples the chain's correlated samples are worth. Low ESS relative to the number of iterations indicates poor mixing.
Eigendecomposition
The factorization of a square matrix A into A = QΛQ⁻¹, where Q contains the eigenvectors and Λ is the diagonal matrix of eigenvalues. The "X-ray" of a matrix, revealing its fundamental axes and scaling factors.
ELBO (Evidence Lower Bound)
A lower bound on the log-marginal-likelihood (evidence) of the data: log p(x) ≥ Eq[log p(x,z)] - Eq[log q(z)]. Maximizing the ELBO is equivalent to minimizing KL(q(z) || p(z|x)). The optimization objective for VAEs and variational inference.
Embedding
A learned dense vector representation of a discrete entity (word, user, item) in a continuous space where geometric relationships capture semantic relationships.
Entropy
The expected information content (or average surprise) of a random variable: H(X) = -Σ p(x) log p(x). Measures the uncertainty or "randomness" of a distribution. Maximum for uniform distributions.
Exclusion Restriction
The assumption that an instrumental variable affects the outcome only through the treatment variable. This assumption is untestable and must be justified on domain knowledge grounds.
Expected Calibration Error (ECE)
A scalar metric that measures miscalibration by computing the weighted average of the absolute difference between predicted confidence and actual accuracy across binned predictions.
Expected Improvement (EI)
An acquisition function for Bayesian optimization that measures the expected amount by which a candidate point will improve upon the current best observed value. Balances exploration and exploitation.
Experimentation Platform
An end-to-end system for running, monitoring, and analyzing online experiments (A/B tests). Includes randomization infrastructure, metric computation, statistical analysis, and guardrail monitoring.
Exploding Gradient
A training pathology where gradients grow exponentially during backpropagation, causing unstable parameter updates and numerical overflow. Mitigated by gradient clipping, proper initialization, and normalization.
F
Fairlearn
An open-source Python toolkit for assessing and improving the fairness of ML models. Provides fairness metrics, constraint-based mitigation algorithms (exponentiated gradient, threshold optimizer), and interactive dashboards.
FAISS
Facebook AI Similarity Search — a library for efficient similarity search and clustering of dense vectors. Implements ANN algorithms (IVF, HNSW, PQ) for billion-scale nearest neighbor retrieval.
Faithfulness
The assumption that d-separation in the causal DAG is the only source of conditional independence in the data distribution. Rules out exact cancellation of causal effects.
Feature Map
The output of applying a convolutional filter to an input. Each filter produces one feature map that highlights the presence of a particular learned pattern at each spatial location.
Feature Store
An infrastructure component that manages the computation, storage, and serving of features for ML models. Ensures consistency between features used during training (offline) and serving (online).
Fisher Information
A measure of the amount of information that an observable random variable carries about an unknown parameter. The expected curvature of the log-likelihood at the true parameter value.
Flash Attention
An exact attention algorithm that reduces memory usage from O(n²) to O(n) by tiling the computation to exploit the GPU memory hierarchy (computing in SRAM rather than reading/writing HBM). Produces identical results to standard attention but 2-4x faster.
FLOPs
Floating-Point Operations. A measure of computational cost. For example, training GPT-3 required approximately 3.14 × 10²³ FLOPs.
Flow Matching
A modern generative modeling approach that learns a continuous-time flow from noise to data by regressing a velocity field. Simpler and more flexible than the discrete-time forward-reverse process of diffusion models.
Forward Process
In diffusion models, the process that gradually adds Gaussian noise to data over many time steps until the data is indistinguishable from pure noise. Defined by a noise schedule. The reverse process (learned by the model) inverts this transformation.
Frobenius Norm
The matrix analog of the Euclidean norm: ||A||F = sqrt(Σij A²ij). Equivalently, the square root of the sum of squared singular values.
Front-Door Criterion
A graphical criterion for identifying causal effects when the backdoor criterion cannot be satisfied (because confounders are unobserved). Requires a mediator variable that fully mediates the effect and has no unblocked backdoor paths from the treatment.
Fully Sharded Data Parallel (FSDP)
A memory-efficient distributed training strategy that shards model parameters, gradients, and optimizer states across GPUs (similar to DeepSpeed ZeRO Stage 3). Each GPU holds only a fraction of the model, gathering parameters on-demand for computation.
Fundamental Problem of Causal Inference
The impossibility of observing both potential outcomes Y(0) and Y(1) for the same unit at the same time. Every causal inference method is, in some sense, an approach to circumventing this problem.
G
Gaussian Process (GP)
A distribution over functions, specified by a mean function and a kernel (covariance) function. Any finite collection of function values follows a multivariate Gaussian distribution. Used as a surrogate model in Bayesian optimization.
GELU
Gaussian Error Linear Unit. An activation function defined as x Φ(x), where Φ is the standard Gaussian CDF. The default activation in modern transformers. Smoother than ReLU with non-zero gradients for negative inputs.
Generalized Random Forest (GRF)
A framework for non-parametric statistical estimation using forest-based methods. Causal forests are the most prominent instance. Provides valid confidence intervals for heterogeneous treatment effects.
Generative Adversarial Network (GAN)
A generative model consisting of two networks — a generator and a discriminator — trained in a minimax game. The generator learns to produce realistic samples, while the discriminator learns to distinguish real from generated samples.
Grad-CAM
Gradient-weighted Class Activation Mapping. A visualization technique that highlights the spatial regions of an input image most relevant to a CNN's prediction by computing gradient-weighted feature map activations.
Gradient Accumulation
A technique that accumulates gradients over multiple forward-backward passes before performing a parameter update, effectively simulating a larger batch size without requiring more GPU memory.
Gradient Checkpointing
A memory-saving technique that trades compute for memory by not storing intermediate activations during the forward pass and recomputing them during the backward pass. Reduces memory usage from O(n) to O(sqrt(n)) layers.
Gradient Clipping
A technique that limits the magnitude of gradients during training to prevent exploding gradients. Implemented either by clipping the global norm or by clipping individual gradient values.
Graph Attention Network (GAT)
A graph neural network that uses attention mechanisms to learn different importance weights for different neighbors during message aggregation.
Graph Convolutional Network (GCN)
A neural network layer that updates each node's representation by aggregating representations from its neighbors, weighted by the normalized adjacency matrix. The simplest message-passing GNN.
Graph Isomorphism Network (GIN)
A GNN architecture proved to be as powerful as the 1-WL graph isomorphism test. Uses sum aggregation (not mean) to maximize expressiveness.
GraphSAGE
A scalable GNN that learns node representations by sampling and aggregating features from a fixed-size neighborhood. Unlike GCN, supports inductive learning on unseen nodes.
Great Expectations
An open-source Python framework for data validation. Allows users to define "expectations" (assertions about data properties) that are tested automatically as part of data pipelines.
H
Hallucination
The tendency of language models to generate plausible-sounding but factually incorrect or unsupported text. A fundamental property of probabilistic generation, not a bug to be simply fixed.
Hamiltonian Monte Carlo (HMC)
An MCMC sampling method that uses gradient information to guide proposals, resulting in much more efficient exploration of the posterior than random-walk methods.
Hessian Matrix
The square matrix of second partial derivatives of a scalar function: Hij = ∂²f / ∂xi ∂xj. Captures the curvature of the loss landscape and determines the nature of critical points (minima, maxima, saddle points).
Heterogeneous Treatment Effect (HTE)
Variation in treatment effects across individuals or subgroups. When HTEs are present, the ATE may be a poor summary because some subgroups benefit much more (or less) than others.
Hidden Markov Model (HMM)
A state-space model with discrete latent states, where the system transitions between states according to a Markov chain and observations are conditionally independent given the current state.
Hierarchical Model
A Bayesian model with multiple levels of parameters, where parameters at one level are drawn from distributions governed by higher-level hyperparameters. Enables partial pooling between groups.
Highest Posterior Density Interval (HPDI)
The narrowest interval containing a specified probability mass of the posterior distribution. Unlike equal-tailed credible intervals, HPDIs always include the most probable values.
Hyperband
A hyperparameter optimization algorithm that extends successive halving by running it with multiple budget configurations, eliminating the need to choose the trade-off between number of configurations and budget per configuration.
I
Idempotency
The property that running an operation multiple times with the same input produces the same result and no additional side effects. The most important design principle for data pipelines.
Ignorability
The assumption that treatment assignment is independent of potential outcomes conditional on observed covariates: Y(0), Y(1) ⫫ T | X. Also called "unconfoundedness" or "selection on observables." This assumption is untestable.
Importance Sampling
A Monte Carlo technique that estimates expectations under one distribution by sampling from a different (proposal) distribution and reweighting. Used when the target distribution is difficult to sample from directly.
InfoNCE
A contrastive loss function derived from noise-contrastive estimation. Maximizes agreement between positive pairs while minimizing agreement with negative samples. The standard loss for modern contrastive learning.
Information Bottleneck
A theoretical framework for understanding neural network representations. A hidden layer should compress the input (minimize I(X; T)) while preserving information about the label (maximize I(T; Y)).
Instrumental Variable (IV)
A variable that affects the outcome only through the treatment variable (exclusion restriction) and is independent of unmeasured confounders. Enables causal estimation when unobserved confounding exists.
Inverse Probability Weighting (IPW)
A causal estimation technique that reweights observations by the inverse of their treatment propensity to create a pseudo-population where treatment is independent of confounders.
J
Jacobian Matrix
The matrix of first partial derivatives of a vector-valued function: Jij = ∂fi / ∂xj. In ML, the Jacobian of a layer maps input perturbations to output perturbations.
K
Kalman Filter
An optimal recursive estimator for linear Gaussian state-space models. Alternates between predict (propagate state forward) and update (incorporate new observation) steps. The Bayesian interpretation: predict gives the prior, update gives the posterior.
Kernel Function (GP)
A function k(x, x') that defines the covariance between function values at points x and x' in a Gaussian process. Common kernels include RBF (squared exponential), Matérn, and rational quadratic.
KL Divergence
Kullback-Leibler divergence. A measure of how one probability distribution p differs from a reference distribution q: DKL(p || q) = Ep[log(p/q)]. Non-negative by Gibbs' inequality and zero only when p = q. Not symmetric.
Knowledge Graph
A graph where nodes represent entities and edges represent typed relationships (e.g., "Paris" → "capitalof" → "France"). Used for structured knowledge representation and reasoning.
KV-Cache
A caching mechanism in transformer inference that stores the key and value tensors from previous time steps, avoiding recomputation during autoregressive generation. Reduces inference complexity from O(n²) per token to O(n) per token.
L
LAMB
Layer-wise Adaptive Moments optimizer for Batch training. Extends Adam with layer-wise learning rate adaptation, enabling stable training at very large batch sizes (up to 64K).
Language Model
A probabilistic model that assigns probabilities to sequences of tokens. Modern language models are typically autoregressive transformer decoders trained on next-token prediction.
Laplacian Matrix
The matrix L = D - A, where D is the degree matrix and A is the adjacency matrix. Its eigenvalues encode graph connectivity properties and form the basis of spectral graph theory.
LARS
Layer-wise Adaptive Rate Scaling. An optimizer that scales the learning rate for each layer based on the ratio of the parameter norm to the gradient norm. Enables stable training at large batch sizes.
Layer Normalization
A normalization technique that normalizes across the feature dimension for each individual sample, rather than across the batch. The default normalization in transformer architectures.
Learning Rate Schedule
A strategy for adjusting the learning rate during training. Common schedules include step decay, cosine annealing, warmup followed by decay, and the one-cycle policy.
LightGCN
A simplified GCN architecture for collaborative filtering that removes nonlinear activation functions and feature transformation, keeping only the neighborhood aggregation. Achieves strong recommendation performance with minimal computation.
Linear Scaling Rule
The heuristic that when increasing batch size by a factor k, the learning rate should also be increased by a factor k. Requires learning rate warmup to work in practice.
Locality-Sensitive Hashing (LSH)
A family of hashing techniques where similar items are more likely to be mapped to the same hash bucket. Enables approximate nearest neighbor search in sublinear time.
LoRA
Low-Rank Adaptation. A parameter-efficient fine-tuning method that freezes pretrained weights and injects trainable low-rank decomposition matrices into each layer. Reduces trainable parameters by 100-1000x while maintaining fine-tuning quality.
Loss Landscape
The high-dimensional surface defined by the loss function over the parameter space. Properties of the loss landscape (curvature, saddle points, local minima, flat regions) determine optimization difficulty.
M
Marginal Likelihood
The probability of the data under a model, integrating over all possible parameter values: p(D) = ∫ p(D|θ) p(θ) dθ. Used for Bayesian model comparison. Also called "model evidence."
Markov Chain Monte Carlo (MCMC)
A class of algorithms that draw samples from a target distribution (typically a posterior) by constructing a Markov chain whose stationary distribution is the target.
Matrix Factorization
Decomposing a matrix R into the product of two or more lower-rank matrices: R ≈ UV^T. In recommendation systems, factorizes the user-item interaction matrix into user and item latent factor matrices.
Maximum Likelihood Estimation (MLE)
Finding the parameter values that maximize the probability (likelihood) of the observed data: θMLE = argmaxθ p(D|θ). Equivalent to minimizing negative log-likelihood, which for classification is cross-entropy.
Message Passing
The computational framework underlying most GNNs: each node updates its representation by aggregating "messages" from its neighbors. Different GNN architectures (GCN, GAT, GraphSAGE) differ in how messages are computed and aggregated.
Meta-Learner
A framework for estimating heterogeneous treatment effects using standard ML models as building blocks. S-learner (single model), T-learner (two separate models), X-learner (cross-learner), and R-learner (residual learner) differ in how they decompose the CATE estimation problem.
Mixed Precision Training
Training neural networks using lower-precision floating-point formats (fp16 or bf16) for forward and backward passes while maintaining fp32 master weights. Reduces memory usage and increases throughput on GPUs with tensor cores.
MLflow
An open-source platform for managing the ML lifecycle, including experiment tracking, model packaging, model registry, and deployment.
MLOps Maturity Levels
A framework for assessing the maturity of an organization's ML operations. Level 0: manual everything. Level 1: ML pipeline automation. Level 2: CI/CD pipeline automation. Level 3: automated retraining with monitoring.
Mode Collapse
A GAN failure mode where the generator produces only a few distinct outputs rather than capturing the full diversity of the data distribution. The generator "collapses" to a mode of the distribution.
Model Parallelism
A distributed training strategy where different parts of the model reside on different GPUs. Required when the model is too large to fit in a single GPU's memory.
Model Registry
A centralized store for managing trained model versions, their metadata, lineage, and lifecycle stage (staging, production, archived).
Monte Carlo Estimation
Approximating an expectation by the sample average of function evaluations at random points drawn from the relevant distribution. Convergence rate is O(1/√n) regardless of dimensionality.
mSPRT
Mixture Sequential Probability Ratio Test. A sequential testing method that allows continuous monitoring of A/B tests without inflating Type I error rates. Based on a mixture of likelihood ratios.
Multi-Head Attention
Running multiple attention operations in parallel with different learned projections, then concatenating the results. Each "head" can learn to attend to different types of relationships (syntactic, semantic, positional).
Mutual Information
The amount of information that one random variable contains about another: I(X; Y) = H(X) - H(X|Y) = H(Y) - H(Y|X). Unlike correlation, captures nonlinear dependencies.
N
N-BEATS
Neural Basis Expansion Analysis for interpretable Time Series. A pure deep learning architecture for time series forecasting that uses backward and forward residual connections with basis expansion.
NCCL
NVIDIA Collective Communications Library. A library for multi-GPU and multi-node communication primitives (all-reduce, broadcast, etc.) optimized for NVIDIA GPUs and high-speed interconnects.
Next-Token Prediction
The pretraining objective for autoregressive language models: given a sequence of tokens, predict the probability distribution over the next token. Equivalent to maximum likelihood estimation on the training corpus.
Neyman Orthogonality
A mathematical property ensuring that the causal estimand is insensitive (to first order) to errors in nuisance parameter estimation. The theoretical foundation of double/debiased machine learning.
No-U-Turn Sampler (NUTS)
An extension of Hamiltonian Monte Carlo that eliminates the need to hand-tune the trajectory length parameter. Automatically determines when the trajectory begins to "turn around" (U-turn). The default sampler in PyMC and Stan.
Noise Schedule
In diffusion models, the schedule that controls how much noise is added at each step of the forward process. Linear, cosine, and learned schedules produce different generation quality.
Normalizing Flows
A class of generative models that transform a simple base distribution (e.g., Gaussian) into a complex data distribution through a sequence of invertible transformations with tractable Jacobian determinants.
Null Space
The set of all vectors x such that Ax = 0. Its dimension equals the number of columns minus the rank. Captures the "directions that the matrix ignores."
O
One-Cycle Policy
A learning rate schedule that increases the learning rate from a low value to a maximum over the first portion of training, then decreases it below the initial value. Empirically enables faster convergence and better generalization.
Online-Offline Consistency
The requirement that features computed during model training (offline, on historical data) are identical to features computed during model serving (online, in real-time). Violations cause training-serving skew.
Opacus
A PyTorch library for training models with differential privacy (DP-SGD). Clips per-sample gradients and adds calibrated Gaussian noise to ensure (ε, δ)-differential privacy.
Optuna
A hyperparameter optimization framework that uses Tree-structured Parzen Estimators (TPE) as the default Bayesian method. Supports pruning (early stopping of unpromising trials) and integration with common ML frameworks.
Over-Smoothing
A pathology in deep GNNs where stacking too many layers causes all node representations to converge to similar values, losing discriminative power. Analogous to information loss in deep networks.
P
Pandera
A Python library for DataFrame validation that integrates with pandas and supports statistical hypothesis testing on columns. More Python-native than Great Expectations.
Parallel Trends Assumption
The identifying assumption in difference-in-differences: absent treatment, the treated and control groups would have followed the same trend over time. Untestable but can be assessed with pre-treatment data.
Partial Pooling
The Bayesian hierarchical modeling strategy where group-level estimates are "pulled" toward the global mean, with the degree of pooling determined by group sample size and variability. Groups with little data are pulled more.
Perplexity
A measure of how well a language model predicts a text, defined as 2^H(p,q) where H is cross-entropy. Lower perplexity means better prediction. Interpretable as the effective number of equally likely next tokens.
Pipeline Parallelism
A distributed training strategy that splits the model into sequential stages, each on a different GPU, and pipelines micro-batches to maintain high utilization.
Point-in-Time Join
A join operation that retrieves feature values as they existed at the time a training example was created, preventing future information from leaking into historical training data. Critical for temporal correctness in feature engineering.
Population Stability Index (PSI)
A metric that quantifies how much a variable's distribution has shifted between two datasets (typically training vs. production). PSI > 0.25 generally indicates significant shift.
Positional Encoding
Information added to token embeddings to inject sequence order, which self-attention cannot infer on its own. Types include sinusoidal (fixed), learned, and rotary (RoPE).
Posterior Distribution
The probability distribution of a parameter after observing data, computed via Bayes' theorem: p(θ|D) ∝ p(D|θ)p(θ). Represents updated beliefs about the parameter.
Potential Outcomes
The pair of outcomes Y(0) and Y(1) that a unit would experience under control and treatment, respectively. Only one is ever observed (the fundamental problem). The Rubin causal model framework.
Prior Distribution
The probability distribution representing beliefs about a parameter before observing data. Choices range from uninformative (minimal prior knowledge) to informative (strong domain knowledge).
Probabilistic Forecast
A forecast that outputs a full probability distribution (or prediction intervals) rather than a single point prediction. Essential when downstream decisions depend on uncertainty.
Progressive Rollout
A deployment strategy that gradually increases the percentage of traffic served by a new model version (e.g., 10% → 25% → 50% → 100%) while monitoring metrics at each stage.
Prompt Engineering
The practice of crafting input prompts to elicit desired behavior from large language models. Techniques include zero-shot, few-shot, chain-of-thought, and structured output formats.
Propensity Score
The probability of receiving treatment given observed covariates: e(x) = P(T=1 | X=x). Balancing on the propensity score is sufficient to remove confounding from observed covariates.
PyMC
A probabilistic programming framework for Bayesian modeling in Python. Supports model specification, MCMC sampling (NUTS), variational inference, and posterior analysis via ArviZ.
PyTorch Geometric
A library built on PyTorch for deep learning on graphs and other irregular structures. Provides standard GNN layers (GCN, GAT, GraphSAGE), data loaders for graphs, and benchmark datasets.
Q
Qini Curve
A diagnostic plot for uplift models that shows the cumulative incremental effect as a function of the fraction of the population treated (ordered by estimated uplift). The causal analog of the lift curve.
QLoRA
Quantized LoRA. Combines 4-bit quantization of the base model with LoRA adaptation, enabling fine-tuning of large language models on a single consumer GPU.
Quantile Regression
A regression method that estimates conditional quantiles (e.g., 10th, 50th, 90th percentile) of the response variable rather than the conditional mean. Produces prediction intervals without distributional assumptions.
Quantization
Reducing the numerical precision of model weights and/or activations (e.g., from fp32 to INT8 or INT4) to reduce memory footprint and increase inference speed. Methods include post-training quantization (GPTQ, AWQ) and quantization-aware training.
R
R-hat (Gelman-Rubin)
A convergence diagnostic for MCMC that compares within-chain and between-chain variance. Values close to 1.0 (typically < 1.01) indicate convergence; values substantially above 1.0 indicate the chains have not mixed.
R-Learner
A meta-learner for CATE estimation that directly targets the treatment effect residual using a loss function based on the Robinson decomposition. The most theoretically grounded meta-learner.
RAG (Retrieval-Augmented Generation)
An architecture that grounds LLM responses in retrieved documents by embedding a query, retrieving relevant chunks from a vector database, and including them in the LLM prompt. Reduces hallucination and enables knowledge updates without retraining.
Receptive Field
The region of the input that influences a particular neuron's output in a CNN. Deeper layers have larger receptive fields due to successive convolutions and pooling operations.
Regression Discontinuity (RD)
A quasi-experimental design that exploits a sharp cutoff in treatment assignment based on a continuous running variable. Units just above and below the cutoff are compared, approximating a local randomized experiment.
Regularization
Techniques that constrain model complexity to prevent overfitting. Includes explicit methods (L1/L2 penalty, dropout, weight decay) and implicit methods (early stopping, data augmentation, batch normalization).
ReLU
Rectified Linear Unit. An activation function defined as f(x) = max(0, x). Computationally efficient and avoids the vanishing gradient problem for positive inputs, but can cause dead neurons.
Reparameterization Trick
A technique for backpropagating through stochastic sampling by expressing the random variable as a deterministic function of parameters and independent noise: z = μ + σ ε, where ε ~ N(0,1). Enables gradient-based optimization of the VAE objective.
Residual Connection
A shortcut connection that adds a layer's input directly to its output: y = F(x) + x. Enables training of very deep networks by providing a gradient highway. Introduced by ResNet.
Roofline Model
A performance model that characterizes the peak achievable performance of a computation based on its arithmetic intensity and the hardware's peak compute and memory bandwidth. Operations below the roofline are memory-bound; those above are compute-bound.
RoPE (Rotary Positional Embeddings)
A positional encoding method that encodes position information by rotating the query and key vectors. Naturally decays attention with distance and supports extrapolation to longer sequences than seen during training.
S
Saddle Point
A point in the loss landscape where the gradient is zero but the point is neither a local minimum nor a local maximum — the Hessian has both positive and negative eigenvalues. Common in high-dimensional optimization and more problematic than local minima.
Scaled Dot-Product Attention
The core attention operation: Attention(Q, K, V) = softmax(QK^T / √dk) V. Scaling by √dk prevents the dot products from growing too large for high-dimensional keys.
Schema Evolution
The process of changing a data schema (adding columns, renaming fields, changing types) while maintaining backward and forward compatibility with existing consumers.
Score Matching
A technique for learning the score function (gradient of the log-density) of a data distribution. The foundation of score-based diffusion models: learn the score, then use Langevin dynamics for sampling.
Selection Bias
Bias in causal estimates arising from systematic differences between treated and untreated groups. In observational studies, treated units are typically selected in ways that correlate with the outcome.
Self-Attention
An attention mechanism where queries, keys, and values all come from the same sequence, allowing each position to attend to all other positions. The fundamental operation of the transformer architecture.
Sequential Testing
Statistical testing methods designed for continuous monitoring, where the analyst can examine results at any point during data collection without inflating the false positive rate.
Shadow Mode
A deployment strategy where a new model receives production traffic and generates predictions, but those predictions are not served to users. Allows evaluation on real traffic without risk.
SHAP
SHapley Additive exPlanations. An interpretability method based on Shapley values from cooperative game theory. Assigns each feature an importance value for a particular prediction, with guarantees of local accuracy, missingness, and consistency.
Shrinkage
The statistical phenomenon in hierarchical models where group-level estimates are pulled ("shrunk") toward the global mean. Groups with less data experience more shrinkage.
Simpson's Paradox
A phenomenon where a trend that appears in several subgroups reverses or disappears when the groups are combined. Often caused by confounding variables. A canonical motivation for causal thinking.
Singular Value Decomposition (SVD)
The factorization of any m×n matrix A into A = UΣV^T, where U and V are orthogonal matrices and Σ contains the singular values on its diagonal. The "fundamental theorem of data science" — connects to PCA, low-rank approximation, and matrix completion.
Spectral Graph Theory
The study of graph properties through the eigenvalues and eigenvectors of associated matrices (adjacency, Laplacian). Graph convolution in GCNs has a spectral interpretation as filtering in the graph Fourier domain.
State-Space Model
A model with a latent state that evolves over time according to a transition model, observed through a noisy observation model. General framework encompassing Kalman filters, HMMs, and structural time series models.
Structural Causal Model (SCM)
A formal framework consisting of structural equations, exogenous variables, and a causal graph that defines the data-generating process. Supports reasoning about interventions and counterfactuals.
Sufficient Statistic
A statistic T(X) that captures all the information in the data X about a parameter θ. Formally, the likelihood conditioned on T(X) does not depend on θ.
Surrogate Model
A probabilistic model (typically a Gaussian process) that approximates an expensive objective function in Bayesian optimization. Cheap to evaluate and provides uncertainty estimates.
SUTVA
Stable Unit Treatment Value Assumption. Two components: (1) no interference — one unit's treatment does not affect another's outcome; (2) consistency — the treatment is well-defined with no hidden variations.
Synthetic Control Method
A causal inference method that constructs a "synthetic" comparison unit as a weighted combination of untreated units. Used when only one (or few) treated units exist, such as the effect of a policy change on a single state or country.
T
Temperature Scaling
A post-hoc calibration method that divides model logits by a learned scalar temperature T before the softmax. T > 1 softens probabilities (reducing overconfidence); T < 1 sharpens them.
Temporal Fusion Transformer (TFT)
A transformer architecture for multi-horizon time series forecasting with interpretable attention. Features variable selection networks, gated residual connections, and temporal attention that identifies which time steps matter most.
Tensor Parallelism
A distributed training strategy that splits individual operations (e.g., large matrix multiplies) across GPUs. Each GPU computes a portion of the result. Requires high-bandwidth inter-GPU communication.
Thompson Sampling
A Bayesian approach to the multi-armed bandit problem: sample a reward estimate from each arm's posterior distribution and select the arm with the highest sample. Naturally balances exploration and exploitation.
Tokenizer
An algorithm that converts text into a sequence of integer token IDs. Common approaches include Byte-Pair Encoding (BPE) and SentencePiece. Vocabulary size affects model capacity and sequence length.
Training-Serving Skew
Discrepancies between how features are computed during training and during serving, causing model performance to degrade in production despite good offline metrics. The most common source of production ML bugs.
Transfer Learning
Leveraging knowledge from a model trained on one task (source) to improve performance on a related task (target). Strategies range from feature extraction (freeze all layers) to full fine-tuning.
Transformer
A neural network architecture based entirely on attention mechanisms (no recurrence, no convolution). Processes all positions in parallel using self-attention, enabling efficient training on GPUs. The dominant architecture for NLP, increasingly for vision and other domains.
Tree-Structured Parzen Estimator (TPE)
A Bayesian optimization algorithm that models the conditional probabilities p(x|y
Two-Stage Least Squares (2SLS)
The standard instrumental variable estimator. Stage 1: regress treatment on instrument. Stage 2: regress outcome on the fitted treatment values from Stage 1.
Two-Tower Model
A neural architecture for retrieval that encodes queries and items with separate "tower" networks into a shared embedding space. Similarity (typically dot product) between embeddings determines relevance.
U
Universal Approximation Theorem
The theorem stating that a feedforward neural network with a single hidden layer of sufficient width can approximate any continuous function on a compact set to arbitrary accuracy. Does not guarantee learnability or sample efficiency.
Uplift Modeling
A technique for predicting the incremental impact of a treatment (e.g., marketing campaign) on an individual's behavior. Identifies "persuadables" (positive uplift) and "sleeping dogs" (negative uplift).
Upper Confidence Bound (UCB)
A bandit algorithm that selects the arm with the highest upper confidence bound: UCB = μ̂ + c√(ln t / n). Implements "optimism in the face of uncertainty." Also used as an acquisition function in Bayesian optimization.
V
Vanishing Gradient
A training pathology where gradients shrink exponentially during backpropagation through many layers, causing early layers to learn extremely slowly. Mitigated by residual connections, proper initialization, normalization, and gating mechanisms (LSTM).
Variational Autoencoder (VAE)
A generative model that learns a latent representation by jointly training an encoder (maps data to latent distribution) and a decoder (maps latent samples to data), optimizing the ELBO.
Variational Inference
An optimization-based approach to approximate Bayesian inference that frames posterior computation as an optimization problem: find the distribution q(θ) that minimizes KL(q || p(θ|D)). Faster than MCMC but provides only an approximation.
Vector Database
A database optimized for storing and querying high-dimensional vector embeddings. Supports approximate nearest neighbor search for retrieval-augmented generation. Examples include FAISS, Pinecone, Weaviate, and ChromaDB.
W
Walk-Forward Validation
A time series cross-validation strategy that trains on data up to time t, evaluates on data from t to t+h, then advances the training window. Respects temporal ordering and avoids data leakage.
Wasserstein GAN (WGAN)
A GAN variant that uses the Wasserstein distance (Earth Mover's distance) as the training objective instead of the Jensen-Shannon divergence. Provides more stable training gradients and a meaningful loss metric that correlates with sample quality.
Weak Instrument
An instrumental variable with only a weak association with the treatment variable. Weak instruments amplify any violations of the exclusion restriction and produce biased, imprecise estimates. Diagnosed by the first-stage F-statistic (rule of thumb: F > 10).
Weight Decay
A regularization technique that penalizes large parameter values by adding a fraction of the current weight magnitude to the gradient update. In AdamW, weight decay is decoupled from the gradient, which is more correct for adaptive optimizers.
Weisfeiler-Leman Test
A graph isomorphism test based on iterative neighborhood aggregation. Standard message-passing GNNs are at most as expressive as the 1-WL test, meaning they cannot distinguish certain non-isomorphic graphs.
X
X-Learner
A meta-learner for CATE estimation that imputes the missing potential outcomes for each unit, then trains a model on the imputed treatment effects. Performs well when treatment and control groups differ in size.
Xavier/Glorot Initialization
A weight initialization strategy that sets initial weights from a distribution with variance 2 / (nin + nout). Maintains constant variance of activations across layers for linear and tanh activations.
Z
Zero-Shot
Using a model to perform a task it was not explicitly trained for, without providing any task-specific examples. Enabled by foundation models trained on diverse data.