Data Engineering Glossary — Key Terms & Definitions

A

ABAC: attribute-based access control. Permissions computed from attributes of the user, the resource, and the context rather than from a named role. More expressive than RBAC and much harder to audit.
acceptance criterion: the single condition that decides whether a project is done. This book's is that gold reconciles to source to the cent, with every difference explained by a documented, tested rule.
accumulating snapshot: a fact table with one row per process instance and several date columns filled in as the process advances. Useful for well-defined milestones, awkward because the rows are updated rather than appended.
accuracy: whether a value matches reality. The quality dimension assertions cannot check alone, because the platform has no independent access to reality; a reconciliation is the closest substitute.
acks: a Kafka producer setting for how many replicas must acknowledge a write. acks=all is the durable choice; the others trade durability for latency invisibly until a broker fails.
action item: the output of a postmortem that has an owner and a date. Without both it is a sentiment.
actionability: whether an alert tells its recipient what to do. What distinguishes an alert from a notification, and whose absence produces alert fatigue.
adaptive query execution: Spark's runtime re-planning from actual statistics: coalescing partitions, switching join strategies, splitting skewed ones. Handles many skew problems and is not a substitute for understanding them.
additive measure: a measure that can be summed across every dimension. Revenue is additive; a ratio is not, and summing a non-additive measure is a common modelling error.
after image: the state of a row after a change, in CDC. With the before image, what makes a CDC stream more informative than a snapshot.
aggregate: in event sourcing, the entity events are about and whose version they are scoped to. If you cannot identify yours, event sourcing is the wrong pattern.
alert fatigue: alerts dismissed in bulk because most need no action. Kestrel's was 412 alerts a quarter, 89% requiring none.
allowed lateness: how long after the watermark passes a window a processor still accepts events for it. A business decision expressed as configuration.
analytics engineering: transforming data in the warehouse with software engineering discipline. Largest job market of the specializations, closest to the business, shallowest technically.
anonymization: rendering data no longer relating to an identifiable person, placing it outside the regulations. The bar is far higher than removing names, and the claim is usually false.
Apache Arrow: an in-memory columnar format letting engines exchange data without serializing. Why pandas, Polars, and DuckDB can hand data to one another cheaply.
append-only: only ever added to, never updated or deleted from. Bronze's requirement, and what makes replay possible.
apply: in Terraform, the step that makes planned changes real. The one to gate, review, and log.
approximate nearest neighbour: finding close vectors without checking every one. Trades recall for latency in a way that must be measured.
AQE: see adaptive query execution.
architecture decision record: a short record of a decision, its context, alternatives, and consequences. Its value is entirely in being written before the decision is executed.
artifact: a build output that can be stored, versioned, and deployed. In a data platform the important one is dbt's manifest.json, because it makes "what changed" computable.
as-of join: a join taking, for each event, the latest matching row at or before its timestamp. What makes training data point-in-time correct.
asset: in asset-based orchestration, a declared thing that exists rather than a task that runs. Shifts a DAG from "do these steps" to "make these things true."
at-least-once: a message may be delivered more than once and will not be lost. The right default, given idempotent consumers.
at-most-once: a message may be lost and will not be duplicated. Almost never what you want for data.
atomic rename: making a file visible at its final path in one step. Object stores lack it, which is why table formats exist.
auto-suspend: a warehouse setting that stops billing when idle. Disabling it was Kestrel's single largest waste: $5,040 a month for a resource used three hours a day.
autonomy: a domain's ability to build, deploy, and change its products without coordinating. What data mesh requires, and what a fan-out analysis measures.
Avro: a row-oriented binary format with a schema, common in Kafka. Good for streams, wrong for analytical scans.

B

B-tree index: the default relational index. Excellent for point lookups and ranges, useless to a query that does not use it, and never free — every index slows writes.
backfill: recomputing history, usually after a bug or a new model. What makes idempotency non-negotiable.
backfill window: how far back a pipeline can be re-run, bounded by source retention. Must cover the age of the data you might reconstruct, not your detection time.
backflow: a lower layer reading a higher one; silver reading gold. It does not fail, which is the problem: it silently destroys the guarantee that the graph computes from raw.
backpressure: a consumer signalling it cannot keep up so the producer slows. The alternative to unbounded buffering, which fails later and worse.
backward compatibility: a new producer's output can still be read by an old consumer. Achieved by adding optional fields and never removing or narrowing existing ones.
batch: processing a bounded set at once. The default, and correct far more often than the streaming literature implies.
batch ingestion: loading on a schedule rather than continuously. Simpler, easier to backfill, and sufficient for most requirements.
batch scoring: running a model over a set of entities on a schedule. The mode that does not need a feature store.
before image: the state of a row before a change. Required to compute a delta, and not emitted by every CDC configuration.
behavioral interview: the round asking how you have worked. Mostly probing whether you can be wrong out loud, and whether you would be pleasant on call.
big bang migration: cutting over at a single moment. Defensible only when running both is impossible.
binlog: MySQL's transaction log, and what most CDC tools read. Postgres' equivalent is the write-ahead log.
bit packing: storing integers in the minimum bits their range needs. One of the encodings that makes columnar formats small.
bitmap heap scan: Postgres collecting matching row locations from an index into a bitmap, then reading the heap in physical order. Appears when an index helps and is not selective enough for a plain index scan.
blameless postmortem: an incident review treating human error as a symptom of system design. Not politeness: the only regime in which people report what actually happened.
blast radius: how much a change or failure can affect. In infrastructure, why state files are separated; in a data graph, the models a corruption invalidates.
bloat: dead tuples left by Postgres updates and deletes, occupying space until vacuumed. Grows silently and degrades scans.
block compression: compressing groups of values together rather than individually. Far better ratios, because it can exploit repetition.
blue-green: running two environments and switching between them. Adapts poorly to data, where the "traffic" is a table other things reference.
bottleneck: the constraint limiting throughput. Data mesh's premise is that a central team has become one — measurable, and usually unmeasured.
bounded context: from domain-driven design, a boundary within which a model and its vocabulary are consistent. Proposed data domains that do not correspond to one are directories.
breaking change: a change invalidating a consumer's assumptions. Removing a column, narrowing a type, or changing a value's meaning — the last does not look like a change.
bridge table: a table resolving a many-to-many between a fact and a dimension. Necessary and awkward: aggregation is ambiguous unless weights are stored.
broadcast join: sending a small table to every executor to avoid a shuffle. The most effective Spark optimization when it applies, and a memory hazard when the table is not small.
broker: a Kafka server, holding partitions and serving producers and consumers.
bronze layer: the layer holding exactly what the source sent: append-only, untyped, keeping what you do not want. What makes "it arrived that way" provable.
bucket: a fixed number of files a table is hashed into, so joins on that column avoid a shuffle. Powerful, inflexible, expensive to change.
budget: a spend threshold with an alert; in reliability, the unreliability an SLO permits.
build cache: reusing a step's output when its inputs have not changed. The difference between a two-minute and a forty-minute CI run.
burn rate: how fast an error budget is consumed, over a short recent window against the rate the SLO permits. Measured over the full window it collapses to "budget spent" and the thresholds cannot fire.
burnout: the symptom. The causes are unplanned work above half, custody without authority, no visible outcome, and a ceiling — with different remedies.
bus matrix: Kimball's grid of business processes against conformed dimensions. Makes shared dimensions obvious before you build them twice.
business metadata: what a dataset means, as opposed to how it is stored. The half of a catalog people actually read.
business rule: a decision about meaning that somebody owns. Three of Kestrel's four reconciliation rules belonged to finance, and all three had been decided silently in SQL.

C

canary: a deployment that exposes a change to a small share of traffic first. Adapts poorly to data, where the equivalent is shadow running and diffing outputs rather than watching for errors.
CAP theorem: under a network partition, a distributed system must choose between consistency and availability. Frequently invoked and rarely the actual constraint; the useful version is that the choice is per-operation, not per-system.
capstone: the assembled platform, judged by reconciliation rather than by whether it runs.
cardinality: the number of distinct values a column or metric label takes. The property that decides whether a time-series label is fine or catastrophic: customerid at 1.9M values produces 9.12 billion series.
catalog: in a table format, the service mapping a table name to its current metadata pointer. In governance, the searchable record of what data exists and what it means.
catalyst: Spark's query optimizer. Rewrites logical plans, and the reason explain() output looks nothing like the code you wrote.
catchup: Airflow's behaviour of running every missed interval when a DAG is unpaused. Extremely useful for backfills and a common cause of a thousand accidental runs.
CCPA: the California Consumer Privacy Act. Different structure from GDPR, the same six technical capabilities, a 45-day response window.
certification: marking a dataset as trusted, by a named owner, with a date. The status tier that makes a catalog's search useful rather than exhaustive.
change data capture: reading a database's transaction log to produce a stream of row changes. Gives update history and low source load, at the cost of operational complexity and a replication slot that can fill the source's disk.
change detection: determining which rows have changed since the last load. Usually by updatedat, which lies in four distinct ways.
change stream: MongoDB's CDC mechanism. The same idea as a binlog reader, different vocabulary.
chargeback: moving cloud spend onto the consuming team's budget. Creates real incentives, and also real incentives to argue about attribution; showback first.
chasm trap: a dimensional modelling hazard where two many-to-one relationships from a shared dimension produce a fan-out on join. Detected by row counts, not by inspection.
check strategy: how an incremental model decides which rows are new: a timestamp, a high-water mark, a hash, or a CDC log. Each has a different failure mode.
checkpoint: writing intermediate state durably so a job can resume. In streaming, the mechanism behind exactly-once processing within a framework's boundary.
chunking: reading or writing data in bounded pieces rather than all at once. The difference between a load that works at 10x volume and one that does not.
circuit breaker: stopping requests to a failing dependency for a period rather than retrying into it. Protects the dependency as much as the caller.
classification: assigning a sensitivity tier to data. A legal determination that engineering makes operable, not one engineering makes.
clear and rerun: Airflow's operation to re-execute a task and its downstream. Safe only if the tasks are idempotent, which is why Chapter 20 comes first.
clickstream: the event stream of user activity. Kestrel's is 14,000,000 events/day, 11.48 GB/day of JSON, and the largest thing in the platform.
clock skew: the difference between machines' clocks. Small, unavoidable, and the reason event-time processing cannot rely on a producer's timestamp being ordered.
clustering key: the column order a table's data is physically sorted by. Determines which predicates can skip data; a storage decision with query consequences.
coalesce: reducing partition count without a full shuffle. Cheap, and it can produce very uneven partitions, which is sometimes what you wanted and sometimes not.
column chunk: in Parquet, one column's data within one row group. The unit that statistics describe and that predicate pushdown skips.
column mapping: a table format feature letting a column be renamed without rewriting data, by mapping names to stable field IDs.
column-level lineage: tracking which source columns feed which output columns. Expensive to produce, and it answers impact-analysis questions table-level lineage cannot.
columnar format: a file format storing values column by column: Parquet, ORC. Compresses far better, and lets a query read only the columns it needs.
columnar storage: the same idea at the storage-engine level. The single largest performance difference between an analytical and a transactional system.
command: a request for something to happen, which can be rejected. Distinct from an event, which has already happened; imperative event names mean you have a command queue.
commit: in a table format, atomically publishing a new table version. The operation object stores cannot do natively and that table formats implement.
communications lead: the incident role that talks to everyone not fixing the problem. Separating it from the incident commander is what stops the person debugging from writing status updates.
compaction: rewriting many small files into fewer large ones. In Kafka a different thing entirely: retaining the latest record per key, which destroys an event-sourced log.
completeness: whether all the expected data arrived. The quality dimension a row count checks, and the one most likely to fail silently.
compression ratio: uncompressed size over compressed size. Kestrel's clickstream is 12.3x from JSON to Parquet, and most of that is the format rather than the codec.
concept drift: the relationship between features and the target changing over time. A modelling problem rather than a data engineering one, and frequently blamed for what is actually skew.
concurrency: how many tasks may run at once, at the DAG, pool, or cluster level. The setting that turns a resource problem into a queue.
conformed dimension: a dimension used consistently across multiple fact tables. What makes cross-process analysis possible, and what silver exists to produce.
connector offset: a CDC connector's position in the source log. Losing it means a full re-snapshot; advancing it wrongly means silent data loss.
consent: permission to process data for a purpose. Temporal, per-purpose, revocable, and one of several lawful bases — not a boolean.
consistency: whether related values agree: across systems, across time, or across two computations of the same metric.
consistent hashing: assigning keys to nodes so that adding or removing a node moves as few keys as possible. The mechanism behind partitioned stores that can be resized.
consumer: a process reading from a topic. Its offset, not the broker, records what it has seen.
consumer group: a set of consumers sharing a topic's partitions, each partition assigned to exactly one member. The unit of parallelism and of rebalancing.
consumer lag: how far behind the log's head a consumer is. The single most important streaming metric, and the one that reveals a stalled consumer before its output does.
consumer rebalance: reassigning partitions when group membership changes. Necessary, and it briefly stops processing and can replay from a committed offset.
consumer-driven contract: a contract asserted by the consumer against the producer, so the producer's tests fail when they break someone. Inverts the usual direction, and is what makes a contract enforceable.
continuous deployment: automatically releasing every change that passes CI. In data, the question is what "release" means when the artifact is a table.
continuous integration: running the build and tests on every change. For a data platform, building the models against a sample is the version that catches most defects.
contract test: a test asserting that an external API still behaves as assumed. The cheapest protection against a vendor changing something quietly.
controller: in privacy law, the party that decides why data is processed. Determines who is accountable; a legal determination worth knowing the word for.
Conway's law: organizations design systems that mirror their communication structure. Data mesh's foundation — and the inverse maneuver, changing the organization to get the architecture, is a reorganization and should be argued as one.
copy-on-write: a table format's update strategy that rewrites whole files on change. Simple reads, expensive writes; the alternative is merge-on-read.
correlated subquery: a subquery referencing the outer query's row. Readable and frequently slow; most can be rewritten as a join or a window function.
cost attribution: assigning spend to teams and pipelines. 36.4% of Kestrel's bill had no owner, and every dollar of it was waste.
cost per order: a unit cost. $0.1526 for Kestrel's platform and $0.0864 after optimization — and the pipeline-only figure of $0.0181 is a different number, and a flattering one.
cost per pipeline: spend attributed to one DAG. What turns "compute is up 69%" into a two-hour investigation.
cost per query: the estimate that should appear in code review, where it can still change a decision.
coupling: how much one component's change forces another's. The dimension along which architecture decisions are actually made.
covering index: an index containing every column a query needs, so the table is never read. Fast, and it duplicates data and slows writes.
CPRA: the California Privacy Rights Act, which amended CCPA. Evidence for §31.1's warning that this material dates.
CQRS: command query responsibility segregation: separate models for writing and reading. In event sourcing it arrives as a consequence rather than as a pattern to adopt.
credit: Snowflake's billing unit. A Medium warehouse consumes 4 an hour, at $2.00 each on this book's frozen rate card.
crypto-shredding: encrypting each subject's data with a per-subject key and deleting the key to render it unreadable. The best available answer to deleting from an immutable store, and a substantial key-management commitment.
CSV: comma-separated values. Ubiquitous, schemaless, ambiguous about types and quoting, and the format your most important vendor will send you.
CTE: a common table expression, WITH... AS. Readability, and in some engines an optimization fence — check yours before relying on it either way.
cursor pagination: paginating by an opaque token rather than an offset. Correct under concurrent writes, which offset pagination is not.
cutover: the moment consumers switch from the legacy system to the new one. Gated by a readiness rule rather than by a feeling, and preceded by a rehearsed rollback.

D

DAG: directed acyclic graph. The shape of a pipeline's dependencies, and in Airflow the unit of scheduling.
data architecture: the set of decisions about where data lives, how it moves, and who may change it. Mostly a record of trade-offs, and worth writing down for that reason.
data catalog: the searchable record of what data exists, what it means, and who owns it. Worth having when it answers questions people currently ask a person.
data contract: an agreement about a dataset's schema, semantics, and guarantees, enforced by tests rather than by goodwill.
data docs: Great Expectations' generated documentation of an expectation suite and its results. Useful because it is a by-product rather than a task.
data downtime: the period during which data is missing, late, or wrong. Distinct from system downtime, and usually far longer because nothing alerts on it.
data drift: a change in the distribution of a model's inputs. Detected by comparing feature distributions in training and serving, which is the highest-value ML monitor.
data engineering: building and operating the systems that move and shape data so somebody can rely on the result. Not machine learning, not analysis, and not plumbing.
data engineering lifecycle: generation, ingestion, storage, transformation, and serving, with governance across all five. A map for locating a problem, not a methodology.
data impact: the required field in an incident record stating which data is wrong, for which period, and who consumed it. What turns an outage report into a data incident report.
data interval: in Airflow, the window of data a run is responsible for, as distinct from when it executes. Confusing them is the most common Airflow bug.
data lake: files in an object store, queried in place. Cheap, flexible, and without a table format it has no atomicity, no schema evolution, and no row-level delete.
data lakehouse: a lake with a table format on top, giving warehouse-like guarantees over open files. The architecture most new platforms land on.
data leakage: training on information that did not exist at prediction time. Produces a model that scores excellently offline and badly in production, immediately.
data mesh: domain-oriented ownership, data as a product, a self-serve platform, and federated computational governance. A real answer to a real coordination problem, and the most misapplied idea in the field.
data minimization: collecting and keeping only what you need. The cheapest privacy control and the most under-used, because dropping a column feels like losing an option.
data pipeline: a sequence of steps that moves and transforms data. The unit most of this book is about.
data product: a dataset treated as a product: a known consumer, a contract, an SLO, documentation, and an owner who can act.
data product owner: the person accountable for a data product's quality and roadmap. In a mesh, a domain role; without a mesh, still a useful one.
data quality: whether data is fit for the decisions made on it. Measured by assertions, not by inspection.
data residency: a requirement that data remain in a jurisdiction. Constrains storage, processing, and — the part that gets missed — logs, metrics, and error payloads.
data subject: the person data is about. The unit privacy obligations operate on, and the access pattern most platforms are bad at.
data swamp: a lake with no catalog, no conventions, and no owners. The predicted outcome of a lake adopted without governance, and a common one.
data transfer: moving bytes between regions, zones, or out to the internet. Priced irrationally, easy to incur accidentally, and 5.6% of Kestrel's bill.
data warehouse: a database optimized for analytical queries over modelled data. Still the right default for most organizations.
DataFrame API: Spark's typed, optimizer-visible API, as opposed to RDDs. Almost always the right choice, because Catalyst can see what you meant.
DataOps: applying DevOps practices to data: version control, CI, monitoring, and small reversible changes. Useful as a checklist, weak as an identity.
dataset: in Airflow, a declared data object that can trigger a DAG when updated. The bridge between task-based and asset-based scheduling.
dbt package: a reusable set of dbt models, macros, and tests. dbtutils is the one nearly every project ends up depending on.
dead letter: a record that failed processing and was set aside rather than dropped. Only useful if somebody drains the queue.
dead letter queue: the streaming version. A queue nobody drains is a queue that grows until somebody truncates it.
Debezium: the most widely used open-source CDC platform, reading database logs and producing Kafka records. Also the standard way to implement the outbox pattern.
declarative: describing the desired end state rather than the steps to reach it. Terraform's model, and the reason a plan can be reviewed before it is applied.
decommissioning: removing a migrated system: the job, the code, the credentials, and the infrastructure. Deleted, not disabled — a disabled job gets re-enabled.
deduplication: reducing multiple rows for one entity to one. Requires a deterministic tie-break, or the result differs between runs and no uniqueness test detects it.
defer: dbt's mechanism for resolving unbuilt models against another environment, so CI can build only what changed.
deferrable operator: an Airflow operator that releases its worker slot while waiting. The fix for sensors deadlocking a pool.
definition of done: the condition under which work is finished. For a pipeline it is not "it runs"; it is "it reconciles."
Definitions are deliberately short and deliberately opinionated: Where a term is contested — "data mesh," "exactly-once," "real-time" — the entry says so rather than pretending consensus.
degenerate dimension: a dimension attribute stored on the fact table because it has no other attributes, such as an order number. Correct, and it looks like a mistake.
degraded mode: running with reduced function rather than stopping. Worth designing in advance, because the alternative is deciding during an incident.
delete+insert: an incremental strategy that deletes a partition and re-inserts it. The simplest way to be idempotent, and it needs a well-chosen partition key.
deletion vector: a file recording which rows in a data file are deleted, so a delete does not rewrite the file. What makes row-level deletes affordable and privacy requests tractable.
delta encoding: storing differences between consecutive values rather than the values. Very effective on sorted or slowly-changing columns.
denormalization: duplicating data to avoid joins. Right in a document store and in a serving layer; a source of update anomalies everywhere else.
DENSE_RANK: a window function ranking without gaps after ties. Distinct from RANK and ROWNUMBER, and the distinction is a common interview question because it is a common bug.
deprecation: announcing that something will be removed, with a date, while it still works. The half of a breaking change that makes it survivable.
depreciation: the loss of value in a skill over time. Products depreciate; properties do not.
dictionary encoding: replacing repeated values with integer references to a dictionary. The encoding that makes low-cardinality string columns nearly free.
dictionary page: where Parquet stores that dictionary, per column chunk.
differential privacy: adding calibrated noise so that an individual's presence cannot be inferred, with a mathematical guarantee that survives composition. Rigorous, and the noise is large enough to matter for business analytics.
digest pinning: referencing a container image by content hash rather than by tag. The difference between a reproducible deploy and a hopeful one.
dimension table: a table of descriptive attributes joined to facts. Wide, small, and where the business's vocabulary lives.
dimensional modeling: organizing a warehouse into facts and dimensions. Thirty years old and still the default for good reasons.
discoverability: whether somebody can find the right dataset without asking a person. The catalog's actual job.
distribution: the shape of a metric's values, as opposed to its average. Where the interesting failures live, and what a p99 exists to expose.
distribution test: an assertion about a column's distribution rather than its individual values. Catches drift that row-level checks cannot.
Docker: containerization. In a data platform, mostly the mechanism for making a job's environment reproducible.
docker compose: running a multi-container local environment from one file. How this book's sandbox runs Postgres, MinIO, and Kafka on a laptop.
document store: a database storing schemaless documents, typically JSON. Good for varied shapes and heterogeneous access; bad for analytical aggregation.
domain: in a mesh, a business area that owns its data. Real only if it can build and change its products without coordinating.
domain-oriented ownership: the mesh principle that the team producing data owns the data product built from it. The only one of the four that requires decentralization.
downsampling: reducing the resolution of time-series data as it ages. The standard way to keep a metrics store from growing without bound.
drift: infrastructure differing from its declared state, usually from a console change during an incident. Detected by a scheduled plan, not by hoping.
driver: the Spark process that plans the job and coordinates executors. Also where collecting a large result will run you out of memory.
DSAR: a data subject access request: produce everything you hold about a person. Harder than deletion, because "everything" is a judgment about scope.
dual write: writing to a database and then publishing an event, without a transaction spanning both. Loses events silently; the fix is a transactional outbox.
DuckDB: an in-process analytical database. Fast, dependency-free, and the reason this book can teach warehouse concepts without a cloud account.
duration: how long a job took. Monitored with an absolute threshold in most platforms, which cannot detect a job doing ten times the work in three times the time.
dynamic allocation: Spark adding and removing executors based on demand. Keeps wall-clock stable and decouples cost from latency, which removes a signal.
dynamic data masking: transforming a column's value at query time based on the querying role. Attached to the object, auditable in one place, and fails closed.
dynamic task mapping: Airflow generating tasks at runtime from a list. Useful, and it makes the DAG's shape depend on data, which complicates monitoring.

E

eager evaluation: computing each operation as it is written. pandas' model; simple to debug and it materializes intermediates you did not need.
egress: data leaving a cloud region or provider. $0.09/GB on this book's rate card, and the least rational line on any bill.
elasticity: the ability to add and remove capacity on demand. What the cloud sells, and it only saves money if you actually release the capacity.
ELT: extract, load, transform: land raw data first and transform in the warehouse. The default today, and wrong when you legally may not land the raw data.
embedding: a vector representation of an item, used for similarity search. A model output, so it has a version as well as a value.
enabling team: from Team Topologies, a team that helps others acquire a capability rather than doing the work for them. What a platform team becomes in a mesh.
entity: the thing a feature is about: a customer, an order, a product, or a pair. Its key is a join key, and Chapter 6's discipline applies.
entity key: the identifier a feature is stored and looked up by.
environment: a separate place to run: dev, staging, production. In data, the hard part is that staging cannot have production's data.
ephemeral model: a dbt model compiled into its consumers as a CTE rather than materialized. Cheap, and it makes debugging harder because it does not exist.
erasure: deleting everything about a person, within a statutory deadline. Requires finding them first, in every identifier space.
error budget: the amount of unreliability an SLO permits. What turns a reliability argument into an arithmetic one.
escalation: handing an incident to someone with more context or more authority. Should have a stated trigger, or it happens too late.
ETL: extract, transform, load: transform before landing. Right when you cannot store the raw data, and the strongest argument for it is legal rather than technical.
event: an immutable, past-tense statement that something happened, identified and versioned within its aggregate.
event carried state transfer: an event carrying the new state of an entity, so consumers need not call back. What CDC produces, and what most "event-driven" data platforms actually run on.
event envelope: the metadata wrapping a CDC record: source, operation, timestamps, and position. Distinct from the payload, and where the useful debugging information lives.
event notification: an event carrying only "X changed" and an id. Cheap and loosely coupled, and it creates a synchronous dependency back on the producer.
event sourcing: treating the event log as the system of record, with every table derived. A large commitment whose payoff is retroactive projections and debugging.
event store: the durable, append-only home of events.
event time: when something happened, as opposed to when it was processed. The distinction the whole of stream processing is organized around.
event-driven architecture: designing around events rather than tables. Covers three different things with different costs, and conflating them is the main source of over-engineering.
eventual consistency: replicas converge given no new writes. Fine for many things and surprising when a read-after-write returns the old value.
exactly-once: a guarantee that a message is processed exactly once. Real within a framework's boundary and not across a boundary; the practical version is at-least-once plus idempotent consumers.
executor: a Spark process that runs tasks and holds cached data.
expand-contract migration: add the new thing, migrate consumers, then remove the old one. The pattern that makes a breaking change non-breaking, at the cost of a period with both.
expectation suite: a named set of assertions in Great Expectations.
exponential backoff: increasing the wait between retries. With jitter, the difference between recovering from an outage and prolonging it.
exposure: a dbt declaration of a downstream consumer — a dashboard, an ML model, an export. Under-used, and it directly answers "if this breaks, who notices?"
extract manifest: a record of what a load extracted: files, row counts, and the high-water mark. Makes a load auditable and re-runnable.

F

fact table: a table of measurements at a declared grain, joined to dimensions. Long, narrow, and the thing everything else exists to describe.
factless fact table: a fact table with no measures, recording that an event occurred. Attendance, coverage, and eligibility are the usual cases.
fan trap: a join that multiplies rows because a one-to-many relationship is traversed before an aggregation. Produces a number that is too large and looks plausible.
feature: a value about an entity, at a moment. The third part is the one that gets dropped.
feature age: the gap between when a feature value was computed and when it is used to predict. Almost never measured, and a 132x mismatch between training and serving is normal.
feature freshness: how recent the served feature is. Must be compared against training, not minimized.
feature store: an offline store for training and an online store for serving, sharing one feature definition. Worth it only with online inference, duplicated computation, and more than one consumer.
feature versioning: tracking a feature definition's changes, so a model can be pinned to the one it was trained on. A materially changed definition is a new feature with a new name.
federated computational governance: global standards decided by a federation and enforced by automation. The best idea in the mesh, and the computational half works without any decentralization.
file compaction: merging small files into larger ones. Improves scan performance and request costs, and its cost justification is usually much weaker than its latency justification.
FinOps: the practice of managing cloud spend as an engineering concern. Useful as a checklist; be skeptical of its maturity models.
fixed and variable cost: the split that makes a unit cost predictive. Kestrel is ~64% variable, so doubling volume raises the bill 64% and lowers cost per order 18%.
fixture: a small, fixed dataset used for testing. The thing that makes a CI run fast enough to be run on every change.
fold: reducing a sequence of events to a state. state = fold(reducer, events) is event sourcing's central identity.
footer: the end of a Parquet file, holding the schema and the row-group statistics. Read first, which is why a reader can skip most of a file.
format-preserving encryption: encryption whose output has the same shape as its input. Useful when a downstream system validates the format of a value it does not need to read.
forward compatibility: an old producer's output can still be read by a new consumer. The mirror of backward compatibility, and the one people forget to test.
forward fix: resolving a bad deploy by shipping a correction rather than rolling back. Frequently the only option in data, because the previous state has been overwritten.
frame clause: the ROWS/RANGE/GROUPS part of a window function. RANGE is the default, it includes peer rows, and the difference matters whenever the ordering column has duplicates.
freshness: how old the data is. The first thing to monitor and the one most often expressed as "did the job run" instead.
freshness check: an assertion that data is no older than a stated threshold.
full compatibility: both backward and forward compatible. The strictest schema-evolution mode and the one that permits the fewest changes.
full load: extracting everything, every time. Simple, correct, and it stops scaling at a size you can compute in advance.
full refresh: rebuilding a model from scratch rather than incrementally. The default in dbt, and the most common expensive mistake in a mature project.

G

gaps and islands: the SQL pattern for finding runs of consecutive values, usually by differencing a row number against the value. Sessionization is its most common instance.
GDPR: the EU General Data Protection Regulation. Six technical capabilities, a one-month response deadline, and a definition of personal data broader than most engineers assume.
generate: the lifecycle stage where data comes into existence, usually in a system you do not control. The only stage where you can prevent a problem rather than handle it.
generic test: a reusable dbt test applied by name in YAML: unique, notnull, acceptedvalues, relationships. Four lines that catch most schema-level defects.
gold layer: the layer answering business questions at a documented grain, with named and owned definitions. Disposable: reconstructible from silver.
grain: what one row represents. The single most important sentence about a table, and the field most often missing from a catalog.
grain test: an assertion that the declared grain holds: a uniqueness check on the grain columns. Nine lines of dbt YAML, and it catches every fan-out.
grant: a permission on a database object. Accumulates, never gets narrowed, and 62 identities had one at Kestrel while 23 used it.
Great Expectations: a Python data-validation framework. Expressive, with generated documentation, and heavier than a dbt test for the same assertion.
gross merchandise value: total sales value before deductions. Kestrel's is $182.0M a year, and it is the denominator for the unit costs in.
gross revenue: revenue before refunds are netted, and before the rules that remove test, cancelled, and gift-card rows. Compute Kestrel's in Chapter 38; it is deliberately not printed here.
group coordinator: the Kafka broker managing a consumer group's membership and offsets.
GUI ETL: a graphical ETL tool such as Informatica or SSIS. The hardest legacy category, because the logic is in a proprietary repository rather than in a file you can read.
gzip: a general-purpose compression codec. Good ratios, slow, and not splittable — which matters when a single large file becomes a single task.

H

handoff: transferring an incident between responders. Needs a stated protocol, or context is lost precisely when it is expensive.
handover: making a platform someone else's to run. Six artifacts, of which the "not done" list is the most valuable and the most often skipped.
hard delete: physically removing a row, as opposed to flagging it. Invisible to a timestamp-based incremental load, which is why deletes need CDC or a reconciliation.
hash diff: hashing a row's attributes to detect change without comparing every column. The mechanism behind efficient SCD Type 2 loads.
heap: the unordered table storage a Postgres index points into. A bitmap heap scan is what happens when the index says "these rows" and the table must be visited.
heartbeat: a periodic signal that a process is alive. Detects a stopped scheduler, which is the failure that produces no events at all and therefore no alerts.
high-water mark: the maximum value of a monotonic column seen in the last load, used to select new rows. Simple, and it silently misses updates to old rows and any hard delete.
Hive-style partitioning: encoding partition values in directory names, year=2026/month=11/day=27. Universally understood and the reason a path is metadata.
hot partition: a partition receiving disproportionate traffic because of key skew. High cardinality does not prevent it: 1.9 million customers, one at 8.1%.
HUGEINT: DuckDB's 128-bit integer, which SUM(BIGINT) returns. It has no Parquet primitive, so it lands as a DOUBLE and money becomes a float.

I

idempotency: running an operation twice produces the same result as running it once. The single most important property in this book.
idempotency key: a caller-supplied identifier letting an API deduplicate retried requests.
idempotent consumer: a consumer that ignores a duplicate, usually by storing the last version seen per aggregate. What makes the outbox pattern's duplicates harmless.
idempotent producer: a Kafka producer that deduplicates its own retries. Prevents reordering within a partition as a side effect.
idempotent rebuild: rebuilding a model from its inputs and getting the same result. The property the quarterly rebuild verifies and that three defects in Chapter 38 violated.
idempotent sink: a stream destination where writing the same record twice has no additional effect. What makes at-least-once delivery acceptable.
image: a container image. Pin it by digest, not by tag.
immutability: never changing what has been written. Bronze's rule, event sourcing's rule, and the property that makes history trustworthy.
import: bringing an existing resource under Terraform's management. The operation that converts drift into declared state.
in-sync replica: a Kafka replica caught up with the leader. acks=all waits for these, and min.insync.replicas decides how many must exist for a write to succeed.
incident commander: the person coordinating an incident, who should not also be the person debugging it.
incremental load: loading only what has changed. Necessary at scale, and every check strategy for deciding "what changed" has a failure mode.
incremental migration: replacing a legacy system one piece at a time, with both running. The strangler fig, and the default unless running both is impossible.
incremental model: a dbt model that processes new rows rather than rebuilding. Requires a unique key and an idempotent strategy, or reruns double-count.
incremental snapshot: Debezium's ability to snapshot a table in chunks while streaming continues. Removes the long lock that made initial CDC loads painful.
index scan: reading an index to find rows. Fast when selective; slower than a sequential scan when it is not, which is why the planner sometimes ignores your index.
individual contributor: an engineer without direct reports. A track, not a consolation, and the staff and principal levels are on it.
inference: running a model to produce a prediction. Batch or online, and which one decides most of the feature infrastructure.
infrastructure as code: declaring infrastructure in version-controlled files. Makes a change reviewable before it is real, which is most of the value.
ingest: the lifecycle stage where data enters your systems. The stage with the most failure modes and the least glamour.
ingestion tool: Fivetran, Airbyte, Meltano, or code. Buy for standard sources, build for the ones your business depends on.
insert_overwrite: an incremental strategy replacing whole partitions. Idempotent by construction, and it requires the partition to align with the load's unit.
integer cents: storing money as an integer number of the smallest unit. This book's rule, and validate.py fails the build on a float money column.
integration test: a test exercising components together. In data, usually "build the models against a fixture and assert the output."
interoperability standard: a global rule every domain follows so their products compose. The "computational" half of federated governance.
inverse Conway maneuver: changing the organization to get the architecture you want. Powerful, legitimate, and a reorganization — which is who must approve it.
inverted index: a map from term to the documents containing it. The structure behind text search.
is_current: the flag marking the active row of an SCD Type 2 dimension. Exactly one per natural key, and asserting that catches every Type 2 bug.

J

jitter: randomness added to a retry delay so that clients do not retry in lockstep. Without it, backoff synchronizes a thundering herd.
job: in Spark, the work triggered by one action. Divided into stages at shuffle boundaries.
JSON Lines: one JSON object per line. Splittable, streamable, verbose, and the format most event data arrives in.
JSONB: Postgres' binary JSON type, indexable with GIN. Useful for genuinely variable attributes; a trap when used to avoid modelling.
junk dimension: a dimension combining several low-cardinality flags into one table. Keeps the fact table narrow, and it is exactly as inelegant as its name.

K

k-anonymity: every combination of quasi-identifiers appears at least k times. A floor, not a target; and a segment extract's k is much worse than its population's.
Kafka: a distributed, partitioned, replicated log. The default event backbone, and more operationally involved than its API suggests.
Kafka Connect: a framework for moving data into and out of Kafka with configuration rather than code. Where Debezium runs.
Kappa architecture: one streaming path, with reprocessing by replaying the log. The response to Lambda's duplicated logic, and its own answer is now mostly obsolete too.
key: in an object store, the full path of an object. There are no directories; the slashes are part of the name.
key-value store: a database with get and put by key. Fast, simple, and it pushes every access pattern into the key design.
Kubernetes: a container orchestrator. In a data platform, mostly a way to run jobs; adopt it because you already have it, not to run a scheduler.

L

l-diversity: requiring the sensitive attribute to vary within each equivalence class. k-anonymity's answer to the homogeneity attack, and the first thing anybody asks about after k.
label: the target a model is trained to predict. Needs a definition, two timestamps, a delay, and a known source — the same rigor as a feature.
labeling delay: the gap between an event and knowing its label. A 90-day churn definition makes your newest training data a quarter old.
LAG: a window function returning a previous row's value. With LEAD, the pair that turns a self-join into one pass.
Lambda architecture: parallel batch and streaming paths reconciled at serving. A careful response to 2011's constraints, and worth reading as history.
late-arriving data: records that arrive after the window they belong to. Handled by a lookback window in batch and by watermarks in streaming.
late-arriving dimension: a fact arriving before its dimension row exists. Handled by an inferred member or by deferring the fact; both are decisions worth writing down.
latency: the delay between a cause and its effect being visible. In data, ask for it as a number in seconds, because "real-time" means four different things.
lateral join: a join whose right side can reference the left side's columns. How an as-of join is written portably.
lawful basis: the legal ground for processing personal data. Consent is one of several, and a consent check on a flow running on contractual necessity is a bug.
layer: in containers, a filesystem diff; ordering them by change frequency is what makes builds fast. In ch34, one of bronze/silver/gold.
layer boundary: the interface between two layers, and the place a guarantee is asserted. Free to draw and worthless unenforced.
layer drift: a model doing work that belongs to a different layer, acquired one four-line shortcut at a time. The normal end state without enforcement.
layer skipping: a layer reading two levels down, such as gold reading bronze. Bypasses every guarantee silver provides, and forces gold to cast.
lazy evaluation: building a plan and executing only when a result is required. Lets the optimizer see the whole query, and makes debugging harder because nothing has happened yet.
LEAD: a window function returning a following row's value.
leader-follower: one replica accepts writes and others copy it. The default replication model, and the reason a failover has a window of possible data loss.
least privilege: granting only the access required. Easy to state, and it decays because narrowing a grant has a cost and no visible benefit.
legacy system: a system that cannot be safely changed: no tests, no owner, no understanding. A property of your relationship with it, not of its age.
levelling: assigning a role to a level. Decided by scope, mostly in the recruiter screen, and under-levelling is far harder to correct after joining.
lifecycle policy: a rule that transitions or expires objects by age. The enforcement half of every retention policy that currently has only a period.
lineage: the record of which datasets derive from which. Answers impact analysis, and the question people actually ask is "if this breaks, who notices?"
lineage depth: how many transformations a model is from raw. A structural measurement that survives relabelling, and a stg model at depth 3 is a self-evident contradiction.
link header: the HTTP header carrying pagination URLs. Preferable to constructing them yourself, because the server knows its own cursor semantics.
lint: automated style and correctness checking. In a data project, SQLFluff plus the project rules that encode your conventions.
liquid clustering: Delta's incremental clustering that avoids full rewrites. A newer answer to the problem bucketing solved rigidly.
live coding: an interview round writing code while observed. Scored on process, so narrate, and say what you are stuck on.
load-bearing spreadsheet: a spreadsheet in a production path, maintained by a person. Not a technical migration: separate the rules from the judgments and migrate only the rules.
log: an append-only record of events. In ch36, the source of truth rather than a diagnostic.
log compaction: Kafka retaining only the latest record per key. Correct for a state-transfer topic, destructive for an event-sourced one.
log sequence number: a monotonic position in a database's write-ahead log. The deterministic tie-break a dedup needs.
logical date: Airflow's identifier for the data interval a run covers, distinct from the wall clock. Keying tasks on it is what makes reruns idempotent.
logical decoding: Postgres' mechanism for turning WAL records into row-level changes. What Debezium consumes.
logical replication: Postgres replication at the row level rather than the block level. The foundation CDC is built on.
long-running transaction: a transaction held open, preventing vacuum and growing bloat. A common cause of a Postgres database degrading for no apparent reason.
lookback window: reprocessing the last N days on every incremental run, to catch late-arriving data. The cheap approximation of a watermark.
LZ4: a fast compression codec with modest ratios. The right default when CPU matters more than bytes.

M

managed service: a component the vendor operates. Buy it unless operating it is your differentiator, and remember you are buying their failure modes too.
manifest: a file listing the data files in a table version. What makes a listing cheap and a commit atomic.
margin: in monitoring, the gap between current performance and the threshold. Publish it rather than the raw value: a deadline met by eight minutes and by three hours look identical otherwise.
mart: a set of gold models serving one business area. A packaging convention, not a guarantee.
materiality: the size below which a difference is not worth chasing. Reasonable against an external source and not reasonable for a system compared against itself, where the correct difference is zero.
materialization: writing a query's result rather than recomputing it; in dbt, the choice of table, view, incremental, or ephemeral. In ch32, copying offline features into the online store.
materialized view: a stored, refreshable query result. Fast reads, and a refresh strategy you now own.
medallion architecture: bronze, silver, gold. Nearly free, applies almost anywhere, and worth nothing unenforced.
memory multiplier: how much RAM an engine needs relative to the data. pandas' is roughly 5–10x and is the reason a "small" file exhausts a laptop.
mentorship: improving someone else's work deliberately. One of the four kinds of senior evidence, and the one that leaves no artifact unless you record it.
merge: an upsert: update matching rows, insert the rest. dbt's default incremental strategy on engines that support it, and it needs a genuine unique key.
merge-on-read: a table format update strategy writing delete vectors and new rows, resolved at read time. Cheap writes, more expensive reads, and compaction to reconcile.
metadata: data about data. Technical (types, sizes, lineage) and business (meaning, grain, ownership); the second is the half that gets read.
metric: a measured value over time. In ch30, a business quantity with a definition somebody owns.
metric definition: the rule computing a metric. Defined once or defined four times, and Kestrel had four definitions of activecustomer for twenty-eight months.
micro-batch: processing small batches frequently. Most of streaming's benefit with most of batch's simplicity, and usually the right answer to "we need real time."
micro-partition: Snowflake's automatically-managed storage unit. Removes manual partitioning and makes the clustering key the lever that remains.
microbatch: dbt's incremental strategy processing a bounded time window per batch. Backfills become parallel and reruns become idempotent by construction.
migration: a change to a schema or a system. In ch37, replacing a legacy platform: the work most data engineers are actually hired to do.
MinIO: an S3-compatible object store that runs locally. Lets this book's boto3 code be unmodified against a laptop.
ML infrastructure: the specialization serving data to models in production. Highest ceiling, most exposure to a moving field.
model registry: the record of trained models, their versions, and the data they were trained on. Where a content-addressed training snapshot is referenced.
modern data stack: the ELT-plus-cloud-warehouse assembly of the early 2020s. A useful shorthand and a phrase that will date.
module: a reusable Terraform component. The unit that turns copied HCL into something reviewable.
monitoring: observing a running system. For data, freshness, volume, and distributions — not just whether the process exited zero.
multi-stage build: a Dockerfile that builds in one image and copies artifacts into a smaller one. Smaller images, fewer packages in production, faster pulls.
multipart upload: uploading a large object in parts. Necessary above a size limit, and abandoned parts are billed until a lifecycle rule removes them.
MVCC: multi-version concurrency control: readers see a snapshot rather than blocking. Why Postgres readers do not block writers, and why bloat exists.

N

narrow transformation: a Spark operation where each output partition depends on one input partition. No shuffle, and therefore cheap.
natural key: the business identifier for an entity, as opposed to a surrogate key. What a reconciliation joins on, and what survives a rebuild.
net revenue: revenue after refunds are netted. Because refunds settle for up to 90 days, it is not final when the month closes, so the figure needs an as-of date — which is why Chapter 38's is stated as of a specific day, and why it is not printed here.
node selection: dbt's --select syntax for choosing models by name, tag, path, or graph relation. state:modified+ is the one that makes CI fast.
non-additive measure: a measure that cannot be summed across dimensions, such as a ratio or a balance. Summing one is a common and invisible error.
non-determinism: producing a different result from the same inputs. Legacy jobs that depend on scan order cannot be matched, only reconciled against a property.
normalization: removing redundancy by splitting tables. Right in an operational database, usually wrong in a warehouse's serving layer.
nullable dtype: pandas' newer types that represent missing values without promoting to float. What stops an integer column silently becoming a float.
numeric type: Postgres' exact decimal type. Correct for money, slower than integers, and this book prefers integer cents.

O

OAuth: a delegated authorization framework. In ingestion, mostly a token you must refresh and must not log.
object dtype: pandas' catch-all type, usually holding Python objects. Slow, memory-hungry, and what a string column becomes by default.
object storage: S3 and its equivalents: a flat namespace of immutable objects addressed by key. There are no directories; the slashes are part of the name.
observability: being able to answer questions about a system from its outputs. For data, the questions are about the data rather than about the process.
observability tool: a product that monitors data freshness, volume, and schema. Good, and frequently bought to substitute for practice rather than to extend it.
offer negotiation: the conversation at the end. Spend the energy on scope before salary; the cost of a wrong job is a year.
offline store: the feature store's historical half, holding all versions for as-of joins. A Type 2 dimension is exactly this.
offset: a consumer's position in a partition. Committed by the consumer, which is why processing and committing must be ordered carefully.
offset pagination: paginating with LIMIT/OFFSET. Simple, and it skips or repeats rows when the underlying data changes between pages.
OIDC: OpenID Connect. In CI, the mechanism for getting short-lived cloud credentials without storing a secret.
OLTP: online transaction processing: many small reads and writes. The workload row-oriented storage and B-trees are built for.
on-demand pricing: paying per query or per unit of usage with no commitment. Higher unit price, no risk, and the right default until your usage is predictable.
online inference: scoring a model on a live request. One of the three conditions that makes a feature store worth it.
online store: the feature store's serving half: latest value only, key-value, sub-10ms. Priced by writes, which is the number nobody estimates.
onsite loop: the set of interviews after the screen. System design and correctness together are usually more than half the decision.
OpenTelemetry: a vendor-neutral standard for traces, metrics, and logs. Worth adopting for the portability rather than for any single feature.
operator: in Airflow, a unit of work. Prefer a few well-understood ones to the ecosystem's long tail.
optimistic concurrency: attempting a commit and retrying if the table version moved. How table formats let multiple writers proceed without locking.
ORC: a columnar format contemporary with Parquet, dominant in the Hive ecosystem. Comparable technically, and Parquet won on adoption.
orchestration: deciding what runs, when, and in what order, and what happens when something fails.
orchestrator: the tool that does it: Airflow, Dagster, Prefect, or cron until it stops being enough.
ordering guarantee: a promise about the order in which records are seen. Needed per-aggregate, rarely globally, and not at all for a commutative projection.
out-of-core: processing data larger than memory by streaming it. What DuckDB and Polars do and pandas does not.
output port: a data product's documented, versioned interface. Framing eleven ad-hoc extracts as ports collapsed them to three at Kestrel.
ownership: who is accountable for a dataset. A team, able to act, verified recently — not the original author.

P

PACELC: an extension of CAP: else, when there is no partition, a system trades latency against consistency. The more useful formulation, because the "else" branch is where you live.
page: the fixed-size block a database reads and writes. Why a row's width affects scan cost more than intuition suggests.
page token: an opaque cursor identifying a position in a result set. Treat it as opaque even when it obviously encodes something.
pagination: returning results in pages. Cursor-based where correctness matters; offset-based where it does not.
paging: waking a human. Reserve it for something both urgent and actionable, or you have built alert fatigue with consequences.
pandas: the default Python DataFrame library. Ubiquitous, eager, memory-hungry, and correct to use for data that fits comfortably.
parallel run: running old and new systems simultaneously and comparing. The technique the whole of Chapter 37's verification rests on.
Parquet: the standard columnar file format. Compresses well, supports predicate and projection pushdown, and is the default for anything analytical.
partial failure: some components failing while others continue. The defining property of a distributed system and the reason idempotency matters.
partition: in Kafka, an ordered log within a topic and the unit of parallelism and ordering. In storage, a physical division of data.
partition alignment: arranging a model's partitions to match its incremental unit, so a rerun replaces exactly one partition.
partition assignment: mapping partitions to consumers in a group. Changes on rebalance, which is why a consumer can see a replay.
partition by: a window function's grouping clause. Determines the frame's scope, and with ORDER BY determines whether peers exist.
partition key: the value determining which partition a record goes to. Decides ordering guarantees and skew simultaneously.
partition pruning: skipping partitions a predicate cannot match. The highest-value optimization in this book, and a function applied to the partition column defeats it.
partitioning: splitting data by a key so that work and storage divide. The decision that most affects cost and is most often made by default.
periodic snapshot: a fact table with one row per entity per period, whether or not anything happened. Right for balances and inventory levels.
personal data: information relating to an identifiable person. Broader than most engineers assume: an IP address, a device id, and a salted hash all qualify.
physical replication: copying a database at the block level. Simple and exact, and it gives you a replica rather than a change stream.
PII: personally identifiable information. A colloquial term; the regulations say "personal data" and mean something broader.
pivot: turning rows into columns. Verbose in standard SQL, and a common interview question because the general case needs dynamic SQL.
plan: Terraform's preview of what an apply would change. The artifact to review; applying without reading it is the whole risk.
platform engineering: building the capabilities other teams use. The default senior path in data, and the most portable.
platform team: the team whose product is the platform rather than the data. In a mesh transition it does not shrink; it changes to harder work.
point-in-time correctness: using only information available at the moment being predicted. Its absence is data leakage, and the lab measures the cost at 0.830 AUC against 0.591.
Polars: a Rust DataFrame library with a lazy API. Fast, memory-efficient, and it will lose leading zeros on an all-numeric string column exactly as pandas does.
poll loop: a Kafka consumer's poll() cycle. Taking too long between polls gets you evicted from the group, which looks like a mysterious rebalance.
polyglot persistence: using several storage technologies for their strengths. Correct in principle and a way to acquire five operational burdens.
polyseme: a concept meaning subtly different things in different domains. The mesh's answer to shared entities, and it applies to a concept rather than to a fact table.
pool: an Airflow concurrency limit shared across tasks. What stops a backfill from exhausting a database, and what a sensor can deadlock.
portfolio: the accumulated evidence of what you have done. In this field the strongest item is a platform that reconciles.
portfolio project: the artifact you bring to an interview. Worth more than a certification because it is checkable — and only if you can say what went wrong.
pre-authorization: deciding in advance who may take a costly action without asking. What stops an incident waiting forty minutes for permission.
predicate pushdown: evaluating a filter as close to the data as possible, skipping blocks whose statistics rule them out. Why Parquet statistics exist.
prefix: the leading portion of an object key. What a listing filters on, and what partitioning manipulates.
processing time: when a record is processed, as opposed to when the event happened. Easy to observe and almost never what a business question means.
processor: in privacy law, a party processing data on a controller's instruction. Determines obligations; a legal determination.
producer: a process writing to a topic. Its partitioner decides ordering and skew.
producer-side validation: the producer checking its own output against the contract before publishing. The only place a breaking change can be stopped rather than detected.
profiles.yml: dbt's connection configuration. Kept out of the repository, and the file that makes "it works on my machine" possible.
progressive refinement: the medallion's core idea: each layer adds guarantees the previous one did not have.
projection: a view of an event stream computed by folding. Pure function of the events, which is what makes replay meaningful.
projection pushdown: reading only the columns a query needs. Free in a columnar format, and SELECT throws it away.
promotion: moving an artifact from one environment to the next. In ch40, the level change and the evidence it requires.
Protocol Buffers: a compact binary format with generated code and strong schema evolution rules. Excellent for RPC and inter-service events.
provider: a Terraform plugin for a platform. Pin its version; a minor upgrade can change a plan.
pseudonymization: replacing direct identifiers, usually with a hash or a token. Reduces risk and changes no obligation: it is still personal data.
publication: in Postgres logical replication, the set of tables a subscriber receives. Where a missing table becomes a silently missing stream.
purpose limitation: data collected for one purpose may not be freely used for another. Makes a row present for some purposes, which the pipeline must know.

Q

QUALIFY: a clause filtering on a window function's result without a subquery. Available in Snowflake, BigQuery, and DuckDB; missing in Postgres.
quarantine: a parallel table for rows that failed an assertion, with the reason and enough context to replay. Needs a size alert and a retention, or it becomes a second landfill.
quasi-identifier: an attribute that is not identifying alone and is identifying in combination: postcode, birth year, sex. Where re-identification actually comes from.
query plan: the engine's chosen execution strategy. Reading one is the highest return-on-time skill in this book.
quorum: the number of replicas that must agree. Where consistency guarantees are actually configured, usually with a default nobody chose.

R

RANK: a window function ranking with gaps after ties. Distinct from DENSERANK and ROWNUMBER, and the distinction is a common bug.
rate limit: a cap on request frequency. Respect the server's headers rather than guessing, and back off with jitter.
RBAC: role-based access control. Simpler to audit than ABAC, and it accumulates roles until nineteen of forty-seven have one member.
RDD: Spark's original low-level API. Still available, opaque to the optimizer, and rarely the right choice now.
re-identification: recovering an individual's identity from supposedly anonymous data. The attack k-anonymity measures resistance to.
read replica: a copy serving reads. Relieves the primary, and it lags, which is visible to a read-after-write.
read-your-writes: the guarantee that you see your own write immediately. Often absent on a read replica, and the source of "the update didn't save" bug reports.
rebalance: reassigning partitions when a consumer group changes. Stops processing briefly and can replay from a committed offset.
rebuild: recomputing the whole platform from raw. The only check that tests a property of the whole graph over time, and it costs $198.96.
reconciliation: comparing a computed result against an independent source. The single most valuable check in this book, and only as good as the independence of its two sides.
recursive CTE: a CTE that references itself, for hierarchies and graph walks. Powerful, and easy to write without a termination condition.
red flag: a signal that a role or team is not what it claims. "We don't really have incidents" is the clearest.
ref: dbt's function referencing another model. What builds the DAG, and why you never write a table name directly.
referential integrity: every foreign key has a matching parent. Enforced by the database in OLTP and by an assertion in a warehouse.
refund netting: subtracting refunds from the month the order was sold, not the month they settled. The rule that makes a closed month mutable.
relevance scoring: ranking search results by estimated usefulness. BM25 and its relatives, and tuning it is a different discipline from running the index.
repartition: redistributing data across partitions with a full shuffle. Expensive and sometimes exactly what is needed to fix skew.
replay: reprocessing historical records from a log. The capability that makes a streaming platform recoverable, and one you do not have until you exercise it.
replay window: how far back a source lets you re-read. Bounds your backfill regardless of what your pipeline can do.
replication: keeping copies on several nodes. Buys durability and availability, and introduces lag and consistency questions.
replication factor: how many copies of a Kafka partition exist. Three is the usual default, and with min.insync.replicas=2 it survives one broker.
replication lag: how far behind a replica is. Small and non-zero, and the reason a read replica can serve stale data.
replication slot: a Postgres structure tracking a consumer's position in the WAL. Retains WAL until consumed, so a stalled consumer fills the source's disk.
reserved instances: committing to capacity for a discount. Real savings, and a commitment converts a future efficiency into a sunk cost.
REST API: the most common ingestion source after a database. Pagination, rate limits, retries, and a schema that changes without notice.
restartability: a load's ability to resume after failure without duplicating or losing. Achieved with a manifest and idempotent writes.
result cache: a warehouse returning a previous identical query's result without computing. Cheap, and it makes naive benchmarks meaningless.
retention: how long data is kept. Needs a period, a mechanism, and an exception path — and the mechanism is what is missing.
retroactive change: a source altering historical records in place. Breaks incremental loads and reproducibility, and is discovered during a rebuild.
retryable error: a failure worth attempting again: a timeout, a 429, a 503. Distinguishing it from a permanent one is most of a robust client.
revenue recognition: the accounting rules deciding when a sale becomes revenue. Gift cards are a liability at sale, and an engineer deciding this alone is Chapter 38's third failure.
reverse ETL: pushing warehouse data back into operational systems. Inverts your risk profile: a dashboard bug is embarrassing, a reverse-ETL bug is in front of customers.
reverse interview: your evaluation of them. The neglected half, and questions whose answers you have not thought about are questions asked to look good.
reversible decision: one that can be undone cheaply. Make it fast; spend the deliberation on the irreversible ones.
right to be forgotten: the erasure right. Requires finding a person in every identifier space before deleting anything.
role explosion: accumulating roles until they describe individuals rather than job functions. At that point they provide no abstraction and all the overhead.
role-playing dimension: one dimension joined several times in different roles, such as order date and ship date. Implemented with views or aliases.
rollback: reverting to a previous state. In data frequently impossible, which is why forward fixes and rehearsed rollbacks both matter.
rotation: the on-call schedule. Its health is measured in pages per week, not in coverage.
routing: sending an alert to the right person. The difference between an alert and a notification nobody owns.
row group: a horizontal slice of a Parquet file, the unit statistics describe and readers skip.
row-binary format: a row-oriented binary encoding such as Avro. Efficient for whole-record reads and wrong for column scans.
row-level security: a policy filtering which rows a role can see, attached to the object. Auditable in one place, and it fails closed.
row-oriented storage: storing all of a row's values together. Right for OLTP, wrong for analytics.
ROW_NUMBER: a window function numbering rows without ties. Needs a deterministic ORDER BY, or it returns a different row each run and no uniqueness test notices.
ROWS versus RANGE: window frame modes. They differ when the ordering column has duplicates; RANGE includes every peer and is the default.
run-length encoding: storing a value and a count instead of repetitions. Extremely effective on sorted low-cardinality columns.
runbook: the document telling an on-call responder what to do. Judged by whether somebody opens it at 3am, not by whether it exists.

S

salting: adding a random component to a skewed join key so the hot key spreads across partitions. The manual fix AQE now handles automatically in many cases.
sampling: testing against a subset of production-shaped data. Makes CI fast enough to run on every change, and it must preserve the awkward rows.
savings plan: a spend commitment for a discount. Commit to the floor, not the current level.
scan cost: what it costs to read data for a query. On a per-byte meter it is the cost, which is why format and layout dominate.
SCD Type 1: overwriting a dimension attribute on change. Simple, and it destroys history.
SCD Type 2: adding a new row with validity dates on change. Preserves history, and it is also a point-in-time-correct feature table.
scheduler: the process deciding what runs when. Its own failure produces no events, which is why it needs a heartbeat rather than an alert.
schema change event: a CDC record signalling a DDL change. Consumed rather than ignored, or the next data record surprises you.
schema drift: a source's schema changing without notice. The most common ingestion failure and the one contracts exist to catch.
schema enforcement: a table format rejecting writes that violate the declared schema. What makes a lake behave like a warehouse.
schema evolution: changing a schema without rewriting data. Adding a column is free in Parquet; renaming one needs column mapping.
schema registry: a service holding schemas and enforcing compatibility rules. Where BACKWARD and FORWARD compatibility are actually configured.
schema-on-read: deferring schema interpretation to query time. Flexible, and it moves every type surprise to the consumer.
schema-on-write: enforcing a schema at load time. Rejects bad data early, at the cost of rejecting data you might have wanted to keep.
scope: the breadth of what you are responsible for. What sets level, and what to negotiate before salary.
search index: an inverted index supporting text queries. A different data structure and a different operational burden from a database.
secondary index: an index on a non-primary attribute. Cheap in a relational database, and a significant design decision in a distributed store.
secret: a credential. Never in a repository, never in a log, and preferably short-lived via OIDC.
secret manager: the service holding them. Referenced by identity rather than by another secret, or you have moved the problem.
seed: a CSV checked into a dbt project and loaded as a table. The right home for a small, version-controlled mapping — and for the hand-made lookup table that broke a rebuild.
self-serve data platform: capabilities that let a domain team ship without a platform engineer. More work than doing the work yourself, for a long time.
semantic layer: where a metric is defined once and consumed everywhere. The structural answer to four definitions of activecustomer.
semantic versioning: major.minor.patch, where major signals a breaking change. Useful for a data contract precisely because it forces you to classify the change.
semi-additive measure: summable across some dimensions and not others, such as an account balance across time. The category most often summed incorrectly.
sensitivity tier: a classification level: public, internal, confidential. Decided by legal, propagated directionally, and enforced by the platform.
sensor: an Airflow task that waits for a condition. Holds a worker slot unless deferrable, which is how a pool deadlocks.
sequence number: a monotonic per-aggregate version on an event. What makes ordering checkable, and what lets a consumer discard a duplicate in one comparison.
sequential scan: reading a whole table. Frequently faster than an index scan for a non-selective predicate, which is why the planner chooses it.
serialization: converting data to bytes for storage or transport. The format decision that determines compression, splittability, and schema evolution.
serve: the lifecycle stage where data reaches a consumer. The entire user interface of your work, and systematically under-invested in.
service level agreement: a promise about a dataset's timeliness or quality, with consequences. Distinct from an SLO, which is internal.
session window: a window defined by a gap in activity rather than by a clock. Requires state per key and is neither commutative nor last-write-wins.
sessionization: grouping events into sessions by an inactivity gap. A gaps-and-islands problem, and Kestrel's is the single largest job in the platform.
set-based thinking: expressing a transformation as an operation on sets rather than a loop. The mental shift that makes SQL productive.
SettingWithCopyWarning: pandas warning that an assignment may not affect the original object. Almost always a real bug, and almost always ignored.
severity: how badly a failed assertion matters: blocking, warning, or note. The classification that keeps a blocking set small enough that nobody disables it.
shadow deploy: running new code alongside old without acting on its output. The data equivalent of a canary, and the only one that compares values.
shadow running: running a legacy and a replacement system in parallel and reconciling. Where 687 differences got classified, 196 of them the legacy system's fault.
shadow schema: building a contract's proposed change into a parallel schema to test consumers against. Expand-contract's mechanism.
sharding: splitting data across independent database instances. Scales writes, and every cross-shard query becomes your problem.
showback: reporting each team's spend without moving budget. The step before chargeback, and usually the one that does the work.
shuffle: redistributing data across the network so that related records meet. The expensive operation, and the one every join and aggregation may require.
signal: evidence an interviewer is looking for. "Have you operated something" is the strongest, and it shows up in what you mention breaking.
silver layer: typed, deduplicated, conformed — and encoding no decision anyone could disagree with. The test: if two competent people could disagree, it is a business rule.
singular test: a dbt test written as a SQL query returning failing rows. The escape hatch for anything a generic test cannot express.
skew: uneven distribution of work across partitions. High cardinality does not prevent it, and one key at 8.1% turned a 22-minute job into 71.
SLA: the promise. Kestrel's is dailyrevenue by 06:00 America/NewYork.
SLA miss: a run finishing later than its agreed deadline. Worth alerting on slack remaining rather than on the miss.
SLI: a service level indicator: the measured quantity an SLO is about. Choose one a consumer would recognize.
sliding window: overlapping windows advancing by less than their length. More output than tumbling, and each event lands in several windows.
slim CI: building only modified models and their descendants, using deferral and state comparison. The difference between a CI that runs and one that is disabled.
SLO: a service level objective: a target for an SLI, with an error budget. Internal, and the number an on-call rotation is designed around.
slot: BigQuery's unit of compute. In Postgres, a replication slot — an unrelated and easily confused term.
slowly changing dimension: a dimension whose attributes change over time. Types 1 through 6, of which 1 and 2 cover almost everything.
small file problem: many tiny files inflating request costs and task overhead. Real, and its cost justification is usually much weaker than its latency one.
small files problem: the same thing, at the storage layer.
snappy: a fast, splittable compression codec. Parquet's usual default, and the right choice unless you have measured otherwise.
snapshot: a consistent point-in-time view. In dbt, the mechanism that builds SCD Type 2 history from a mutable source.
snowflake schema: a star schema with normalized dimensions. Saves storage that costs nothing and adds joins that cost something.
soft delete: flagging a row as deleted rather than removing it. Visible to an incremental load, which is why it is kinder to downstream systems.
sort key: the attribute rows are ordered by within a partition. Determines which range queries are cheap.
sort order: the physical ordering of data in a file or table. Determines compression ratio and which predicates can skip.
sort-merge join: sorting both sides and merging. Spark's fallback when a broadcast is not possible, and the reason a shuffle appears in the plan.
source: in dbt, a declared raw table with freshness expectations. Where the lineage graph starts.
source fidelity: bronze's guarantee: exactly what the source sent, unmodified. What makes "it arrived that way" a provable statement rather than a claim.
source freshness: dbt's check that a source table is no older than a threshold. The assertion that catches an upstream problem before your models run on stale data.
source of truth: the system whose value is authoritative. Naming it explicitly is what stops two systems both being right.
source system: where data originates, usually outside your control. The stage with the most leverage and the least authority.
specialization: a track: streaming, platform, analytics engineering, ML infrastructure, governance. Each trades something, and the two columns nobody reads are on-call and obsolescence.
spill: writing intermediate data to disk when memory is exhausted. Slow, and a sign that a partition is too large or a join too wide.
splittable: a file a reader can start consuming from the middle of. gzip is not; snappy in Parquet is, which is why a single large gzip file becomes a single task.
spot instances: heavily discounted capacity that can be reclaimed with short notice. Excellent for a re-runnable batch job, unusable for a deadline that cannot be missed twice.
SQL round: the interview testing whether you notice the awkward rows, not whether you know window functions.
staff engineer: the level whose scope is the platform, whose output is writing and conversations, and whose best work is often invisible.
stage: a set of Spark tasks executable without a shuffle. Stage boundaries are shuffle boundaries.
staging: a pre-production environment. In data the hard part is that it cannot hold production data, which is Chapter 31's largest exposure.
staging model: a dbt model doing one-to-one cleanup of a source: renaming, casting, deduplicating. Silver, in dbt's vocabulary.
STAR: situation, task, action, result. A behavioral-answer structure that survives follow-ups if you add a number.
star schema: a fact table surrounded by denormalized dimensions. The default warehouse shape, and correct far more often than its critics allow.
state comparison: comparing an artifact against a previous run to find what changed. What makes slim CI possible.
state file: Terraform's record of what it manages. Remote, locked, and separated by blast radius.
state store: a stream processor's durable per-key state. Grows without a TTL, and its size is the thing that fails at 3am.
statement timeout: a limit on query duration. The cheapest protection against one query consuming a database.
StatsD: a simple metrics protocol. Old, widely supported, and adequate for pipeline counters.
steward: a person accountable for a dataset's meaning, as distinct from its pipeline. Useful when they can act; a name on a page when they cannot.
storage class: S3's tiers by access frequency. Cheaper storage and more expensive retrieval, and a recommendation that omits retrieval cost is worse than none.
storage-compute separation: scaling storage and compute independently. The property that makes cloud warehouses elastic and makes idle compute a pure waste.
store: the lifecycle stage where data rests. Where format, layout, and retention decisions are made, mostly by default.
stored procedure: logic held in the database. Readable and entangled with transaction management, error handling, and business rules nobody knew were there.
strangler fig: growing a new system around an old one until the old one does nothing. The default migration pattern, and step six — actually deleting it — is the one that does not happen.
stream processing: computing continuously over unbounded data. Necessary less often than the literature implies, and the four questions in Chapter 29 decide.
streaming: see stream processing. Also the specialization with the deepest technical content and the worst on-call, which are the same fact.
streaming execution: processing data in chunks that fit memory. What lets DuckDB and Polars handle files larger than RAM.
structured logging: emitting logs as key-value records rather than prose. Queryable, and the right place to refuse to serialize a field tagged as personal data.
surrogate key: a meaningless integer key for a dimension row. Stable across source changes, and it must not leak into a reconciliation.
synthetic data: generated data standing in for production. Safe, and only as good as the awkward cases you thought to generate.
system design interview: the main event. Requirements are 25% of the score and the diagram is 10%, and most candidates invert that.

T

table format: Iceberg, Delta, or Hudi: metadata over files giving atomicity, schema evolution, time travel, and row-level deletes.
tagging: labelling resources with team and purpose. The mechanism cost attribution depends on, and it fixes the future and nothing else.
take-home: an interview exercise done on your own time. The round with the widest quality range, and the most expensive one for you.
target: a dbt connection profile: which warehouse, which schema, which credentials.
target leakage: a feature containing information about the label that would not exist at prediction time. Data leakage's most common form.
task: the smallest unit of Spark work, one partition of one stage. In Airflow, one node of a DAG.
task group: an Airflow construct for organizing tasks visually and logically. Cosmetic and genuinely helpful at scale.
TaskFlow API: Airflow's decorator-based authoring style. Less boilerplate, and ast.walk will descend into nested @task functions if you write a linter.
technical currency: being up to date. Achieved by learning one tool per category deeply, not five shallowly.
technical debt: the accumulated cost of past shortcuts. In data it is usually a rule nobody wrote down rather than code nobody refactored.
technical metadata: types, sizes, partitions, lineage. Cheap to crawl, and it is not what makes a catalog useful.
technical screen: the first technical round, usually SQL. The widest cut after the résumé.
telemetry: the metrics, logs, and traces a system emits. And a place personal data escapes a residency boundary.
terminal error: a failure that will not succeed on retry: a 400, a 404, an auth failure. Retrying it wastes quota and hides the problem.
Terraform: the dominant infrastructure-as-code tool. Declarative, with a plan you can review, and a state file you must protect.
testing the data: asserting properties of the output. What catches a pipeline that succeeded and produced wrong numbers.
testing the pipeline: asserting that the code behaves. Necessary, insufficient, and what most teams have.
third normal form: a normalization level removing transitive dependencies. Right for an operational schema; a source of joins in a warehouse.
threads: dbt's parallelism setting. Bounded by the warehouse's concurrency, not by your machine.
threshold: the value at which an assertion fails. Absolute thresholds miss relative regressions, which is how a 3.75x slowdown passes an 8-hour alarm.
throughput: volume processed per unit time. Trades against latency, and both are usually quoted without the other.
thundering herd: many clients retrying simultaneously and overwhelming a recovering service. Prevented by jitter, not by backoff alone.
time travel: querying a table as of a past version. A table format feature, a debugging superpower, and a privacy problem.
time-series database: a store optimized for timestamped metrics. Excellent within its model, and adding a high-cardinality label produces billions of series.
timeline: the ordered record of an incident. The most valuable postmortem artifact and the one hardest to reconstruct afterwards, so write it during.
timeliness: whether data arrived when expected. Freshness, expressed as a quality dimension.
timestamp strategy: an incremental check based on updatedat. The most common, and it misses hard deletes and retroactive changes.
timestamp with time zone: Postgres' timestamptz. Stores an instant; the session timezone decides how a boundary comparison behaves, which is Chapter 38's first failed reconciliation.
toil: manual, repetitive operational work that scales with the system. The thing an SRE practice exists to bound, and above about half it is why people leave.
token bucket: a rate-limiting algorithm allowing bursts up to a capacity. What most APIs implement, so matching it client-side is more effective than a fixed delay.
token refresh: exchanging a refresh token for a new access token. A step that must be thread-safe and must not be logged.
tokenization: replacing a value with a random token, with the mapping in a vault. Everything a hash gives you plus reversibility, and a vault on the critical path.
tombstone: a marker recording a deletion. In Kafka, a null-valued record that removes a key under compaction.
topic: a named Kafka stream, divided into partitions.
total cost of ownership: licence plus infrastructure plus the engineering time to operate it. The third term dominates and is the one omitted from comparisons.
trace: a record of a request's path through a system. Rarer in data than in services, and useful across a multi-hop ingestion path.
trailing median: a baseline computed from recent history rather than a fixed value. What makes a volume band adapt to growth without adapting to a defect.
training/serving skew: the same feature computed differently in two code paths. 5.37% overall at Kestrel, and 98% among the customers the model was most confident about.
transaction boundary: the grouping of CDC events belonging to one source transaction. Preserving it is what stops a consumer seeing half an update.
transaction isolation: how concurrent transactions see one another. The setting whose default you have not read and whose anomalies you have blamed on something else.
transaction log: the ordered record of changes to a table. Delta's deltalog, Postgres' WAL, and the thing CDC reads.
transactional outbox: writing the row and the event in one database transaction, with a relay publishing from the outbox. Trades loss for duplication, which is the right trade.
transactional sink: a destination supporting atomic commits, so a stream can write exactly-once within its boundary.
transform: the lifecycle stage where data is shaped for use. Where the business's rules live and where they get decided by accident.
transformation framework: dbt or an equivalent. Brings version control, tests, and lineage to SQL, which is most of what a platform needs.
transitive compatibility: compatibility across all previous versions, not just the last one. The strictest schema-registry mode.
trigger: what causes a DAG run: a schedule, a dataset update, or a manual invocation.
TTL: time to live: automatic expiry. The mechanism that makes a state store or a cache bounded.
tumbling window: fixed, non-overlapping windows. The simplest windowing and usually the right one.

U

undercurrent: a concern spanning every lifecycle stage: security, data management, DataOps, architecture, orchestration, and software engineering.
undocumented dependency: an input a job needs that is not named in any configuration. Found by running the job, and 36 of them across Kestrel's twelve legacy jobs.
uniqueness: no duplicates on a key. The assertion that catches fan-out, and the one that passes while a non-deterministic dedup returns a different row each run.
unit economics: cost per business unit. $0.1526 per order, 0.201% of GMV — and a total answers no question anyone has.
unit test: a test of one transformation with fixed inputs and expected outputs. dbt's version arrived late and is the right tool for a gnarly CASE expression.
unknown member: a placeholder dimension row for facts whose dimension has not arrived. Keeps the join inner and the counts right.
unpivot: turning columns into rows. Usually easier than pivoting and equally verbose.
unplanned work: engineering time consumed by interrupts. Above about half it is why people leave, and it is the resting state without deliberate effort.
upcasting: converting old event versions to the current shape at read time, in one place. Where schema evolution actually lives in an event-sourced system.
updated_at: the column most incremental loads depend on, and which lies in four distinct ways: not set on every path, set by the application clock, unchanged by bulk updates, and absent on deletes.
upsert: insert or update. See merge.

V

vacuum: Postgres' process reclaiming dead tuples. Autovacuum handles most cases, and a long-running transaction defeats it.
valid_from: the start of an SCD Type 2 row's validity. Half of what makes an as-of join possible.
valid_to: the end. Null or a sentinel for the current row, and the choice matters for range predicates.
validity: whether a value conforms to its expected format or domain. The easiest quality dimension to assert and the least likely to be the actual problem.
vector database: a store for embeddings with approximate nearest-neighbour search. A real capability, and frequently a feature of a database you already run.
vectorization: operating on batches of values rather than one at a time. The reason a columnar engine is an order of magnitude faster than a row-at-a-time loop.
vendor lock-in: the cost of leaving. Real, usually overstated for storage and understated for proprietary transformation logic.
verification: confirming a migration produces the same result. A parallel run and a reconciliation, and its absence is why "we migrated it" is a claim.
virtual warehouse: Snowflake's compute cluster. Sized independently of storage, billed by the second while running, and the thing to auto-suspend.
volume: how much data arrived. Monitored against a band, and a band set from ordinary days fires every Black Friday.
volume floor: a minimum row count below which a load is treated as failed. Catches a truncated source, which a percentage band can miss.

W

watermark: an assertion that no more events older than a timestamp will arrive. The mechanism that lets a stream close a window, and always wrong sometimes.
wide transformation: a Spark operation requiring a shuffle. The expensive kind.
wide-column store: Cassandra and its relatives: rows with dynamic columns, partitioned by key. Excellent write throughput, and the query patterns must be known in advance.
window function: a function computing over a set of rows related to the current one, without collapsing them. The single most valuable SQL feature for data engineering.
windowing: grouping unbounded stream data into finite chunks. Tumbling, sliding, or session, and the choice follows the question.
worker: the process executing an Airflow task. Its slots are the resource a sensor can hold.
workload identity: a cloud identity attached to a running workload rather than to a stored key. Removes the long-lived credential entirely.
workspace: a Terraform environment sharing configuration with separate state.
write-ahead log: the durability mechanism of a database, and the source CDC reads. Postgres' WAL, MySQL's binlog.

X

XCom: Airflow's mechanism for passing small values between tasks. Small: it is stored in the metadata database, and it is not a data transport.

Z

Z-ordering: a multi-dimensional clustering technique co-locating rows similar across several columns. Improves skipping when queries filter on more than one dimension.
zero-copy: passing data between systems without serializing or copying. Arrow's central promise, and what makes engine interop cheap.
zone map: per-block min/max statistics used to skip data. Parquet's row-group statistics are the same idea under a different name.
zstd: a modern compression codec with better ratios than snappy at comparable speed. Usually the right choice today, and check that every reader in your stack supports it.