Self-Assessment Quiz: The Data Engineering Lifecycle

Twenty questions. Aim for 16 or more. Work the whole quiz before reading the key.

Question 1

The five stages of the data engineering lifecycle are:

• A. Extract, transform, load, validate, publish
• B. Generate, ingest, store, transform, serve
• C. Collect, clean, model, test, deploy
• D. Source, stage, warehouse, mart, report

Question 2

Store is drawn beneath the other stages rather than between two of them because:

• A. Storage is the most expensive component
• B. Every other stage reads from and writes to storage
• C. Storage decisions are made last
• D. It emphasizes that storage is optional in streaming architectures

Question 3

Which is NOT one of the six undercurrents?

• A. Security
• B. Orchestration
• C. Visualization
• D. Software engineering

Question 4

The defining property of the Generate stage is that:

• A. It produces the highest data volume
• B. You do not control it — source systems exist for other purposes
• C. It is where schemas are defined
• D. It is always append-only

Question 5

Which source update pattern breaks naive incremental ingestion most completely?

• A. Append-only inserts
• B. Updates in place with a reliable updated_at
• C. Hard deletes
• D. Bulk loads

Question 6

The sixth question to ask about a source system — the one the chapter says nobody asks — is:

• A. What load can it take?
• B. Who can change the schema?
• C. What does the data actually mean?
• D. Is there a reliable change timestamp?

Question 7

In the discount_cents story, the bug was invisible to pipeline tests because:

• A. The pipeline had no tests
• B. The values were valid integers in a plausible range; only the meaning had changed
• C. The column was nullable
• D. The failure only occurred at peak load

Question 8

The watermark pattern WHERE updated_at > :last_watermark loses rows when:

• A. The source database is under heavy load
• B. updated_at is assigned at transaction start but the row becomes visible at commit
• C. The extract runs more than once per day
• D. The column is indexed

Question 9

Which combination does the book recommend for delivery guarantees?

• A. Exactly-once end to end
• B. At-most-once, with monitoring to detect loss
• C. At-least-once delivery plus idempotent writes
• D. Whichever the transport layer provides by default

Question 10

Why is at-least-once plus idempotent writes preferred?

• A. It is cheaper in network bandwidth
• B. It moves the hard problem from the transport layer to the write layer, where a DELETE solves it
• C. It is the only option Kafka supports
• D. It eliminates the need for monitoring

Question 11

Which storage decision does the chapter identify as most often made by default and paid for on every read?

• A. The storage system
• B. The file format
• C. The layout — partitioning, file size, sort order
• D. The retention policy

Question 12

Committing clickstream files every 30 seconds across 12 partitions produces roughly how many files per year?

• A. 34,560
• B. 420,000
• C. 12.6 million
• D. 150 million

Question 13

The five layers of transformation named in §2.5 are structural, cleaning, conforming, modeling, and:

• A. Validating
• B. Aggregating
• C. Publishing
• D. Indexing

Question 14

Stitching an anonymous clickstream session identifier to a known customer_id is an example of:

• A. Structural transformation
• B. Cleaning
• C. Conforming
• D. Aggregating

Question 15

A LEFT JOIN to a promotions table doubles SUM(net_revenue_cents) for some orders. What went wrong?

• A. The join key was null
• B. The join changed the grain of the result set
• C. The revenue column was the wrong type
• D. The promotions table had duplicate primary keys

Question 16

Which is NOT one of the three defenses against a grain-changing join?

• A. Declare the grain of every table in its documentation
• B. Test grain with a uniqueness assertion on the declared key
• C. Assert row counts across the join
• D. Use SELECT DISTINCT on every join result

Question 17

Reverse ETL is:

• A. Reversing a failed transformation
• B. Pushing warehouse data back into operational systems
• C. Loading before transforming
• D. Reading a warehouse from a transactional application

Question 18

Drawing the Transform/Serve boundary too far downstream produces:

• A. A combinatorial explosion of pre-aggregated tables
• B. Definitions of business metrics scattered across many BI queries
• C. Excessive storage cost
• D. Slower ingestion

Question 19

Which undercurrent fails against a statutory deadline once personal data is involved?

• A. Security
• B. Data management
• C. DataOps
• D. Data architecture

Question 20

The diagnostic habit this chapter exists to build is:

• A. Always check the orchestrator's logs first
• B. Locate the lifecycle stage before choosing a tool
• C. Reproduce the failure locally before investigating
• D. Escalate to the source system owner immediately

Answer Key

1. B — §2.1. The stages are stable across every tool generation, which is the whole reason to learn them.

2. B — §2.1, §2.4. Store is a substrate. Modeling it as a waypoint produces architectures that treat storage as something data passes through rather than the medium every stage uses.

3. C — §2.7. The six are security, data management, DataOps, data architecture, orchestration, and software engineering. Visualization is a serving concern, not an undercurrent.

4. B — §2.2. You are a guest in someone else's system, and that system owes analytics nothing.

5. C — §2.2. A deleted row leaves no trace; "absent" is indistinguishable from "not in this batch." This is the case CDC (Chapter 14) exists for.

6. C — §2.2. Meaning is answerable only by asking the owning team, and the answers are rarely written anywhere. Chapter 17's data contracts are the mechanism that scales it.

7. B — §2.2. Two services wrote opposite sign conventions to one column. No structural test can see that.

8. B — §2.3. The row commits behind the watermark and becomes permanently invisible. Estimated a few hundred rows a night at Kestrel — too small for a row-count check, too large for a reconciliation to the cent.

9. C — §2.3.

10. B — §2.3. Distributed exactly-once is expensive and usually a lie about the whole system. Idempotent writes make duplicate delivery harmless.

11. C — §2.4. Layout is a query optimization made in advance. Chapter 1's $3,840 job was a layout decision meeting a query that ignored it.

12. C — §2.4. 86,400 ÷ 30 × 12 = 34,560/day × 365 ≈ 12.6 million/year, averaging 27 KB.

13. B — §2.5.

14. C — §2.5. Conforming is making sources agree, and identity stitching is where the hardest bugs in this class live.

15. B — §2.5. Valid SQL, correct values, wrong grain. Nothing in the type system notices.

16. D — §2.5. SELECT DISTINCT masks the symptom, does not fix the grain, and will silently collapse legitimately distinct rows. It is the tempting wrong answer.

17. B — §2.6. And it inverts your risk profile: a dashboard bug is embarrassing, a reverse-ETL bug is in front of customers.

18. B — §2.6. Kestrel before its first data hire (Case Study 1 of Chapter 1) is exactly this failure.

19. B — §2.7, 🔐 callout. GDPR gives a controller one month to respond to an erasure request, extendable by two months for complex cases. Five undercurrents fail gradually; this one fails on a clock.

20. B — §2.1, §2.9. Four hypotheses, four classes of answer, and the wrong guess costs a day.

Topic map