Exercises: What Is Data Engineering?

DataField.Dev

Exercises: What Is Data Engineering?

This is an orientation chapter, so most of these are about judgment and arithmetic rather than code. Two of them are code, and one of those you should run before you have installed anything at all.

Difficulty: ⭐ warm-up · ⭐⭐ standard · ⭐⭐⭐ deeper. Solutions: worked answers to the daggered (†) and odd-numbered problems are in appendices/answers-to-selected.md. Try each problem before you look — several of these answers only land once you have been wrong in the specific way the problem invites.

Part A — Warm-ups ⭐

1.1 † In one sentence, state the difference between a software service failing and a data pipeline failing, and say why the second is more expensive to detect.

1.2 Name the six artifacts a data engineer produces (§1.2). For each, name one person or role who consumes it.

1.3 † Kestrel's FY2025 gross merchandise value is $182.0M across 2,400,000 orders. Compute the average order value to the cent. Then compute average orders per day. Show your working.

1.4 The clickstream is 14,000,000 events a day. Compute the average events per second. The peak is 2,900 per second — what is the peak-to-average ratio? Compare it to the peak-to-average ratio for orders (41,300 on Black Friday against a 6,575 daily average) and say, in one sentence, why browsing spikes harder than buying.

1.5 † Define idempotency in one sentence without using the word "idempotent." Then give an example of a shell command that is idempotent and one that is not.

1.6 The chapter says money is stored in integer cents throughout. Kestrel's average order value is $75.83. What integer would represent that, and what is the name of the column type convention being used?

1.7 † In §1.4 the roles are distinguished by "who finds out why the number is wrong." For each of these, say which role owns it: (a) The nightly load failed because the source database rotated a credential. (b) Revenue is 8% high because someone added a promotions join that fans out rows. (c) The churn model's precision dropped from 0.71 to 0.44 after a good deploy. (d) The BI tool is returning 502s.

Part B — Standard ⭐⭐

1.8 Reproduce the Spark job arithmetic from §1.6 in Python, from the inputs rather than the printed results:

RATE_PER_NODE_HOUR = 2.400   # frozen basis, all-in

def nightly_cost(executors: int, hours: float) -> float:
    """Cost of one nightly run. Returns dollars."""
    ...

Compute the before cost (160 executors, 10.0 hours), the after cost (24 executors, 1.3 hours), the annual figures, the annual saving, and the reduction factor. Compare each against the chapter.

Then answer: the chapter says the reduction is 51.3×. Compute it to three decimal places. Is "51.3×" a fair rounding, and what would you have written if it came out to 51.49?

1.9 † A colleague proposes storing raw clickstream as gzipped JSON instead of Parquet, arguing that gzip gets similar compression and JSON is easier to debug. Using the figures in §1.5 and the 💸 Cost Check callout, write a three-paragraph response. Your response must (a) concede the point where it is right, (b) identify the cost the colleague has not considered, and (c) state what measurement would settle it.

1.10 Rewrite this backfill to be idempotent. State which of the four idempotency strategies named in §1.6 you used and why the other three are worse here.

INSERT INTO fct_session (session_id, customer_id, session_date, events, revenue_cents)
SELECT session_id, customer_id, session_date, COUNT(*), SUM(revenue_cents)
  FROM bronze.events
 WHERE session_date BETWEEN '2025-03-01' AND '2025-03-07'
 GROUP BY session_id, customer_id, session_date;

1.11 † Kestrel's peak-to-average ratio for orders is 6.28×. Suppose you size the order-processing pipeline for the average and it degrades linearly above capacity. On Black Friday, how far behind would a job that normally takes 22 minutes fall? State your assumptions explicitly — this is a question about assumptions, not arithmetic, and a good answer names at least three.

1.12 The chapter claims "the hard part of data engineering is usually the definition, and the code is the easy part that follows it." Write the definition — not the code — for Kestrel's net_revenue_cents. Your definition must resolve, in writing: cancelled orders, refunded orders, partial returns, shipping charges, tax, and promotional discounts. Aim for under 150 words. Notice how hard this is.

1.13 † Read the two Spark physical plans in the 🔎 Read the Plan callout in §1.6. Without looking at the surrounding prose, write down the single line that distinguishes them and explain in two sentences what it means.

Part C — Deeper ⭐⭐⭐

1.14 The duplicate-rows incident went undetected for thirty-one days. Design a check that would have caught it on day one. Your answer must specify: what is measured, what it is compared against, the threshold, where the check runs, and who is notified. Then — the harder half — name one way your check could produce a false positive, and what you would do about it.

1.15 † The chapter argues against "distributed by default" (📐 in §1.3). Construct the strongest possible case for the opposite position, in three paragraphs, without strawmanning. Then say which of Kestrel's specific properties makes that case fail here, and what would have to change about Kestrel for it to succeed.

1.16 Estimate the total annual cloud cost of storing Kestrel's clickstream for seven years under three retention policies, using the frozen rates in §1.8: (a) All seven years in S3 Standard as Parquet. (b) Current year in Standard, prior years in Standard-IA ($0.0125/GB-month). (c) Current year in Standard, years 2–3 in Standard-IA, years 4–7 in Glacier Instant Retrieval ($0.004/GB-month). Assume 341 GB per year and no growth. Then state the non-cost consideration that should decide this, and why it is not the money.

1.17 † Chapter 1 says a data engineer is accountable for "the data matching the source." Give a concrete example where that standard is not sufficient — where the data matches the source perfectly and the number is still wrong — and say which role owns it.

Part D — The Kestrel Platform ⭐⭐⭐

1.18 — Increment 1: the charter.

Create the repository skeleton and write the project charter. This is referenced directly in Chapter 38.

mkdir -p platform/{seed,ingest,transform,orchestrate,quality,docs}
touch platform/CHARTER.md

Write platform/CHARTER.md with five sections:

Purpose — three sentences. Name the consumers by role, not by name.
Acceptance criterion — the reconciliation statement from §1.7, in your own words. Be precise about which revenue measure and which time grain.
Service level — the 06:00 America/New_York freshness deadline. Write down what "fresh" means operationally: fresh as of when, measured how, from which clock.
Non-goals — at least three things you are explicitly not building, each with one sentence of justification.
Constraints — the four-engineer team, and one sentence on how you will use that constraint to decide between two otherwise reasonable designs.

Then the part most people skip: commit it to git. git init, git add, git commit -m "charter". Every artifact in this book lives in version control from the first one, because Chapter 27 is going to ask you to test and deploy this repository and a project that starts outside git rarely moves inside it.

1.19 † Add a sixth section to the charter: Open questions. List at least five things you do not yet know how to decide. Examples of the right shape: "Do we need CDC or is a nightly batch extract sufficient?" "Where does the boundary between silver and gold sit?" "Who owns the definition of an active customer?"

Date the list. You will answer these across Parts II through VII, and rereading your day-one questions in Chapter 38 is one of the more instructive twenty minutes in this book.

Reflection

Two questions with no answer key.

A. The chapter claims trust is the actual product of data engineering, and that it is harder to rebuild than a table. Think of a system — at work, at school, anywhere — that you stopped trusting. What would it have taken to restore your trust? Was it more information, more time, or a structural change? Now apply that answer to a data platform.

B. You chose a learning path in §1. Write down, in one sentence, what you are giving up by choosing it, and one condition that would make you switch. Revisit this in Chapter 20.