Chapter 40: The Data Engineering Career

"I spent four years getting good at a tool that no longer exists, and the odd thing is that the four years were not wasted — but not for any of the reasons I thought at the time."

Overview

This is the last chapter, and it is the one most likely to be wrong, because career advice ages badly, generalizes poorly, and is usually unfalsifiable.

So it does what it can to be checkable. code/career_map.py puts numbers and weights on four things that are normally left to vibes: what each level means, what each specialization costs you, whether a team is healthy, and which of your skills is depreciating. Every weight is a judgment, and the point of writing them down is that a judgment you can see is one you can argue with.

Four claims the chapter makes, none of them comfortable:

Levels are about the question you are trusted to answer, not years and not a skill list — and the mid-to-senior transition is not technical (§40.3).

Every specialization trades something, and the two columns people do not look at are on-call burden and obsolescence risk (§40.5).

Eighteen of the twenty signals of a healthy data team are observable from outside, in an interview, by asking (§40.9). Which means Chapter 39's reverse interview is a career tool rather than a job-search tool.

And the durable skills are properties and the perishable ones are products (§40.11). That single distinction tells you where to spend a Saturday, and it is why four years on a dead tool are not necessarily wasted.

40.1 What the Job Actually Is

Before levels: the job has changed shape three times in fifteen years and the core has not moved.

~2010   write MapReduce; run a cluster; the job is DISTRIBUTED SYSTEMS
~2016   write Spark and Hive; run a cluster; the job is BIG DATA
~2021   write SQL and YAML; the cloud runs the cluster; the job is MODELLING
~2026   all of the above, plus governance, privacy, cost, and ML serving

What has been constant across all four: somebody has to be able to say what a number means and whether it is right.

That is the sentence the whole book has been building toward, and it is the reason the field is durable even though its tools are not. The infrastructure keeps getting easier and the custody does not.

40.2 The Levels

--levels states each one as the question you are trusted to answer, because years predict almost nothing and a skill list predicts a little:

JUNIOR      How do I build this?
            scope: the task            judged on: a code review

MID         Is this built correctly?
            scope: the component       judged on: a design review

SENIOR      Should this be built?
            scope: the system          judged on: the outcome

STAFF       What should we be building?
            scope: the platform        judged on: other teams' outcomes

PRINCIPAL   What will we need in two years?
            scope: the strategy        judged on: technical direction

Read the "judged on" column downward. It moves from your work to the result of your work to other people's results, and that progression is the whole levelling model.

A junior is judged on whether the code is good. A staff engineer is judged on whether three other teams shipped, which is uncomfortable and is the actual bargain.

40.3 The Transition That Matters

Mid to senior is the transition people get stuck at, and it is not a technical one.

The mid engineer's job is to answer the question correctly. Given "build an incremental model for order facts," they build a correct one, with tests, handling late data. That is a real skill and most engineers never fully acquire it.

The senior engineer's job is to decide which question to answer, which means saying things like:

"Before I build this — who consumes it? Because §25.12 says four of our tables have no readers, and I would rather check than add a fifth."

⚠️ Failure Mode — getting better at the mid-level job and expecting a senior title

The most common career stall in this field, and the mechanism is that the strategy that got you here is precisely the one that stops working.

A mid engineer promoted to senior by doing more mid-level work faster is a common ask and a rare promotion, because the two levels are judged on different things: the component versus the outcome. Being twice as fast at building components is not evidence about outcomes.

What the evidence actually looks like, and it is uncomfortable because none of it is code:

• A thing you did not build, with a written reason. "We scoped this, found the consumer had stopped using the report, and deleted the request." This is the single strongest artifact and almost nobody has one.
• A decision you changed, with the argument that changed it. Chapter 35's withdrawn mesh proposal or Chapter 36's withdrawn ordering proposal are exactly this shape.
• A problem you found that nobody had asked about. Chapter 38's four failed reconciliations, or Chapter 30's four definitions of active_customer.
• Someone else's work that got better because of you — a review, a design you improved, a person you unblocked.

The trap is that all four are less visible than shipping, and a performance cycle rewards what is visible. So write them down as they happen, in a running document, with dates — because at review time you will remember what you shipped and not what you prevented.

And the honest caveat: some organizations do not have senior work available. If every request arrives fully specified and no decisions are open, you cannot demonstrate a level the job does not contain — which is §40.9's subject, and is a reason to leave rather than a reason to try harder.

40.4 What Staff Actually Is

Staff is the level most misunderstood, because the title implies "very senior engineer" and the job is different in kind.

Three things change:

Your scope becomes the platform, not a system. You are responsible for things you do not build.

Your output becomes mostly writing and conversations. Design documents, standards, reviews, and the meeting where two teams stop building the same thing twice. Engineers who become staff and keep coding full-time usually revert, not because coding is wrong but because the leverage is elsewhere.

And you are judged on other teams' outcomes, which means your best work is often invisible: the migration that did not happen, the mesh that was not adopted, the ordering guarantee nobody bought.

One honest warning: staff roles are scarce, and principal roles much more so. Do not plan a career around a level that may not exist at your company. Plan around scope, and take the title when it appears.

40.5 The Specializations

--tracks scores five tracks on five dimensions, and the chart's purpose is to show what each one costs:

                       demand  portab  on-cal  depth   obsole
streaming              ###     ####    #####   #####   ##
platform / infra       ####    ####    ####    ####    ##
analytics engineering  #####   #####   #       ###     ###
ML infrastructure      ####    ###     ###     #####   ####
governance / privacy   ###     #####   #       ###     #

No track dominates another, and the self-check asserts it — if one did, this would be advice rather than a trade-off.

Streaming is the deepest technically, has the smallest job market, and the worst on-call — and the depth and the on-call are the same fact.

Platform/infra is the default senior path and the most portable.

Analytics engineering has the largest market, the lightest on-call, and the shallowest technical depth. It is also closest to the business, which is where §40.1's durable skill lives.

ML infrastructure has the highest ceiling and the most exposure to a field that is still moving.

Governance and privacy has the smallest market and the most durable skills — GDPR is nine years old and the questions in Chapter 31 have not changed.

📐 Design Decision — read the on-call and obsolescence columns, because nobody does

People choose a specialization on demand and depth, which are the two columns that appear in job postings. The two that determine whether you are still doing it in five years are the other two.

On-call burden is a lifestyle decision disguised as a technical one. Streaming scores 5 on depth and 5 on on-call burden, and these are not independent: a system that processes events continuously fails continuously, and the interesting technical problems are interesting precisely because they happen at 3am. You cannot take the depth without the pager, and people routinely try.

Obsolescence risk is about what fraction of your skill is a product. ML infrastructure scores 4 because a large part of the current toolchain is under five years old; governance scores 1 because the skill is reading a regulation and making it operable, and the regulation moves slowly.

And the cross-column trade is the useful one: analytics engineering has the best demand and the lightest on-call and the shallowest depth. That is not a worse deal; it is a different one — and the honest framing is that it optimizes for breadth of employment and against technical specialization, which is exactly right for some people and wrong for others.

The recommendation, such as it is: pick the column you are least willing to compromise on, and let it choose. Most people who are unhappy in this field are unhappy about a column they did not look at.

40.6 The First Ninety Days in a Data Role

A data role has a distinctive onboarding failure, and it is not technical: you are handed custody of numbers you do not understand, and asked to change them.

A useful ninety days, in three parts:

Weeks 1–4: find out what is true. Not the architecture — the numbers. Which tables are read, by whom, and what decisions depend on them (Chapter 25 §25.12). Run the reconciliation if there is one; ask why not if there is not.

Weeks 5–8: break something small, on purpose, in staging. Ship a change, watch what catches it, and find out what does not. You will learn more about the platform's real controls in one afternoon than in a month of reading.

Weeks 9–12: ship one visible thing, and write one document. The document is the more valuable half.

🔎 Read the Plan — write the thing you wished existed on your first day

You have a capability in your first month that you will never have again: you do not know how this platform works.

Everything confusing to you is confusing, and in three months it will be obvious and you will have stopped being able to see it. Every new joiner has this and almost none of them uses it, because the instinct in a new job is to hide confusion rather than to record it.

So keep a running file from day one, and put in it every moment you were stuck:

text - took 40 min to find where orders are loaded (three repos, no README) - `stg_orders` vs `stg_orders_v2` -- nobody knows which is live - the 6am SLA is in a Slack message from 2024, not in any doc - I broke staging and nothing told me; found out from an analyst

At week twelve, that file is the onboarding document, and it is better than one written by a five-year veteran because they cannot remember what was hard.

Three reasons this is the single highest-leverage thing a new joiner does:

• It is the fastest credibility you can build. A useful artifact in month three, from someone nobody expected output from yet.
• It is §40.3's evidence — someone else's work got better because of you — in a form you can point at.
• And it is the only work that gets strictly harder every week you delay it. Everything else you will do can be done later; this one has a deadline set by your own memory.

The one caution: keep it factual and about the system, not about people. "Three repos, no README" is a finding. "Nobody documents anything" is a complaint, and it will be read by the person who wrote the repos.

40.7 The Four Ways People Leave

Not "burnout," which is a symptom. Four distinct causes, and they have different remedies:

Unplanned work above about half. Chapter 26's toil, unbounded. The remedy is organizational and you cannot supply it alone — if unplanned work is the culture, leaving is the correct response.

Custody without authority. You are responsible for whether numbers are right and cannot change the systems that produce them. This is the specific misery of a data role, and it is why Chapter 30's ownership and Chapter 38's rule-ownership matter beyond their technical content.

No visible outcome. Chapter 37's political problem, permanently. A career of migrations and maintenance with nothing to point at wears people down even when the work is good.

And the ceiling. A team where the senior work does not exist (§40.3).

🏭 From the Pipeline — the one that is fixable from inside

Three of the four require leaving. The second one sometimes does not, and it is worth knowing which because it is the most common.

"Custody without authority" usually has a specific, findable shape: you are accountable for a number, and the thing that makes it wrong is upstream, in a system owned by someone with no incentive to care.

What does not work: escalating, being right, and building more monitoring on your side. All three increase your certainty and change nothing, and doing them repeatedly is how the second cause becomes the first.

What sometimes works is making the cost visible to the person who can act, which is Chapter 33 §33.12's four rules applied to a data quality problem:

• Bring the alternative, not the problem. Not "your service sends bad timestamps" but "if the checkout service sent an offset, we'd stop having this conversation monthly — is that a two-line change?"
• Use their units. Not "0.24% of orders are affected" but "about sixteen customers a day get a confirmation email with the wrong delivery date."
• Go to them privately first.
• And attach a number to the cost of not fixing it, because an unquantified complaint is indistinguishable from a preference.

Kestrel's engineers used exactly this to get the supplier timezone fixed (Chapter 34 Case Study 2) and to get is_test respected upstream (Chapter 38 Case Study 1). Both had failed as escalations for months and succeeded as a quantified, alternative-bearing conversation in one meeting.

The honest limit: this works when the other party is reasonable and under-informed, and not when they are unreasonable or differently incentivized. Try it twice. If it fails twice with different people, the problem is structural and you are back to the three that require leaving.

40.8 What Makes a Data Team Good to Work On

The inverse of §40.7, and it is a short list:

Someone can say whether the numbers are right, and it is checkable. A reconciliation.

The work has a shape. Sprints, a roadmap, and unplanned work bounded well under half.

Deletion is possible. Chapter 39 §39.10's question, from the inside.

And decisions are made at the level that owns them. Finance owns the accounting rules; engineering owns the implementation; nobody is silently deciding somebody else's job in SQL (Chapter 38 Case Study 1).

Notice what is not on that list. Not the technology, not the scale, not whether the stack is modern. A team running cron and Postgres that can tell you whether its numbers are right is a better place to work than one running every tool in this book that cannot — and the second is more common than the first, because the tools are purchasable and the property is not.

The four items also have an order. Reconciliation first, because without it the other three are optimizing something you cannot evaluate. Then bounded work, then deletion, then decision ownership — which is roughly the order in which they become possible, and roughly the order Parts V through VIII of this book built them.

40.9 Scoring a Team From Outside

--health weights twenty signals, and eighteen are observable in an interview.

GOOD SIGNS
   +5  a reconciliation exists and runs on a schedule
   +4  they can name their last incident, with a timeline
   +4  on-call exists and pages less than twice a week
   +4  they have deleted something recently
   +4  unplanned work is under 40% of engineering time
   +3  pipelines outnumber dashboards
   +3  data quality assertions run in CI
   +3  engineers can deploy without a ticket
   ...
BAD SIGNS
   -5  'we don't really have incidents'
   -5  most engineering time is unplanned
   -4  nothing has ever been deleted
   -4  dashboards outnumber pipelines by more than 3x
   -4  no one can say what a number means without asking
   -3  the last person in the role left within a year
   ...
range: -31 to +39

🔎 Read the Plan — two good signs do not cancel two bad ones

The weighting matters more than the list, and the arithmetic makes a point that a checklist cannot.

A team with a catalog people use (+2) and cost attribution (+2), which says "we don't really have incidents" (−5) and where most engineering time is unplanned (−5), scores −6.

That team looks good on a tour. The catalog is visible, the cost dashboard is impressive, and both are real achievements. They are also the cheap signals — a catalog can be bought and populated by a crawler (Chapter 30 §30.1), and cost attribution is a tagging exercise.

The expensive signals are the ones with the big weights, and they cannot be bought:

text +5 a reconciliation on a schedule requires knowing what right means +4 a named incident with a timeline requires having had one and learned +4 something deleted recently requires measurement AND permission -5 unplanned work above half requires nothing; it is the default

The asymmetry is the finding: good signals require sustained effort and bad ones are the resting state. A team that has done nothing scores negative, which is correct, and it is why "they seem fine" is weak evidence.

And the two signals not observable from outside — whether a rebuild-from-raw has been attempted, and whether the platform roadmap survives a quarter — are worth asking about anyway. The answers are given honestly more often than you would expect, because nobody has thought to be defensive about them.

40.10 Getting Better on Purpose

Four things that work, ranked by how much they are underrated:

Operate something. §39.2's strongest interview signal is also the strongest development signal. Nothing substitutes for having been paged.

Write. Design docs, postmortems, and — the underrated one — the "not done" list (Chapter 38 §38.12). Writing forces the decisions that a conversation lets you defer.

Read other people's pipelines, especially bad ones. Chapter 37's migration is the fastest learning available in this field, because you are forced to understand a system built by somebody with different assumptions.

And teach. The fastest way to find out that you do not understand watermarks is to explain them.

🧱 Kestrel Platform — the three artifacts that compound, and the one that does not

Kestrel's team keeps a shared record of what actually made people better, gathered from four years of retrospectives. Three things compounded and one did not, and the one that did not is the one most people spend their time on.

```text COMPOUNDED being on call you learn the failure modes of YOUR system, which no course teaches writing the postmortem you learn them a second time, precisely, and somebody else learns them once reviewing someone else's you see designs you would not have written, design which is the only cure for one person's habits

DID NOT COMPOUND courses and certifications knowledge with no attachment point; forgotten in months unless immediately used ```

The distinguishing property is whether the learning is attached to a system you are responsible for. Being paged for a pipeline you own teaches you that pipeline's failure modes permanently, because the memory is attached to consequence. A course on streaming teaches you a vocabulary that decays unless you use it within weeks.

This is not an argument against courses — they are excellent for acquiring a vocabulary you are about to need, which is a real and specific use. It is an argument against them as a substitute for responsibility, which is how they are most often used, because responsibility is uncomfortable to ask for and a course is easy to enroll in.

The practical version, and it is one sentence: if you want to get better this year, ask to own something you are slightly not ready for, and make sure it pages you. Everything else on this list is downstream of that.

And the honest caveat about postmortems specifically: they compound only if the organization writes them without blame (Chapter 26). In a team where a postmortem is a disciplinary artifact, people write them defensively, learn nothing, and the second-best learning mechanism in the field is destroyed. That is worth checking before you join (§40.9).

40.11 Staying Current Without Chasing Everything

--skills splits fifteen years of tooling into what depreciated and what did not:

DURABLE                             PERISHABLE
SQL                    1974         Hadoop / MapReduce      2006-2016
dimensional modelling  1996         Hive                    2008-2018
idempotency and replay  -           a specific orchestrator ~5 years
partitioning and layout -           a specific streaming fw ~6 years
reconciliation         1494         a cloud's product names ~4 years
distributed systems    1978         the 'modern data stack' ~5 years
knowing what a number means

The durable list is mostly properties. The perishable list is mostly products.

🧭 Version Note — the rule that tells you where to spend a Saturday

"Learn fundamentals, not tools" is true and useless as stated, because every tool vendor claims their tool teaches fundamentals. The properties-versus-products distinction is the operational version, and it is checkable:

Ask of any skill: if this product disappeared tomorrow, what would I still know?

• Learning Airflow → you know DAGs, retries, idempotent tasks, backfills, and scheduling semantics. A great deal survives. Airflow is perishable; what you learned is not.
• Learning Airflow's UI and its operator zoo → almost nothing survives.
• Learning a cloud vendor's certification syllabus → the product names are the content, and the half-life is about four years.

This is why four years on a dead tool are not necessarily wasted. Someone who ran Hadoop from 2012 to 2016 learned partitioning, shuffle, skew, and the cost of moving data across a network — all of which are load-bearing in Spark, in a warehouse, and in whatever replaces both. Someone who learned the Hadoop ecosystem's product names got four years of nothing.

Three practical consequences:

• Learn one tool per category deeply rather than five shallowly. The second orchestrator takes a fortnight once you know the first properly.
• When something new appears, ask what problem it claims to solve and whether you have that problem. Chapters 29, 32, and 35 are three chapters of that question, and the answer was no twice.
• And read the durable list's sources. Kimball is thirty years old and still correct; Lamport is from 1978. Nothing in this book's further-reading sections has a shorter half-life than its subject.

40.12 Compensation, Briefly

Three things that are true across markets, and one warning that this section dates faster than any other page in the book.

Level dominates everything else. The gap between mid and senior at the same company is normally larger than the gap between companies at the same level. Which makes §40.3 the compensation section, and makes under-levelling the expensive mistake (Chapter 39 §39.13).

The specialization premium is smaller than the demand difference. Streaming and ML infrastructure pay somewhat more per role; analytics engineering has several times as many roles. Over a career, the number of doors open usually matters more than the differential on one.

And the largest single lever is changing jobs, which is unromantic, well documented, and worth knowing so that you can decide deliberately rather than discover it later.

💸 Cost Check — price the year, not the offer

Chapter 39 §39.13's claim, made concrete: the cost of a wrong job is a year, and it dominates any difference you could negotiate.

text negotiating an offer up typically 3-8%, once a year in the wrong role one year of learning nothing you wanted + a re-run of the whole search + a CV entry you have to explain

Case Study 2 of Chapter 39 is the worked example: eleven months, two pipelines, and information that was available in a forty-minute conversation.

Two practical consequences:

• Spend the negotiating energy on scope before salary. "Will I own a pipeline end to end, including its on-call?" is worth more than 5%, and it is negotiable at offer stage in a way it is not later.
• And get the scope in writing, or at least in an email you send summarizing the conversation. Not as a legal instrument — it will not be one — but because writing it down is what makes both parties notice a disagreement while it is still cheap.

The honest caveat: none of this helps if you need the job. Advice about walking away is advice for people with a runway, and pretending otherwise is the most common dishonesty in career writing. If you do not have that runway, take the job, use §40.9's scoring to know what you are walking into, and start the next search from a position of employment, which is a much better one.

🎓 Interview Angle — the two questions that set your level, and they are asked early

Levelling is usually decided in the recruiter screen and the hiring-manager conversation, not in the technical rounds — which surprises people who prepare only for the technical rounds.

"What are you looking for in your next role?" is a levelling question. "More interesting work" is a mid answer. "I want to own a system end to end, including what gets built on it" is a senior one, and it is the same person describing the same preference at two levels of specificity.

"Tell me about your current scope" is the other one, and the useful structure is what you own, who depends on it, and what you decided — the third being the part that distinguishes levels (§40.2). "I own the order pipeline; four teams consume it; last quarter I decided not to build the real-time version and wrote up why" is a complete senior answer in nineteen words.

And say the level you are targeting, early and plainly. It feels presumptuous and it is the single most effective thing you can do about levelling, because the alternative is that somebody guesses — and a guess made from a résumé defaults low.

📏 Scale Note — the job is different at four engineers and at four hundred

"Data engineer" describes work that varies more by organisation size than by seniority, and knowing which shape you are in — and which you want — is most of a career decision.

text 1-2 engineers ~10 ~50 400+ ───────────────────────────────────────────────────────────────────────── what you do everything most things a layer a component what you own the whole a domain a system a service platform the hard part choosing boundaries coordination scope, and what NOT to finding the build work what you learn breadth, fast depth in 2-3 depth in one specialism areas the risk no second the platform politics invisibility opinion becomes somebody's identity the best thing you see the you can real genuinely hard whole actually specialisms problems at lifecycle finish things scale

The column people underestimate is the first. A two-person team is where you learn the whole of this book in eighteen months, because there is nobody else to do any of it — and it is where the habits form, badly or well, with nobody to correct them.

The column people overestimate is the last. Four hundred engineers means the interesting problems exist and finding them is the job, which is a different skill from solving them and is the one nobody warns you about.

Two practical consequences:

Match the column to what you are trying to learn, not to the title. A staff title in a small team and a mid-level title in a large one can involve the same work at different pay, and the scope of ambiguity you resolve is the axis that transfers (§40.4).

And be honest about the risk row. Every column has one. A career decision that only weighs the best-thing row is a decision you will re-make in eighteen months.

🧪 Try It — write down what you know, and what you have only read about

Twenty minutes, and it is uncomfortable in a useful way.

Take the forty chapters of this book. For each, mark one of four:

text BUILT I have built this in production and been paged for it DONE I have done it once, in this book or a project READ I understand the argument and have not done it NO I have not read it

Then count. Most readers who have worked through this book will have something like:

text BUILT 4-8 the things their current job touches DONE 20-30 the increments READ 5-15 NO 0-3

Three things to do with the result, and the third is the point.

Look at the BUILT column and ask what it has in common. It is usually one part of the lifecycle, and it is the thing your CV says you are. That is fine and it is worth knowing.

Look at the gap between DONE and BUILT. Those are the chapters where you have the shape and not the scars, and they are exactly the chapters where an interviewer's third follow-up will find the edge. Prepare two of them properly rather than all of them shallowly (Chapter 39).

And pick one READ that you can convert to DONE at your current job, this quarter. Not the most interesting one — the one that would remove a real problem your team has. Chapter 17's observed contract, Chapter 23's coverage audit, Chapter 25's margin, and Chapter 30's metric audit are the four that most readers can ship without anybody's permission.

Then date the list and keep it (§40.1). Redo it in a year. The interesting output is not the counts; it is which rows moved, and whether they moved because you chose them or because your job happened to require them — which is Exercise 40.10's question and is the difference between a career and a sequence of jobs.

🔐 Privacy & Governance — the part of this job that is not technical, and does not go away

Every chapter in Part VI put a privacy consideration next to a technical decision, and the reason is a career one rather than a compliance one: you will be the person who knows where the data is.

text what the job increasingly involves why it lands on data engineering ───────────────────────────────────────────────────────────────────────── answering "what do we hold about nobody else can enumerate it this person" implementing a deletion that is actually complete it spans every store (ch 12 🔐) saying no to a request that is you are the only person in the technically easy room who knows it is easy being in the room when a decision because the decision is about about people is automated features you built (ch 32 🔐)

The third row is the career-relevant one. A request to join two datasets is frequently trivial to execute and consequential to have executed, and the person who can say "yes, and here is what it would mean" is more valuable than the person who can only do it.

Three things worth developing deliberately, none of which is legal expertise:

The ability to state what you hold, quickly. Not from memory — from a generated artifact (Chapter 31). It is the difference between a two-hour answer and a two-week one, and it is asked at the worst moments.

The habit of asking "and how does this forget?" at adoption (Exercise 5.20, Exercise 12.20). One sentence, at the cheapest possible moment.

And a working relationship with whoever owns legal or compliance, before you need one. They are usually delighted to be asked and almost never are, and the questions that genuinely need them (Exercise 6.17's three) are few and identifiable.

The framing that has aged best: privacy is not a constraint on the work, it is part of the specification. A pipeline that produces the right number and cannot forget a person is not finished — and being the engineer who notices that, early, in a design review, is a specific and durable kind of seniority.

🔁 Idempotency Check — run your own career review twice

This book's central technical habit has a career version, and it is the same test: do it twice and see whether you get the same answer.

text the artifact re-run it in ───────────────────────────────────────────────────────────────────── the skills inventory (§40.13's 🧪) 12 months the track scoring (Ex. 40.7) 12 months your team's score (Ex. 40.8) 6 months -- teams change faster the five-year letter (Ex. 40.14) annually, WITHOUT rereading the previous one first

The last row is the one with the method in it. Write the letter, seal it, and write next year's before you open it. Then compare.

Three things the comparison tells you that a single writing cannot.

Whether your answers are stable. A five-year sketch that is unrecognisable each year is not necessarily wrong — it may mean you are learning fast — and it does mean you should not be making irreversible decisions on it.

Which rows moved, and why. The useful question from §40.13's 🧪: did a skill move from READ to DONE because you chose it, or because your job happened to require it? Both are fine and they are different careers.

And whether the thing you said mattered still does. The most common finding is that the goal stayed and the reason changed — which is worth noticing before you spend another year on it.

The technical parallel is exact and worth stating, because it is the last idea in the book. A pipeline you have never run twice is a pipeline you cannot trust. A conclusion you have never re-derived is a conclusion you have inherited from a previous version of yourself, and the only way to find out which is to do it again and diff.

Exercise 40.15's final question is the same test in one line: how would you know if you were wrong about any of this? A career with an answer to that — a review date, a person who will tell you, a number you track — has the property this whole book has been arguing for.

🧭 Version Note — the job title has moved twice, and will again

"Data engineer" has meant three different things in fifteen years, and knowing which one a job description means matters more than the title.

text era the job was the adjacent titles ───────────────────────────────────────────────────────────────────────── ~2010 an ETL developer, in a GUI BI developer, warehouse or in Java/Hadoop developer ~2016 a software engineer who "big data engineer", which built pipelines mostly meant Hadoop ~2020 split into two: the platform analytics engineer (dbt, SQL, half and the analytics half modelling) and data platform engineer (infra, streaming) now splitting again, at the edges ML platform / feature engineering on one side; data reliability and governance on the other

The 2020 split is the one that matters most to a reader of this book, because this book covers both halves deliberately — and most job descriptions cover one. Read the responsibilities rather than the title: a role that mentions dbt, modelling, and stakeholders is the analytics half; one that mentions Kubernetes, streaming, and on-call is the platform half.

What has been stable across all four eras, and is the reason the split is less important than it looks: somebody has to know where the data came from, what it means, and whether it is right. Every era has employed people to do that under a different name, and none has stopped.

Two consequences for a career decision:

The title tells you about the market; the responsibilities tell you about the job. A "senior data engineer" posting can be either half or both, and the interview is where you find out (Chapter 39's reverse-interview questions).

And a specialism is a two-year bet with a review date (Exercise 40.7). The half you pick will change shape within that window — which is an argument for picking one and re-examining it, not for avoiding the choice.

💸 Cost Check — what learning costs, and the only currency that matters

Time is the constraint, not money, and the ratios are worth knowing before you spend a year on the wrong thing.

text activity cost what it reliably buys ───────────────────────────────────────────────────────────────────────── reading a book like this one ~120 h the shape of the whole field, and vocabulary building the platform in it ~200 h the ability to do it, and an interview story a certification ~40 h + a fee a filter passed; rarely a skill a conference 2 days + travel two or three ideas, and the people, which is the actual value shipping ONE thing at work you have not done before ~20-60 h more than any of the above, per hour

The bottom row is the highest return and it is the one nobody schedules, because it does not look like learning. Chapter 40 §40.13's 🧪 asks you to pick one READ and convert it to DONE this quarter — an observed contract, a coverage audit, a margin extrapolation — and that is what the row means in practice.

Two honest observations about the middle rows.

A certification is a filter, not a skill, and it is worth exactly what the filter is worth in your market. In some markets that is a great deal; the mistake is expecting the study to teach you something the exam does not test.

And a conference's value is the corridor, not the talks. The talks are recorded. Budget the time for the conversations and treat the sessions as optional, which inverts how most people plan one.

The measurement that makes any of this tractable is Exercise 40.10's: what did you learn last year, and what was it for? The uncomfortable result — most learning was reactive, driven by whatever the job required — is not a failure. It is the highest-return row on the table happening by accident, and the only change worth making is to choose one of them deliberately each quarter.

40.13 The Last Thing

Forty chapters, and the argument has been the same one throughout, stated in different domains.

A pipeline that runs is not a pipeline that is right. A test that passes is not a control that operates. A control that exists is not a control that works. A number that is green is not a number that is true.

And the reason this matters is not craftsmanship. It is that:

You are the reason a number on a screen means what someone thinks it means, and nearly nobody downstream is in a position to check.

The dashboard's reader cannot verify it. The executive making a decision on it cannot verify it. The analyst who built the chart is usually taking your table's meaning on trust, and the further from the pipeline a number travels, the more completely it is believed.

That is custody, and it is the part of this job that does not depreciate, does not get automated, and does not appear in any tool's documentation.

Everything else in these forty chapters — the partitioning, the idempotency, the assertions, the reconciliation, the layer rules, the deletion manifest, the cost model — is machinery in service of one question, which is the question §40.9 weights at +5 and which the whole book has been teaching you to answer:

How do you know?

40.14 Summary

The job has changed shape three times in fifteen years and the core has not moved: somebody has to be able to say what a number means and whether it is right. The infrastructure keeps getting easier and the custody does not.

Levels are the question you are trusted to answer: how do I build this · is this built correctly · should this be built · what should we be building · what will we need in two years. And the "judged on" column moves from your work to your outcome to other people's outcomes.

⚠️ The mid-to-senior stall is doing more mid-level work faster. The two levels are judged on different things, so being twice as fast at building components is not evidence about outcomes. The evidence is four things, none of them code: a thing you did not build with a written reason (the strongest, and almost nobody has one) · a decision you changed · a problem you found that nobody asked about · someone else's work that got better. Write them down as they happen, because a review cycle rewards what is visible.

And if the senior work does not exist at your company, you cannot demonstrate a level the job does not contain — a reason to leave rather than to try harder.

Staff is different in kind, not degree: scope becomes the platform, output becomes writing and conversations, and your best work is often invisible — the migration that did not happen, the mesh not adopted, the ordering guarantee nobody bought. Plan around scope; take the title when it appears.

📐 Every specialization trades something, and the two columns nobody reads are on-call and obsolescence. Streaming's depth and its pager are the same fact. Governance's durability comes with the smallest market. Analytics engineering optimizes for breadth of employment and against specialization, which is right for some people and wrong for others. Pick the column you are least willing to compromise on and let it choose.

🔎 In your first ninety days, write the thing you wished existed on day one. You have a capability you will never have again — you do not know how this works, and in three months you will have stopped being able to see what was confusing. Keep a running file of every moment you were stuck; at week twelve it is the onboarding document, and it is better than one written by a five-year veteran because they cannot remember what was hard. It is also the only work that gets strictly harder every week you delay it. Keep it about the system, not about people.

Four ways people leave, and three require leaving: unplanned work above half · custody without authority · no visible outcome · a ceiling.

🏭 Custody without authority is the fixable one, sometimes. Escalating, being right, and building more monitoring all increase your certainty and change nothing. What works is Chapter 33 §33.12's four rules: bring the alternative · use their units · privately first · attach a number to the cost of not fixing it. Two of Kestrel's upstream fixes had failed as escalations for months and succeeded in one meeting. Try it twice; if it fails twice with different people, the problem is structural.

🔎 Eighteen of twenty team-health signals are observable from outside, and two good signs do not cancel two bad ones. The cheap signals — a catalog, cost attribution — are visible on a tour; the expensive ones cannot be bought: a reconciliation requires knowing what right means, a named incident requires having learned from one, a deletion requires measurement and permission. Good signals require sustained effort and bad ones are the resting state, so a team that has done nothing scores negative — which is why "they seem fine" is weak evidence.

💸 Level dominates compensation — the mid-to-senior gap at one company usually exceeds the gap between companies at one level — which makes §40.3 the compensation section. Spend the negotiating energy on scope before salary, and get it in writing, not as a legal instrument but because writing it down is what makes both parties notice a disagreement while it is cheap. And none of this helps if you need the job; advice about walking away is advice for people with a runway.

🎓 Levelling is decided in the recruiter screen, not the technical rounds. "More interesting work" is a mid answer; "I want to own a system end to end, including what gets built on it" is a senior one. Say the level you are targeting, early and plainly — the alternative is that somebody guesses, and a guess made from a résumé defaults low.

🧭 The durable skills are properties; the perishable ones are products. Ask of anything: if this product disappeared tomorrow, what would I still know? Learning Airflow teaches DAGs, retries, idempotency, and backfills — all of which survive Airflow. Learning its operator zoo teaches nothing that does. This is why four years on a dead tool are not necessarily wasted, and why someone who learned an ecosystem's product names got four years of nothing.

Learn one tool per category deeply. Ask of anything new whether you have the problem it solves — Chapters 29, 32, and 35 are that question asked three times, and the answer was no twice.

And the last thing. A pipeline that runs is not a pipeline that is right; a control that exists is not a control that operates; a number that is green is not a number that is true.

You are the reason a number on a screen means what someone thinks it means, and nearly nobody downstream is in a position to check. Everything in these forty chapters is machinery in service of one question:

How do you know?

Key terms: levelling · scope · the mid-to-senior transition · staff engineer · specialization · on-call burden · obsolescence risk · unplanned work · custody without authority · technical currency · properties versus products