Key Takeaways: The Data Engineering Career
The one thing
The job has changed shape three times in fifteen years and the core has not moved: somebody has to be able to say what a number means and whether it is right. The infrastructure keeps getting easier and the custody does not.
Levels
The question you are trusted to answer, not years and not a skill list:
JUNIOR How do I build this? judged on: a code review
MID Is this built correctly? judged on: a design review
SENIOR Should this be built? judged on: the outcome
STAFF What should we be building? judged on: other teams' outcomes
PRINCIPAL What will we need in two years? judged on: technical direction
Read the "judged on" column downward: from your work, to the result of your work, to other people's results. A junior is judged on whether the code is good; a staff engineer on whether three other teams shipped.
Senior is the first level where the answer can be "no."
The transition that stalls people
⚠️ Getting better at the mid-level job and expecting a senior title. The two levels are judged on different things, so being twice as fast at building components is not evidence about outcomes.
Four kinds of evidence, none of them code:
- A thing you did not build, with a written reason — the strongest, and almost nobody has one.
- A decision you changed, with the argument that changed it.
- A problem you found that nobody had asked about.
- Someone else's work that got better because of you.
📐 Write them down as they happen. Shipped work leaves artifacts; prevented work leaves nothing. Two lines, ninety seconds, within the hour, without judging whether it counts — and the same file is the source for Chapter 39's behavioral round.
🔎 Ask for an instance, not a criterion. "Show me something a person did that got them promoted." Four minutes. It is answerable where the criteria are not, it is not a negotiation, and it reveals the pattern — all three of Case Study 2's examples were about something not happening.
And if the senior work does not exist at your company, you cannot demonstrate a level the job does not contain. The test: ask what decision the next request serves. Welcomed → the work exists. Treated as obstruction → a reason to leave, not to try harder.
Staff
Different in kind, not degree. Scope becomes the platform · output becomes writing and conversations · your best work is often invisible.
Engineers who become staff and keep coding full-time usually revert — not because coding is wrong but because the leverage is elsewhere.
Staff roles are scarce and principal roles much more so. Plan around scope; take the title when it appears.
Specializations
demand portab on-cal depth obsole
streaming ### #### ##### ##### ##
platform / infra #### #### #### #### ##
analytics engineering ##### ##### # ### ###
ML infrastructure #### ### ### ##### ####
governance / privacy ### ##### # ### #
📐 No track dominates — the self-check asserts it, because otherwise this would be advice rather than a trade-off.
Read the two columns nobody reads: on-call burden and obsolescence risk.
Streaming's depth and its pager are the same fact — a system that processes events continuously fails continuously, and the interesting problems are interesting because they happen at 3am. You cannot take the depth without the pager.
Analytics engineering optimizes for breadth of employment and against specialization, which is right for some people and wrong for others. Governance is the most durable and the smallest market.
Pick the column you are least willing to compromise on and let it choose. Most people unhappy in this field are unhappy about a column they did not look at.
The first ninety days
🔎 Write the thing you wished existed on day one.
You have a capability you will never have again: you do not know how this works, and in three months everything confusing will be obvious and you will have stopped being able to see it.
Keep a running file of every moment you were stuck. At week twelve it is the onboarding document, and it is better than one written by a five-year veteran because they cannot remember what was hard.
It is the only work that gets strictly harder every week you delay it. And keep it about the system, not about people — "three repos, no README" is a finding; "nobody documents anything" is a complaint that will be read by the person who wrote the repos.
Why people leave
Four causes, and three require leaving: unplanned work above half · custody without authority · no visible outcome · a ceiling.
🏭 Custody without authority is sometimes fixable. Escalating, being right, and building more monitoring all increase your certainty and change nothing. What works is Chapter 33 §33.12's four rules: bring the alternative · use their units · privately first · attach a number to the cost of not fixing it.
Two of Kestrel's upstream fixes had failed as escalations for months and succeeded in one meeting. Try it twice. If it fails twice with different people, the problem is structural.
A team worth working on
A reconciliation exists · the work has a shape (unplanned well under half) · deletion is possible · decisions are made at the level that owns them.
Notice what is not on that list: the technology, the scale, whether the stack is modern. A team running cron and Postgres that can tell you whether its numbers are right is a better place to work than one running every tool in this book that cannot — and the second is more common, because the tools are purchasable and the property is not.
🔎 Eighteen of twenty health signals are observable from outside, and two good signs do not cancel two bad ones.
+5 a reconciliation on a schedule -5 "we don't really have incidents"
+4 a named incident with a timeline -5 most engineering time is unplanned
+4 something deleted recently -4 nothing has ever been deleted
+4 unplanned work under 40% -4 dashboards >> pipelines
The cheap signals — a catalog, cost attribution — are visible on a tour. The expensive ones cannot be bought. A reconciliation requires knowing what right means; a deletion requires measurement and permission.
Good signals require sustained effort and bad ones are the resting state, so a team that has done nothing scores negative — which is why "they seem fine" is weak evidence.
Getting better
Four things, ranked by how underrated they are: operate something · write · read other people's pipelines, especially bad ones · teach.
🧱 Three compound and one does not. Being on call, writing the postmortem, and reviewing someone else's design compound; courses and certifications do not.
The distinguishing property is whether the learning is attached to a system you are responsible for. Being paged teaches permanently because the memory is attached to consequence; a course teaches a vocabulary that decays unless used within weeks.
Not an argument against courses — they are excellent for acquiring a vocabulary you are about to need — but against them as a substitute for responsibility, which is how they are most often used, because responsibility is uncomfortable to ask for and a course is easy to enroll in.
One sentence: ask to own something you are slightly not ready for, and make sure it pages you.
And postmortems compound only in a blameless organization. Where a postmortem is a disciplinary artifact, people write them defensively and the second-best learning mechanism in the field is destroyed. Check before you join.
Staying current
🧭 The durable skills are properties; the perishable ones are products.
DURABLE PERISHABLE
SQL 1974 Hadoop / MapReduce 2006-2016
dimensional modelling 1996 a specific orchestrator ~5 years
idempotency and replay - a specific streaming fw ~6 years
partitioning and layout - a cloud's product names ~4 years
reconciliation 1494 the 'modern data stack' ~5 years
Ask of anything: if this product disappeared tomorrow, what would I still know? Learning Airflow teaches DAGs, retries, idempotency, and backfills — all of which survive Airflow. Learning its operator zoo teaches nothing that does.
This is why four years on a dead tool are not necessarily wasted — and why someone who learned the same tool's product names got four years of nothing.
Learn one tool per category deeply. The second is a fortnight once the first is properly understood. Ask of anything new whether you have the problem it solves — Chapters 29, 32, and 35 are that question asked three times, and the answer was no twice.
Compensation
💸 Level dominates. The mid-to-senior gap at one company usually exceeds the gap between companies at one level, which makes the levels section the compensation section.
Spend the negotiating energy on scope before salary, and get it in writing — not as a legal instrument but because writing it down is what makes both parties notice a disagreement while it is cheap.
And none of this helps if you need the job. Advice about walking away is advice for people with a runway, and pretending otherwise is the most common dishonesty in career writing.
🎓 Levelling is decided in the recruiter screen, not the technical rounds. "More interesting work" is a mid answer; "I want to own a system end to end, including what gets built on it" is a senior one. Say the level you are targeting, early and plainly — the alternative is that somebody guesses, and a guess from a résumé defaults low.
The last thing
A pipeline that runs is not a pipeline that is right. A test that passes is not a control that operates. A control that exists is not a control that works. A number that is green is not a number that is true.
You are the reason a number on a screen means what someone thinks it means, and nearly nobody downstream is in a position to check.
That is custody, and it does not depreciate, does not get automated, and appears in no tool's documentation.
Everything in these forty chapters is machinery in service of one question:
How do you know?
The code
code/career_map.py — five levels as questions, five tracks scored on five dimensions with a
no-dominance check, twenty weighted team-health signals of which eighteen are observable from outside,
and the durable-versus-perishable skill split. Seventy-two self-checks, including that no
specialization dominates and that two good signals do not outweigh two of the worst bad ones.