Chapter 38: Quiz
Test your understanding of technical leadership, organizational dynamics, strategic decision-making, and career development for staff-level data scientists. Answers follow each question.
Question 1
What are the four archetypes of staff-level work identified by Will Larson, and how does each manifest in a data science organization?
Answer
The four archetypes are: **(1) Tech Lead** — steers a specific team or project; in data science, owns the modeling approach for a product area (e.g., defining evaluation metrics, selecting architectures, reviewing experiment designs). **(2) Architect** — sets technical direction across teams; in data science, defines organizational standards for how models are trained, validated, deployed, and monitored. **(3) Solver** — parachutes into the hardest problems; in data science, is called when a model underperforms and the team cannot diagnose why, or when confounding structures defeat standard analysis. **(4) Right Hand** — extends a senior leader's reach; in data science, translates between the VP/CDO and individual teams, converting strategic priorities into technical roadmaps and technical progress into executive communication. Most staff data scientists blend 2-3 archetypes rather than fitting cleanly into one.Question 2
Why is the management track described as a "lateral move into a different profession" rather than a promotion from the IC track?
Answer
Because the core activities of management — hiring, developing, and retaining people; allocating resources; shielding the team from organizational dysfunction; and representing the team to stakeholders — are fundamentally different skills from the core activities of an IC — building models, designing systems, and making technical decisions. Excellence in one does not predict excellence in the other. A brilliant modeler may be a poor manager (unable to give constructive feedback, uninterested in career development conversations), and a brilliant manager may lack the technical depth to evaluate architectural decisions. At companies with well-defined IC tracks, staff and principal ICs are peers with directors and VPs in scope, compensation, and influence — the authority source differs (credibility vs. positional authority), not the level.Question 3
What is the most important section of a design document, and why?
Answer
The **Alternatives Considered** section. A design document that presents a single approach is an announcement, not a proposal. The alternatives section forces the proposer to justify their choice against at least two other credible approaches, making the reasoning explicit and reviewable. Without documented alternatives, future engineers cannot evaluate whether the original decision still makes sense when context changes (e.g., an alternative that was rejected due to infrastructure complexity may become viable after an infrastructure upgrade). This mirrors the ADR discipline from Chapter 36: the "why not" is often more informative than the "why."Question 4
A design review ends with four suggested changes from four different reviewers, some contradictory. Should all four be incorporated? Why or why not?
Answer
No. Incorporating all suggestions produces the **Design-by-Committee** anti-pattern — a Frankenstein design that reflects no coherent vision. The proposer owns the design. Reviewers provide input; the proposer makes the final decision. The correct process is: the proposer evaluates each suggestion, incorporates those that improve the design, explains why others are not adopted (documenting the reasoning), and maintains a coherent architectural vision. A design review is a collaborative conversation that improves the design, not a vote where every participant's suggestion must be implemented.Question 5
When should a data science team write an RFC instead of conducting a design review?
Answer
The threshold is: **does this decision constrain or affect teams beyond the proposer's own?** A design review evaluates a specific technical proposal for a single project (e.g., "should we use doubly robust estimation for this experiment?"). An RFC proposes a change that affects the entire organization (e.g., "all experiments must report doubly robust causal ATE estimates"). Examples requiring RFCs: standardizing a model serving framework across the organization, changing the feature naming convention, establishing a policy on experiment analysis standards, or deprecating a model training framework. Examples not requiring RFCs: choosing a model architecture for one project, adding a feature to the feature store (standard operational process).Question 6
What are the three mechanisms through which a staff data scientist exercises "influence without authority"?
Answer
**(1) Credibility** — a track record of technical decisions that led to good outcomes, earned slowly over years and lost quickly through a single catastrophic recommendation. Protected by honesty about uncertainty, early admission of mistakes, and never overpromising. **(2) Reciprocity** — helping other teams (reviewing designs, sharing data, lending expertise) creates goodwill that can be drawn upon later. **(3) Clarity** — communicating technical concepts in language that non-technical stakeholders understand. This is not simplification but *translation*: the same concept expressed technically ("high epistemic uncertainty"), in product language ("not confident in recommendations"), or in business language ("40% less effective, costing approximately $2M annually"). All three mechanisms compound over time.Question 7
Using the four-step framework from Section 38.5, how should a staff data scientist say no to a project request?
Answer
**(1) Acknowledge the value.** Demonstrate that you understand the business motivation and the potential impact. **(2) Explain the tradeoff.** Make explicit what would be displaced or delayed if this project is accepted — framing the decision as a resource allocation problem, not a rejection. **(3) Offer an alternative.** Propose a lower-effort approach that partially addresses the need (e.g., a rules-based system instead of an ML model, a simpler model instead of a complex one, a phased delivery instead of a full build). **(4) Escalate if needed.** If the stakeholder insists and the tradeoff involves a decision above your scope (e.g., competing P0 priorities), escalate to leadership with a prepared business case for both options. The alternative is critical: saying "no" without an alternative is obstruction; saying "not this way, but here's what we can do" is leadership.Question 8
What is the "differentiation test" in a build vs. buy decision, and how does it apply to StreamRec's feature store vs. recommendation model?
Answer
The differentiation test asks: **does this capability differentiate us from our competitors?** Capabilities that create competitive advantage should be built; capabilities that need to work reliably but do not differentiate should be bought or adopted from open source. StreamRec's **recommendation models** (retrieval, ranking, personalization) are differentiators — they encode StreamRec's unique understanding of its users and content, and no vendor can sell a better version because quality depends on proprietary data and domain knowledge. These should be **built**. StreamRec's **feature store, pipeline orchestration, and monitoring stack** are not differentiators — they need to work reliably, but mature open-source tools (Feast, Dagster, Grafana) exist. Engineering time spent maintaining custom infrastructure is time not spent improving differentiating models. These should be **adopted (OSS) or bought (managed service)**.Question 9
What five elements should a well-constructed technical roadmap contain?
Answer
**(1) Technical Vision** — a 1-2 page narrative describing the desired end state in terms of outcomes (not technologies), providing direction for 2-3 years. **(2) Key Bets** — the 3-5 major technical investments that move toward the vision, each justified with a business case and a technical case, sequenced by dependencies, risk, and expected value. **(3) Team Gaps** — an honest assessment of missing capabilities and the plan to acquire them (hiring, training, contracting). **(4) Success Metrics** — measurable outcomes (leading indicators preferred over lagging indicators) for each key bet. **(5) Dependencies and Risks** — external factors that could block or accelerate roadmap items, with mitigation plans and escalation triggers. A roadmap without all five elements is either a shopping list (no prioritization), a technology showcase (means without ends), or a fantasy (no risk assessment).Question 10
What is the difference between an OKR's Objective and its Key Results? Why is this distinction important for data science teams?
Answer
An **Objective** is a qualitative description of what the team wants to achieve (e.g., "Reduce new-user churn"). A **Key Result** is a specific, measurable outcome that indicates progress toward the objective (e.g., "7-day churn rate for new users decreases from 42% to 38%"). The distinction is critical because data science teams are prone to confusing activities with outcomes. "Deploy Bayesian cold-start model" is an *activity* — it can be completed without achieving the desired business outcome. "7-day churn rate decreases from 42% to 38%" is an *outcome* — it measures whether the activity actually worked. Key Results should be measurable outcomes, not activities. This discipline forces the team to confront whether their work is creating business value, not just technical artifacts.Question 11
What are the four platform bet evaluation criteria, and why is "community and ecosystem" often more important than technical superiority?
Answer
The four criteria are: **(1) Community and ecosystem** — size and activity of contributors, tutorials, Stack Overflow answers, and third-party integrations. **(2) Hiring signal** — whether you can hire people with this technology on their resume. **(3) Migration path** — how difficult it will be to move away when the platform is eventually replaced; open standards reduce switching costs. **(4) Organizational fit** — whether the platform requires fundamental changes to team workflows and mental models. Community and ecosystem is often decisive because you are adopting not just a technology but an *ecosystem*. A technically superior platform with a small community means fewer learning resources, fewer pre-built integrations, fewer answered questions, and a smaller hiring pool. A slightly inferior platform with a massive community provides abundant resources, rapid bug fixes, extensive integrations, and a deep talent pool — which typically outweighs pure technical merit over a 3-5 year horizon.Question 12
Why is mentoring described as "compound interest" for technical leadership?
Answer
Because the data scientists you develop will collectively produce more value over their careers than any single system you personally build. A staff DS who mentors 3 people per year, each of whom becomes 20% more effective, creates compounding returns: those mentees mentor others, apply better judgment to their own decisions, and carry the technical culture forward even after the mentor moves on. This is the definition of leverage — the mentor's influence outlasts their direct involvement. However, the compound interest metaphor also implies that mentoring requires *consistent investment over time*. Sporadic, ad-hoc advice does not compound; structured, persistent mentoring (biweekly sessions, tracked commitments, progressive skill development) does. The highest-ROI investment a staff DS can make is in the two or three mentees who have the potential and motivation to become staff-level themselves.Question 13
A senior data scientist asks you to review their design for a new model. The design is solid but uses a complex approach when a simpler one would likely achieve 90% of the benefit. How do you handle this in the design review?
Answer
Ask: **"What is the simplest version of this that would test the core hypothesis?"** (Theme 6: Simplest Model That Works). This question is not a criticism — it is a calibration tool. If the proposer can articulate why the simple approach would fail and the complex approach is necessary, the design is justified. If they cannot, then the design review has surfaced an untested assumption. The constructive approach is to propose a phased plan: implement the simple version first, measure its performance, and invest in the complex version only if the simple version falls measurably short. This respects the proposer's expertise (they may be right that the complex approach is needed) while protecting against over-engineering (they may also be wrong). The key is to frame the question as "help me understand" rather than "you're overcomplicating this."Question 14
What are the three criteria for promotion from senior to staff data scientist?
Answer
**(1) Judgment** — the ability to make good technical decisions under uncertainty, demonstrated through design reviews, RFCs, and architectural decisions that stand the test of time. Not about being right every time, but about being right more often than not, admitting errors, and learning from outcomes. **(2) Scope** — operating beyond a single project or team. A senior DS is responsible for their model; a staff DS is responsible for the modeling approach across a product area. Demonstrated by the breadth of work you *influence*, not the breadth you personally do. **(3) Impact** — creating value visible at the organizational level. "Built a model that improved CTR by 3%" is senior-level impact. "Designed the experimentation framework every team uses" is staff-level impact. All three must be present simultaneously — deep judgment without scope produces a brilliant individual contributor who does not multiply others; broad scope without judgment produces a well-connected person who makes poor decisions.Question 15
Why should a fairness audit precede the finalization of a technical roadmap?
Answer
Because a fairness audit can fundamentally reorder roadmap priorities by revealing cross-cutting concerns invisible to purely technical analysis. The StreamRec example from Chapter 36 demonstrated this: the FAISS index rebuild delay was initially classified as a P2 latency optimization, but the fairness audit revealed it was also a fairness problem — new creators are disproportionately non-English-speaking, so the 24-hour window where new items are invisible amplifies language-based exposure inequity. This reframed the item from P2 to P0. Without the fairness audit, the roadmap would have underinvested in a fairness-critical fix. More broadly, fairness audits reveal which system behaviors disproportionately affect which populations, which directly informs where engineering effort creates the most equitable impact. A roadmap built without fairness analysis risks optimizing for already-well-served users.Question 16
How does the staff data scientist's role differ when communicating with a product manager vs. an executive vs. a legal/compliance stakeholder?
Answer
Each stakeholder cares about different things: **Product managers** care about feature impact, user experience, and ship dates. The staff DS communicates in terms of user-facing metrics (CTR, retention, satisfaction) and proposes phased delivery to balance quality with speed. **Executives** care about revenue impact, competitive position, and cost. The staff DS uses the three-slide rule: (1) what and why, (2) how we know it works, (3) what comes next with cost. No architecture diagrams unless asked. **Legal/compliance** stakeholders care about regulatory risk and documentation. The staff DS provides written analysis referencing specific regulations (ECOA, GDPR, SR 11-7), with concrete evidence of compliance (fairness metrics, audit logs, model documentation). The meta-skill is *translation*, not simplification — the same underlying concept must be expressed in the vocabulary and framework that each audience uses to evaluate proposals.Question 17
What is "organizational debt" and how does it relate to technical debt in ML systems?
Answer
By analogy with technical debt (Sculley et al., 2015) — accumulated shortcuts in code, infrastructure, and architecture that slow future development — organizational debt is the accumulated cost of shortcuts in processes, documentation, knowledge sharing, and team development. Examples: undocumented tribal knowledge that leaves when people leave; inconsistent experiment analysis standards across teams; no design review process (decisions made via Slack without structured evaluation); no writing culture (lessons learned are lost); skill gaps that are worked around rather than addressed. Like technical debt, organizational debt compounds: each shortcut makes the next shortcut more likely and more costly to reverse. A team with high organizational debt moves slowly despite individual talent, because coordination costs dominate. The staff DS's role includes identifying, quantifying, and systematically reducing organizational debt — which often requires the same kind of unglamorous, sustained investment as paying down technical debt.Question 18
What distinguishes a "Shopping List" roadmap from a well-constructed roadmap?
Answer
A Shopping List roadmap lists every possible project without prioritization or sequencing — it is a backlog, not a roadmap. A well-constructed roadmap makes *choices*: it explicitly states what the team will do, in what order, and (implicitly) what it will *not* do. It sequences work based on dependencies (feature store improvements before models that depend on them), risk (high-uncertainty bets earlier when there is time to pivot), and expected value (highest-impact bets first). It also includes team gap analysis (acknowledging that the team may not currently have all needed skills), success metrics (measurable outcomes, not activities), and risk mitigations (what happens if a key assumption is wrong). A Shopping List creates the illusion of productivity by listing work; a roadmap creates the reality of progress by making strategic choices about where limited capacity should be invested.Question 19
A staff data scientist writes one imperfect blog post per month. Another plans a perfect post for six months and never finishes. Which approach creates more value, and why?
Answer
The one who publishes monthly creates more value — for three reasons. **(1) Cumulative output:** Twelve imperfect posts create a body of work that covers diverse topics, demonstrates sustained expertise, and provides organizational reference material. Zero perfect posts create nothing. **(2) The writing habit forces reflection:** Regular writing forces the author to organize their thoughts, identify gaps in their understanding, and articulate lessons learned — which improves technical judgment independent of whether anyone reads the post. **(3) Writing skill improves through practice:** Post #12 will be substantially better than post #1, because writing (like modeling) improves through iteration, not planning. The discipline of regular, imperfect publication is more valuable than the aspiration of irregular perfection. This principle applies broadly at staff level: shipping an imperfect RFC that starts a conversation is more valuable than planning a perfect RFC that never circulates.Question 20
Why is the statement "soft skills are hard skills" more than just a clever phrase in the context of staff-level data science?