Chapter 8: Regression to the Mean — Why Hot Streaks Cool Down

"The most dangerous moment in any endeavor is right after it's going brilliantly." — Dr. Yuki Tanaka

Opening Scene

Marcus has the deck open on his laptop, and he's been practicing the pitch for a week. The slides are good. The metrics are real.

"Look at this," he says, spinning the laptop toward Dr. Yuki. Three months of revenue data for his chess tutoring app: Month 1, $8,200. Month 2, $11,500. Month 3, $14,800. A clean upward staircase.

"I'm thinking about dropping to part-time school next semester," he says. "The trajectory is clear. If this keeps going, in six months I could hire someone. In a year, I could be full-time."

Dr. Yuki looks at the numbers. She looks at Marcus. She does the thing she always does when she's about to say something he might not want to hear — she takes a small breath first.

"Those are impressive numbers," she says. "I mean that. Three months of growth in a startup is not nothing."

"Right," Marcus says.

"And I want you to seriously consider that those three months might be the luckiest three months you'll ever have, and that Month 4 might look very different from what that trend line suggests."

Marcus sits back. "That's not — the app is good. I've improved the matching algorithm. I've been doing targeted outreach to chess clubs—"

"I believe you," Dr. Yuki says. "I believe all of that. And I'm still saying: three extraordinary months do not predict a fourth extraordinary month. Not because your app isn't good. Because of something called regression to the mean. And it is the most invisible trap in business, in sports, in medicine, and in life."

She reaches into her bag and pulls out a book — worn, clearly read many times. "Francis Galton figured out part of this in 1886," she says. "By measuring people's heights."

Marcus looks at the staircase on his laptop screen. "So you're saying Month 4 is going to be bad?"

"I'm saying Month 4 is going to be honest," Dr. Yuki says. "And right now, you don't know what that means yet."

She closes the laptop screen gently. "That's what we need to figure out. Together."

Galton's Discovery: The Tall Father's Shorter Children

Sir Francis Galton was a nineteenth-century polymath — cousin of Charles Darwin, pioneer of statistics, and a man of almost absurdly wide interests. In 1886, he published a paper with a puzzling finding: when he plotted the heights of parents against the heights of their adult children, the children of unusually tall fathers tended to be taller than average — but not as tall as their fathers. And the children of unusually short fathers tended to be shorter than average — but not as short as their fathers.

The children, in other words, were "regressing" toward the average of the population.

Galton called this "regression toward mediocrity." (We now use the less loaded term "regression to the mean.") He thought he had discovered something about heredity — that nature had a built-in tendency to pull extreme traits back toward the center.

He was right about the pattern but wrong about the mechanism.

The truth, which took another generation of statisticians to fully formalize, is simpler and more universal. Regression to the mean is not a biological force or a cosmic correction. It is a mathematical inevitability that arises whenever two things are imperfectly correlated and you measure extreme values on one.

Here is the key insight: Any measurement of a complex trait contains a component of luck (random variation) in addition to the true underlying value.

A person's height in one measurement isn't just their "true" genetic height. It also includes measurement error, time of day (we're slightly taller in the morning), posture, and many other random fluctuations. In height, these are small. In performance domains — athletic performance, academic testing, startup revenue, investment returns — they are enormous.

When you select people or outcomes at the extreme end of a distribution, you have by definition selected a group that had unusually favorable luck in addition to whatever underlying ability they possess. Their next measurement will likely show less extreme luck — which means their next result will typically be closer to the true mean.

This is regression to the mean. It requires no cosmic justice, no balancing force, no jinx. It requires only that measurements contain noise and that you selected based on extreme observed values.

Galton's original dataset was elegant in its design: he collected height measurements of 928 adult children born to 205 sets of parents, then created what he called a "mediocrity diagram" — essentially a scatter plot showing parental height against offspring height. The diagonal he expected (tall parents produce equally tall children) was tilted toward the center. Every two-inch excess in parental height above average was associated with only about one inch of excess in the children's heights. The ratio — later formalized as the regression coefficient — was approximately 0.5.

This coefficient of 0.5 tells you something about the relative role of luck in the system. If the coefficient were 1.0 — if children's heights perfectly predicted their parents' — luck plays no role and there would be no regression. If it were 0 — if children's heights were entirely unpredictable from parental heights — luck dominates entirely. At 0.5, height transmission is roughly half predictable by heredity and half explained by other factors including measurement variation.

The lesson generalizes: every correlation below 1.0 implies regression to the mean. And in most human performance domains, correlations are well below 1.0.

Why Regression to the Mean Happens Mathematically

Let's build this up from the ground.

Suppose performance in any domain can be modeled as:

Observed performance = True ability + Random luck

Where "random luck" is drawn from a distribution with mean zero (luck is equally likely to be positive or negative) and some variance.

When you observe a very high performance, you are observing a case where the sum of ability and luck was large. This could happen because: - Ability is high and luck is average - Ability is average and luck was unusually high - Ability is high and luck was also unusually high

The critical point: for the top observed performers, the third case (high ability, high luck simultaneously) is overrepresented relative to how common it is in the population. The top 5% of observed performers will, on average, have had better-than-average luck in addition to high ability.

When you then observe that same group again, their luck is now drawn fresh from the distribution. On average, it will be average — because that's what "average" means. But their ability is the same. So their second observed performance will be: same ability + average luck = lower than the first observation.

This is regression to the mean. It is not about ability declining. It is about the lucky component of the first extreme observation not being repeated.

The mathematics is captured by a simple formula. If the correlation between first and second performance is r (between 0 and 1), and the first performance was z standard deviations above the mean, the expected second performance will be r×z standard deviations above the mean.

If performance from one year to the next is perfectly correlated (r = 1) — if luck plays no role — there is no regression. The exceptional performer stays exceptional.

If performance from one year to the next is entirely uncorrelated (r = 0) — if it's entirely random — there is complete regression. Anyone who was exceptional last year will be average next year on average.

In reality, most performance domains have correlations between 0 and 1. The weaker the correlation (the more luck matters), the stronger the regression to the mean. Chess ratings are moderately correlated year-to-year (skill matters a lot). Startup revenue in early months is weakly correlated (luck, timing, and randomness matter enormously). Baseball batting averages are somewhere in between.

Let's run the numbers for Marcus's situation as a concrete worked example. Suppose month-to-month revenue correlation for early-stage consumer apps is approximately 0.4 — a reasonable estimate based on the high variance of new customer acquisition. His Month 3 revenue of $14,800 was, let's say, two standard deviations above the average for apps at his stage. The regression formula predicts his Month 4 expected performance at 0.4 × 2 = 0.8 standard deviations above the mean — still above average, but substantially less spectacular. If the mean for similar apps is around $7,000 per month and one standard deviation is $4,000, then 0.8 standard deviations above mean is about $10,200. Not bad — but a long way from the staircase extrapolation of $18,000 he was privately hoping for.

The model is rough. But the direction is almost certainly correct.

Myth vs. Reality: Regression to the Mean Edition

Myth: After a terrible performance, you should change your approach — something must be wrong. Reality: Terrible performances often regress to the mean even without any change. The regression is not feedback about your strategy; it's a statistical inevitability. This makes it very hard to tell whether an intervention actually helped or whether the situation would have improved anyway.

Myth: A three-month hot streak in my business means my new strategy is working. Reality: Extraordinary periods contain extraordinary luck. The new strategy may or may not be contributing. You cannot know from the hot streak alone, because regression to the mean will appear to "validate" any action you took at the peak.

Myth: Great rookie seasons predict continued greatness. Reality: Exceptional debut seasons are selected partly because of exceptional luck in addition to exceptional ability. The sophomore slump is, in significant part, regression to the mean — and survivorship bias (only the players with exceptional debuts are watched closely). (See Case Study 8.2.)

Myth: If I get a higher score on a test after changing my study habits, the study habit change must have worked. Reality: If you decided to change your study habits after a poor test score, regression to the mean predicts that the next score would likely be higher even without the change. This is one of the core reasons we need controlled comparisons in educational research.

The Illusion of Coaching and Intervention Effects

This is where regression to the mean becomes genuinely dangerous. It creates false beliefs about what causes improvement or decline — beliefs so powerful that they convinced military trainers that punishment works better than praise.

The setup: a performance that is unusually bad is followed by improvement. An observer attributes the improvement to the intervention that came after the bad performance. The intervention "worked."

The truth: the bad performance would have been followed by improvement anyway — because regression to the mean guarantees that an unusually bad result tends to be followed by a less bad result. The intervention didn't cause the improvement. The math did.

Daniel Kahneman, who won the Nobel Prize in Economics in 2002 for his work on cognitive biases, describes encountering this illusion in a flight instruction context so vividly that it's worth quoting at length. He was teaching Israeli flight instructors about cognitive biases when one of them made a remark that changed how Kahneman thought about this problem.

The instructor said: "I've noticed that when I praise a trainee after a particularly good maneuver, the next maneuver is typically worse. And when I yell at a trainee after a particularly bad maneuver, the next maneuver is typically better. So praise makes people worse and criticism makes people better. I've seen this hundreds of times."

Every other instructor in the room nodded. They had all "seen" the same thing.

Kahneman recognized the pure statistical mechanism immediately. The exceptionally good maneuver was exceptionally good partly because of luck. After praise, the next maneuver regresses to the mean — slightly worse. The exceptionally bad maneuver was exceptionally bad partly because of bad luck. After criticism, the next maneuver regresses to the mean — slightly better. The praise and criticism had nothing to do with it.

But the instructors had taught this lesson to themselves — praise hurts, punishment helps — thousands of times. And they were completely wrong. Not because they were bad observers, but because regression to the mean is so reliable and the intervention is so often timed to coincide with extreme performances that the false causal story is nearly irresistible.

This is why controlled experiments with comparison groups are so important: they let you see what would have happened without the intervention.

The flight instructor fallacy has a name: it is sometimes called the "regression trap" in intervention design. Its logic is worth spelling out in full, because the same trap shows up everywhere:

1. You observe an extreme performance (very bad).
2. You take an action (criticism, coaching, medication, strategy change).
3. The next performance is better.
4. You attribute the improvement to your action.
5. You are wrong. The improvement was coming anyway.

Step 5 is the one humans almost never reach on their own. We are story-seeking creatures, and cause → effect is the most satisfying story there is. When our action is followed by improvement, it genuinely feels like our action caused the improvement. The mechanism producing the improvement — regression toward a stable mean — is invisible. It produces no narrative. It is just math.

Research Spotlight: Kahneman's Flight Instructors

Kahneman, D. (2011). Thinking, Fast and Slow. Farrar, Straus and Giroux (Chapter 17, "Regression to the Mean").

Kahneman's account of the Israeli Air Force flight instructor episode has become one of the most-cited examples of how regression to the mean produces false beliefs about causation. The instructors were sophisticated professionals with extensive experience. They were not careless observers. And they had systematically learned the wrong lesson — that punishment outperforms praise — because the timing of their interventions was naturally correlated with extreme performances, and extreme performances naturally regress.

The broader implication: any time we evaluate the effectiveness of an intervention (a coaching change, a new drug, a business pivot, a study strategy) and we do not have a proper control group, we risk attributing natural regression to the intervention. This is why medical trials use control groups. It is why "A/B testing" exists in business. The alternative — before-and-after comparison without a control — almost always misleads.

Research Spotlight: The Regression Trap in Medicine

McDonald, C.J., Mazzuca, S.A., & McCabe, G.P. (1983). How much of the placebo "effect" is really statistical regression? Statistics in Medicine, 2(4), 417–427.

One of the most consequential applications of the regression-to-the-mean problem is in clinical medicine — particularly in the evaluation of treatments for conditions that fluctuate naturally, like chronic pain, blood pressure, depression, and anxiety.

Patients typically seek medical care when their symptoms are at their worst. A patient visits the doctor for back pain when the pain is unusually severe. They are prescribed a treatment. A week later, they return and report improvement. The doctor attributes the improvement to the treatment.

But there is a serious confound: pain that peaks at a clinical visit will often regress toward the patient's typical (lower) level in the following week regardless of treatment. The "improvement" may be statistical regression, not treatment effect.

This is one reason why the placebo effect is so hard to separate from regression to the mean in clinical trials: both produce apparent improvement after the worst moments. The randomized, placebo-controlled trial is specifically designed to separate these effects — you need to see that the treated group improves more than the untreated control group, not just that the treated group improves at all.

The lesson is universal: improvement after intervention, on its own, tells you almost nothing about whether the intervention caused the improvement.

Sports: Hot Streaks and the Regression Trap

Professional sports is one of the richest environments for observing regression to the mean, because performance is measured precisely and frequently, and the human appetite for narrative is enormous.

The batting average roller coaster:

A baseball player who hits .380 in the first two months of a season (well above the typical .260-.280 range) will almost certainly "slump" in the second half of the season. Commentators will speculate about mechanical changes, psychological pressure, or fatigue. Managers might adjust lineups or strategies. And the hitter will likely end the season somewhere closer to their true talent level — perhaps .310 or .320.

Was there a slump? Technically, yes — performance declined. Was there a cause for the slump in the ordinary sense? Often, no. The first-half numbers were inflated by luck. The second half brought regression. The "slump" is largely the regression component of the first-half performance paying its debt.

This phenomenon is so consistent in baseball that analysts have a name for it: BABIP, or Batting Average on Balls in Play. A player's BABIP in any given stretch reflects both their skill at making contact and an enormous amount of luck — whether the ball happened to land where a fielder wasn't, the exact angle of a deflection, the timing of a defensive shift. When a player's BABIP is very high over two months, the analytical community treats this as a strong signal that their batting average will regress, regardless of what the player or their coaches do.

The newly-acquired star:

A team trades for a player who just had a breakout year — career-high home runs, exceptional on-base percentage, dominant stats across the board. They pay a premium. The next year, the player is good but not spectacular.

Did the trade fail? Did the new environment hurt the player? Or was the breakout year partly lucky, and regression inevitable?

The answer matters enormously for how teams should evaluate trades. And the research on this is fairly clear: exceptional seasons are partially the product of exceptional luck, and regression to the mean is a more common explanation for subsequent "disappointment" than changes in underlying ability.

The data from major league baseball's player contract market supports this: players who sign large free-agent contracts after career-best seasons — which are, by definition, selected as peaks — tend to outperform the contract in the first year less frequently than players who sign after typical seasons. The peak season inflates the contract value; regression to the mean deflates the subsequent performance.

The coaching hire:

A team hires a new coach after a terrible season. The following season, they improve dramatically. The coach is praised for the turnaround. Is the praise warranted?

Maybe. But before attributing the improvement to the coach, you must ask: what would have happened without the coaching change? Teams that have bad seasons tend to regress to the mean anyway — the terrible season contained unusual bad luck, and that bad luck tends not to repeat. Coaches hired after terrible seasons benefit from regression even if they do nothing to change the team's true quality.

This doesn't mean coaching doesn't matter — it often does. It means we systematically overestimate coaching effects because we don't account for what would have happened without the change.

Sports analysts have a specific version of this insight: "bad team hires coach, gets better" is only meaningful if you compare the team to others who had equally bad seasons but didn't hire a new coach. In fact, the average team that has an unusually bad season improves the following year regardless of coaching changes — because the bad season was partly bad luck.

NBA performance and the regression pattern:

In basketball, similar patterns emerge in striking fashion. Players who lead the league in a specific advanced metric during one season — true shooting percentage, defensive rating, three-point percentage — regress substantially toward the mean the following season. The degree of regression is inversely proportional to the stability of the underlying skill: free-throw percentage, which is largely a true skill, regresses less. Three-point percentage over a limited sample, which has a larger luck component, regresses more.

The "hot hand" debate in basketball is directly related: researchers examining whether players who make several shots in a row are genuinely more likely to make the next shot have found much weaker evidence for the hot hand than the basketball intuition suggests. More often, the apparent streak is the random clustering that characterizes any probabilistic process — and the "cooling off" afterward is regression to the mean, not a curse.

Business: The Exceptional Quarter Problem

Marcus's situation — three great months followed by big strategic decisions — is one of the most common business traps. Here is why it's dangerous.

Early-stage startups are characterized by extremely high variance. Revenue in any given month depends on a small number of customers, deals that close or don't close in a given period, timing effects, word-of-mouth that suddenly ignites or stalls, and dozens of factors outside the founder's control. Month-to-month correlation in revenue is low when the sample is small.

In this environment, three exceptional months in a row means one of two things: 1. The company has genuinely found product-market fit and has a real underlying growth rate 2. The company has had three lucky months in a high-variance environment, and regression is coming

From inside the three exceptional months, these two situations can be nearly impossible to distinguish. The startup that has found genuine product-market fit looks exactly like the startup that got lucky — until the fourth month, when the trajectories diverge.

The dangerous response to exceptional months is to: - Over-hire: Staff up for the growth rate shown in lucky months, then face the costs when revenue regresses - Over-spend: Lock in fixed costs (office space, subscriptions, salaries) based on exceptional revenue - Under-invest in understanding: Stop asking hard questions about the underlying drivers of growth, because the numbers are so good they feel self-explanatory - Make irreversible strategic pivots: Change direction based on what worked in the hot period, potentially abandoning things that work in normal periods

The experienced investor or operator knows this. They look for evidence of the underlying growth rate, not just the recent exceptional observations. They ask about customer retention (which is harder to luck into than new sales), about cohort-by-cohort performance, about the mechanisms of growth. They apply the law of large numbers (Chapter 7) and ask for more time and more data before drawing conclusions from a hot streak.

This insight is codified in modern startup methodology. The concept of "cohort analysis" — tracking user groups over time rather than looking at aggregate monthly numbers — exists precisely to see through regression-to-the-mean noise in aggregate metrics. If Marcus's Month 3 peak came from a single large batch of new customers who all signed up through one referral channel, a cohort view would reveal whether those customers stayed and paid month after month (evidence of real underlying value) or churned immediately (evidence that the month was lucky rather than structurally sound).

The venture capital world has a version of this wisdom too: professional investors who see many early-stage startups have learned — through painful experience — not to invest based on exceptional recent metrics without understanding the underlying drivers. A seed-stage company showing hockey-stick growth over three months is exciting. It is also potentially just three months of noise. The experienced investor asks for the cohort data, the retention curve, the source-level breakdown of new users. They are trying to separate the true underlying growth signal from the regression-prone noise.

Research Spotlight: Regression to the Mean in Investment Returns

Carhart, M.M. (1997). On persistence in mutual fund performance. Journal of Finance, 52(1), 57–82.

Mark Carhart's landmark study examined whether the performance of mutual funds persists from year to year — a question directly bearing on whether fund managers exhibit genuine skill that produces lasting above-market returns.

The finding: past performance does predict future performance — but only weakly, and largely in the wrong direction for investors seeking to pick winners. The funds that performed best in one year tended to be the same funds that underperformed in subsequent years. And the pattern was cleaner than most expected: each year's top decile of performers reverted substantially toward the mean in the following year.

Carhart's explanation combined regression to the mean (the lucky component of exceptional performance doesn't persist) with momentum effects (some persistence exists, but it's short-lived and eroded by transaction costs). The upshot: selecting mutual funds based on their recent exceptional performance is one of the most reliable ways to buy into regression.

This is why financial regulators in many countries require the disclaimer "past performance is not indicative of future results." It is not legal boilerplate. It is a direct acknowledgment of regression to the mean in financial returns.

Social Media: Viral Videos and the Silence After

Nadia knows this feeling, though she wouldn't have named it regression to the mean.

A video goes unexpectedly viral — 400,000 views when her typical video gets 3,000. She studies everything about it: the hook, the format, the time she posted, the topic, the hashtags. She makes five more videos with the same formula. They get 2,800 views. 3,100 views. 2,500 views. 3,400 views. Back to normal.

What happened?

The viral video contained, almost certainly, a substantial luck component. Something about that specific video, in that specific moment, connected with the algorithm in a way she couldn't have fully engineered — a share from someone with a large following, a momentary void in the trending topic, a quirk of how the algorithm distributed it on that particular day. The formula she extracted from the video captured the non-luck components — the parts that were repeatable — but couldn't recreate the luck component.

Regression to the mean in social media has a cruel feature: it looks like confirmation that the creator "doesn't know what they're doing," when in fact the viral video was as close to her true level as anything she's ever made — it was just also lucky.

The practical implication: treat viral outliers with the same skepticism you treat non-viral outliers. If a video dramatically underperforms, it might be bad — or it might be unlucky. If a video dramatically overperforms, it might be exceptional — or it might be lucky. The response to either extreme should be to look at the pattern across many videos (law of large numbers), not to read extraordinary meaning into any single result.

Nadia's instinct after the viral moment — to study every element of that video and replicate it — is an extremely natural human response to an extreme positive event. It is also, unfortunately, nearly guaranteed to disappoint. The features she identified in the viral video (the hook, the format, the posting time, the topic) are the things she can see and reproduce. The luck component — the specific algorithmic conditions, the shares, the timing — is invisible and non-reproducible.

This is the double trap of social media regression: first, you falsely attribute the viral peak to your strategy. Then, when the strategy fails to reproduce the results, you conclude that you must have gotten some element wrong — maybe the hook, maybe the audio, maybe the thumbnail. You iterate. You try more combinations. Some of them will slightly outperform your baseline by chance, and you'll add those features to your "strategy." Meanwhile, you're optimizing around noise.

The cleaner approach, which is harder emotionally but more accurate, is to treat a single viral video as a data point that suggests your content has the capacity to connect with large audiences — and then build systematically from the baseline, measuring improvement across groups of videos rather than chasing the reproduction of a lucky moment.

Lucky Break or Earned Win?

Marcus's Three-Month Hot Streak

Marcus's chess tutoring app had revenues of $8,200 / $11,500 / $14,800 across three months — a perfect staircase. He's considering dropping to part-time school.

Apply regression to the mean to this situation. The three months almost certainly contained some luck: a few unusually large contracts, a viral chess content moment that drove downloads, a referral chain that briefly accelerated. None of this makes the growth fake. But all of it makes the question "what's the true underlying growth rate?" very hard to answer from three data points.

What should Marcus do? Not abandon the startup — the three months are evidence worth taking seriously. But the questions he needs to answer are: What is the month-to-month retention of paying users? Is the growth coming from repeatable channels or one-time events? What does Month 4 look like before making irreversible decisions?

The hot streak is real. Whether it's telling him his true growth rate is a different question entirely.

How to Recognize Regression to the Mean vs. Genuine Decline

This is the practical challenge: when performance drops after a peak, is it regression or decline? The distinction matters enormously for what you should do.

Signs the drop is regression to the mean: - The initial peak was at an extreme — dramatically above the historical baseline - There was no identifiable structural change before the peak (nothing specifically changed that would explain why performance suddenly became so much better) - The drop brings performance back toward the historical average, not below it - The same pattern applies to comparison groups who had no intervention

Signs the drop may be genuine decline: - Performance was stable for a long time before declining - The decline continues well below the historical average - There is an identifiable structural cause (injury, competitor disruption, changed market conditions) - Comparison groups without the intervention don't show the same pattern

In practice, these signals overlap and are difficult to disentangle with small samples (which brings us back to Chapter 7). The epistemic tool for separating them is the comparison group: find people or outcomes at a similar level who experienced different things, and see if they show the same pattern. If the regression happens regardless of what you do, it's the math. If it happens only for those who received the intervention, it might be the intervention.

There's also a useful heuristic from the world of quantitative sports analysis: examine whether the performance drop is symmetric. If the high-water mark was 40% above the historical average and the subsequent performance settled at around the historical average, that's regression-shaped. If the performance dropped to 30% below the historical average and stayed there, that looks more like genuine decline.

This asymmetry check works because genuine regression to the mean produces a predictable shape: performance returns toward the center, not below it. A bounce from the peak followed by a stabilization near the baseline is the regression signature. A bounce from the peak followed by continued deterioration is a different story.

The Python Simulation: Seeing Regression to the Mean in Data

One of the most powerful ways to internalize regression to the mean is to generate it yourself in code. The following simulation produces a synthetic dataset that demonstrates the phenomenon cleanly.

import random
import statistics

def simulate_regression_to_mean(
    n_people=1000,
    true_mean=100,
    true_sd=15,
    luck_sd=10,
    top_n=50
):
    """
    Simulate regression to the mean in a performance context.

    Each 'person' has a true ability (stable) and a luck component
    that varies between observations. We select the top performers
    from round 1 and observe whether they are still top performers
    in round 2.
    """
    # Generate true abilities for each person (these don't change)
    true_abilities = [random.gauss(true_mean, true_sd) for _ in range(n_people)]

    # Round 1: observed performance = true ability + luck
    round_1 = [
        ability + random.gauss(0, luck_sd)
        for ability in true_abilities
    ]

    # Round 2: same true abilities, new luck draw
    round_2 = [
        ability + random.gauss(0, luck_sd)
        for ability in true_abilities
    ]

    # Find the top performers in Round 1
    indexed_round_1 = sorted(enumerate(round_1), key=lambda x: x[1], reverse=True)
    top_indices = [idx for idx, score in indexed_round_1[:top_n]]

    # Calculate their average in Round 1 and Round 2
    top_round_1_scores = [round_1[i] for i in top_indices]
    top_round_2_scores = [round_2[i] for i in top_indices]

    avg_round_1 = statistics.mean(top_round_1_scores)
    avg_round_2 = statistics.mean(top_round_2_scores)

    print(f"Overall population mean: {statistics.mean(round_1):.1f}")
    print(f"Top {top_n} average in Round 1: {avg_round_1:.1f}")
    print(f"Same group average in Round 2: {avg_round_2:.1f}")
    print(f"Regression amount: {avg_round_1 - avg_round_2:.1f} points")
    print(f"Still above overall mean: {avg_round_2 > statistics.mean(round_1)}")

# Run the simulation
simulate_regression_to_mean()

Running this code will typically produce output showing that the top 50 performers — who averaged perhaps 125-130 in Round 1 — average only 110-115 in Round 2. They are still above the population mean (their true ability is genuinely higher), but they are meaningfully lower than their Round 1 peak. The regression is automatic — no one did anything differently. The true abilities didn't change. The luck component simply drew from the center of its distribution, as it always does on average.

This is what Dr. Yuki means when she calls it a mathematical inevitability. The code doesn't have a "cosmic correction" subroutine. It simply adds a new random number each round, and the distribution does the rest.

You can modify the luck_sd parameter to see how regression strength depends on luck magnitude. Set it to 0 (no luck) and regression disappears completely: the top performers in Round 1 are the top performers in Round 2, in the same order. Increase it to 20 (very high luck) and the regression becomes nearly complete: the top performers in Round 1 are barely distinguishable from the rest of the population in Round 2.

A Return to Marcus: Month 4

Three weeks after the coffee meeting, Marcus texts Dr. Yuki a screenshot.

Month 4 revenue: $9,800.

She texts back: "How are you feeling about it?"

A pause. Then: "Honestly? Better than I expected. I spent the past three weeks building the cohort analysis like you suggested. The retention numbers are actually good. The Month 2 and Month 3 users are still paying. The Month 4 dip is mostly acquisition — the referral chain from Month 3 didn't keep going."

"Which tells you what?" she writes.

Another pause, longer this time.

"That the product is worth paying for once people try it. The problem is finding the next referral chain. That's where the luck comes in."

She texts back a single word: "Exactly."

Because that's the thing about understanding regression to the mean — it doesn't tell you that the business isn't good. It tells you where the real information lives. Month 4's $9,800 is not a disaster. It is an honest data point. And an honest data point, correctly interpreted, is worth infinitely more than a misleading trend line.

Marcus didn't drop to part-time school. He built a better acquisition model instead. The decision was harder to make — no one fires themselves up by saying "I should remain cautious until I have better statistical evidence." But it was the right decision, informed by the right understanding of what those first three months actually meant.

The Deeper Lesson: Don't Change Strategy After Extreme Outcomes

The most dangerous practical consequence of misunderstanding regression to the mean is that it leads to inappropriate strategy changes.

When performance is unusually bad, we change strategy. When performance is unusually good, we double down on current strategy. But if the extreme performance was partly luck, both responses are based on noise. We're changing strategy in response to randomness, not in response to meaningful feedback.

The better framework is to make strategic evaluations based on: 1. Average performance over a long period (not recent extremes) 2. Leading indicators that predict future performance (not just lagging outcomes) 3. Counterfactual reasoning (what would have happened without this strategy?) 4. Comparison groups that weren't exposed to your strategy

If Marcus drops to part-time school based on three great months and Month 4 regresses to the mean, he will have made an irreversible decision based on misleading data. Dr. Yuki isn't saying the startup isn't good. She's saying that three months is not enough to know whether the staircase is the underlying reality or the lucky extreme.

The prepared mind understands regression to the mean not as a discouragement but as a form of protective wisdom. Know that peaks are partly luck. Know that troughs are partly luck too. Evaluate yourself and your strategies across the distribution, not at its extremes.

This insight also applies to how you evaluate yourself. If you got an exceptional grade on one exam, that grade contains your true ability plus luck. The next exam will likely feel harder — not because you got worse, but because the first exam was partly a lucky day. Similarly, if you bombed a presentation but normally present well, the next presentation will likely go better — not because your skills improved, but because the bad one was partly an unlucky day.

Knowing this frees you from two traps simultaneously: from overconfidence after peaks, and from despair after troughs. The distribution is more stable than the extremes suggest. Your true ability is more consistent than your best day or your worst day implies. And both of those facts, properly understood, are genuinely useful.

Regression to the Mean Across Life Domains

The phenomenon is not limited to business and sports. Consider how it appears in contexts that touch everyone's life:

Academic performance: Students who score unusually well on one standardized test tend to score somewhat lower on the next sitting, and students who score unusually poorly tend to score somewhat higher. Test-prep companies that work with low scorers can falsely attribute score improvements to their curricula when regression to the mean would have produced much of that improvement regardless. This has been demonstrated in studies of SAT prep effects that include appropriate control groups.

Health and wellness: When patients with chronic conditions enter treatment, they often do so at a moment of crisis — when symptoms are at their worst. Treatment of almost any kind, including placebo treatment, tends to be followed by improvement, because the crisis was a peak of bad luck that regresses. This is one reason why "before and after" comparisons in wellness contexts (supplement marketing, wellness programs, dietary interventions) are so unreliable without controlled comparison groups.

Relationships: After a particularly heated argument — an extreme negative event — couples often feel closer and communicate better for a period. Some attribute this to "clearing the air." It may also be regression: the heated argument was partly driven by an unusual confluence of stress, fatigue, and bad luck, and the subsequent improvement is the relationship returning toward its stable baseline, not a consequence of the argument itself.

Mental health: Good days and bad days in depression and anxiety are partly expressions of regression to the mean around a chronic underlying level. People sometimes credit or blame specific events — a conversation they had, something they ate, a change in their routine — for shifts that were going to happen anyway because an extreme bad day was partly bad luck. This does not mean interventions don't work. It means before-and-after comparisons without control groups cannot tell you whether they work.

Regression to the Mean in Education and Self-Assessment

Students encounter regression to the mean constantly, and almost never recognize it.

Consider standardized tests. A student who prepares extensively for the SAT, takes a practice test, and performs unusually well (a lucky day — well-rested, familiar questions, good guessing luck) will almost certainly score lower on the next practice test without changing anything about their preparation. They may conclude that their preparation "peaked" and they're now "declining." Neither is true. The first score was extreme. The second score is more representative.

This misread runs in both directions. A student who performs unusually badly on one test — a bad night's sleep, an anxious morning — and then improves on the next test tends to attribute the improvement to some change they made: the new study technique, the earlier bedtime, the pre-test ritual. Any of these might have helped. But even without any change, improvement was expected — because the first score was unusually bad and was going to regress.

The result is that students accumulate false beliefs about what works and what doesn't work for their performance, built entirely on regression-shaped data. The study strategy they adopted after a bad test gets credit for the improvement that regression was going to deliver anyway. The strategy they dropped after a good test gets blamed for a subsequent decline that regression was going to produce regardless.

A more accurate approach is to evaluate study strategies across many tests — not based on the one that happened to follow a peak or a trough. And to recognize that single-test scores, especially at the extremes, contain substantial luck and should be interpreted with appropriate skepticism.

This applies equally to grades on individual assignments, to athletic performances in practice, to presentations at work, and to any other measured outcome that contains both a skill component and a luck component. The assignment you thought was your worst might just be one of your unlucky days. The presentation you thought was your best might just be one of your lucky ones. Neither is a reliable signal about what you're truly capable of.

Dr. Yuki puts it this way: "The best athletes don't celebrate their best practice sessions and panic after their worst ones. They watch the trend across many sessions. That's the only place where the signal lives — in the average of many observations, not in the extremes."

Research Spotlight: The Sports Illustrated Jinx Revisited

Schall, T. & Smith, G. (2000). Do baseball players regress toward the mean? The American Statistician, 54(4), 231–235.

The "Sports Illustrated jinx" is a piece of sports folklore: players featured on the cover of Sports Illustrated are said to suffer a decline in performance afterward. This has been attributed to everything from distraction to hubris to supernatural bad luck.

The statistical explanation is simpler and more mundane. Players appear on the Sports Illustrated cover after extraordinary performances — career bests, record-setting seasons, exceptional runs. Extraordinary performances, as we now know, contain extraordinary luck components. The performance after the cover appearance regresses toward the player's true mean — as it would have regardless of whether they appeared on any magazine cover.

Schall and Smith examined this specifically in baseball data and confirmed what regression to the mean predicts: players who appeared on the cover after exceptional periods did tend to perform worse afterward. But the same pattern held for equally exceptional players who did not appear on the cover. The jinx was the math, not the magazine.

This matters beyond sports trivia. Every time a person, team, or organization is celebrated at the peak of their performance and then "falls" — the post-celebration "jinx" narrative is almost always regression to the mean, not causation from the celebration. Companies that win "Best Place to Work" awards tend to show slightly less exceptional results in subsequent years. Athletes who sign lucrative contracts after career-best seasons tend to perform at their true level — lower — in the next season. The celebration coincides with the peak; the regression follows; the two are correlated but not causally connected.

Regression to the Mean and the Feedback Loop Problem

There is a particularly vicious trap that regression to the mean creates in feedback systems — places where your performance influences your future circumstances, which in turn influences your future performance.

Imagine a student who performs exceptionally well on an entrance exam (partly luck) and is placed in an advanced program. The advanced program provides better instruction, better peers, and more challenge — all of which genuinely improve the student's future performance. The student thrives.

Now imagine the same student had a bad day on the exam (bad luck) and was placed in a standard program. The feedback loop runs in the other direction.

The initial placement was influenced by regression-prone extreme performance. But the downstream consequences — the educational environment, the peer group, the opportunities — are real and lasting. Regression to the mean at the measurement stage has amplified into a genuine performance difference over time.

This is one of the mechanisms through which luck at critical junctures (exam days, auditions, job interviews) can have effects that look like skill differences but are partly the product of which way the luck ran at one pivotal moment.

Understanding this doesn't make the system fairer. But it should make us humble about interpreting performance gaps as purely reflecting underlying ability — and it should make us cautious about treating single high-stakes measurements as definitive.

The Luck Ledger

What this chapter gave you: Regression to the mean is the mathematical inevitability that extreme observations — the best months, the worst test scores, the hot-hand streaks, the career-low performances — tend to be followed by less extreme ones. It happens because extreme observed values contain an unusually large (or small) luck component, and luck does not persist in the same direction indefinitely. This creates systematic illusions: that interventions work, that hot streaks are sustainable, that viral successes can be formulaically repeated.

What's still uncertain: In any specific case, how much of an exceptional period is skill/structural and how much is luck? The answer requires more data, better comparison groups, and clearer understanding of the mechanisms driving performance. We are never fully certain. But we can be much more calibrated — much less likely to be fooled by regression in both directions.

Chapter Summary

• Regression to the mean was discovered by Francis Galton studying parents' and children's heights. The mechanism is not hereditary pull toward mediocrity but the mathematical behavior of any variable that contains both a true-value component and a random-luck component.
• When you select observations at extreme values, you select observations with unusually favorable or unfavorable luck. The next observation is likely to show more average luck — and thus a result closer to the true mean.
• Regression creates the illusion of intervention effects: things that improve after a bad spell seem to respond to whatever you did, even when the improvement would have happened anyway.
• In sports, business, and social media, hot streaks are followed by cooling not because something went wrong but because the luck component of the hot streak regresses.
• The practical response: evaluate performance over long periods and multiple data points, not in the wake of extremes. Comparison groups are essential for disentangling regression from genuine causal effects.
• The most dangerous mistake is making irreversible decisions during hot streaks based on the assumption that the streak represents the new baseline.
• Regression to the mean affects academic performance, health outcomes, relationships, and mental health — anywhere performance fluctuates around a stable true level, the math of regression operates.
• The Sports Illustrated jinx, the post-award decline, and the "celebration curse" are all regression to the mean in disguise: the celebration coincides with the peak, and the subsequent regression looks like a consequence of the recognition rather than a mathematical inevitability.
• Students misread regression constantly: study changes adopted after bad test scores get credit for improvements that regression would have delivered anyway; strategies dropped after good scores get blamed for declines that regression was always going to produce.
• Python simulation lets you see regression emerge from the data itself: select the top performers, observe them again, and watch the math bring them closer to center — automatically, inevitably, without any cosmic force involved.