Chapter 32: Reliability, Testing, and Why Rockets Fail

"For a successful technology, reality must take precedence over public relations, for Nature cannot be fooled." — Richard P. Feynman, Appendix F to the Report of the Presidential Commission on the Space Shuttle Challenger Accident (1986)

Overview

Every other chapter in this book has taught you how to make something work. This one is about the far harder problem of making sure it works when it has to, the first time, with no one aboard able to fix it and no way to call it back. It is the chapter where theme #2 of this whole book — space is an unforgiving environment; everything must work — stops being a slogan in the margin and becomes arithmetic you can compute, a table you can fill in, and a test you can run.

Consider what is different about a rocket. When your laptop crashes you reboot it. When a car's engine misbehaves you pull over. When a bridge shows a crack you close a lane and send a crew. A spacecraft offers none of these mercies. The moment the first-stage engines light, a launch vehicle becomes a controlled explosion that must proceed, in the correct order, for eight or nine minutes, with hundreds of things going right and not one catastrophic thing going wrong — and then the payload it carries must operate, untouched by human hands, for years, in vacuum and radiation and temperature swings of hundreds of degrees, at the end of a radio link too weak and too slow to micromanage. There is no roadside, no reboot that helps, no repair truck. The heat shield of Chapter 7 must ablate correctly the first time it meets the plasma, because there is no second re-entry. This is what engineers mean when they say a mission has "no second chances," and it changes the entire character of the engineering. You are not designing for average performance; you are designing against the tail of a probability distribution.

That word — probability — is the key that unlocks the chapter. Reliability is not a feeling or a promise; it is a number between 0 and 1, and it obeys laws you can calculate with. Once you can compute it, you can engineer it: you can find the weakest link, decide where a backup is worth its mass, prove by test that a design has margin, and put a defensible figure on the risk you are asking a mission to accept. And once you can compute it, the great failures of spaceflight history stop being tragedies you read about and become case studies you can dissect — Ariane 5 destroyed by a line of reused software, Mars Climate Orbiter lost to a units mismatch, the Soviet N-1 defeated four times by an untested cluster of engines. Each one failed for a reason the mathematics of this chapter would have flagged.

In this chapter, you will learn to:

• See a mission as a chain of things that must all work, and treat reliability as a computable probability.
• Multiply series reliabilities, add redundancy, and see why one backup on the weakest link is worth more than gold-plating the strong ones.
• Hunt for single points of failure with an FMEA, and understand when a backup does not help.
• Tell qualification testing from acceptance testing, and apply "test like you fly" — the rule that space punishes any gap between how you tested and how you fly.
• Read real failures as root-cause investigations, and price reliability against margin, mass, and cost.

Learning Paths

🚀 Space Enthusiast: Read 32.1 for the mindset, then 32.2 for the one piece of math that will change how you watch a launch (series reliability), and 32.5 for the stories behind the equations. The Ariane 5 and N-1 cases are the two that stay with you.

📐 Engineering Student: Read everything and do the series/redundancy calculation of 32.2 and the FMEA-lite of the Checkpoint by hand. Reliability block math and FMEA are standard tools you will be expected to use; the ⭐⭐/⭐⭐⭐ exercises here are the practice.

🎮 KSP Player: You have felt this every time a booster RUD'd on the pad. Focus on 32.2 (why more engines is a double-edged sword) and 32.4 (why you revert-to-launch and test a design before you commit a career to it). The Falcon 9 engine-out case study is your favorite trade, quantified.

🛰️ Industry Prep: This is the language of every readiness review. Master the qualification-vs- acceptance distinction (32.4), the FMEA process (32.3), and the reliability-allocation idea of 32.6 and the Checkpoint. Add a real risk assessment to your Mission Design Review here.

32.1 Engineering for no second chances

Start with the physical reality, because the mindset follows from it. A rocket in flight cannot stop. Once the propellant is burning, the vehicle is committed to a trajectory that gravity and aerodynamics will not pause while an engineer thinks. A probe on its way to Mars is, after its launch window closes, on a ballistic path that only its own tiny thrusters can nudge, across a gap where a radio command takes many minutes to arrive (a latency we met in Chapter 31). A satellite in geostationary orbit is 36,000 km beyond the reach of any wrench. The defining constraint of spaceflight engineering is not power, or mass, or even the tyranny of the rocket equation — it is the absence of a second chance. Everything else in this chapter is a disciplined response to that one fact.

Definition (reliability). The reliability $R$ of a system is the probability that it performs its intended function, under stated conditions, for a stated period of time (or a stated event). It is a number in $[0,1]$. Three parts of that definition carry all the weight: an explicit function (what counts as success), explicit conditions (the environment it must survive), and an explicit duration (a single launch, a 15-year service life, one re-entry). A reliability quoted without all three is meaningless — "reliable" is an adjective; "$0.97$ probability of delivering the payload to GTO on a single flight" is an engineering requirement you can design toward and verify.

Because reliability is a probability, it behaves like one. Over time, most hardware follows a characteristic bathtub curve of failure rate: high at the very start (infant mortality — the manufacturing defects, cold-solder joints, and contamination that reveal themselves quickly), then low and roughly constant through the useful life (random failures, striking unpredictably), then rising again at the end as parts wear out. The two ends are the reason for two of this chapter's central practices: we screen out infant mortality with acceptance testing and burn-in (32.4), and we keep missions inside the flat middle by derating parts and retiring them before wear-out (32.6).

In that flat middle, where failures arrive at random with a constant rate $\lambda$ (failures per hour), a standard model gives the probability a unit is still working after time $t$:

$$ R(t) = e^{-\lambda t}, \qquad \text{MTBF} = \frac{1}{\lambda}, $$

where the mean time between failures (MTBF) is just the reciprocal of the failure rate. This exponential law is the same mathematics as radioactive decay, and it delivers the same sobering message.

Worked Example: even a good part is not a sure thing. Suppose a spacecraft avionics box has an MTBF of $1{,}000{,}000\ \text{hours}$ — an average life of about 114 years, which sounds indestructible. Ask for its reliability over a 15-year mission. First convert: $15\ \text{yr} \times 8{,}766\ \text{hr/yr} = 131{,}490\ \text{hr}$, so $\lambda t = 131{,}490 / 1{,}000{,}000 = 0.1315$. Then $$R = e^{-0.1315} = 0.877.$$ A part that lasts a century on average has only about an 88% chance of surviving a single 15-year mission — a one-in-eight chance of failing. And a spacecraft does not contain one such part; it contains hundreds. That is the problem the rest of this chapter exists to solve. (The MTBF here is illustrative, Tier 3; real space-grade parts are derated to push effective MTBF far higher — precisely because of this calculation.)

The historical record makes the point in the other direction. Orbital launch, across the whole industry and all of history, succeeds something like 90–95% of the time — which sounds high until you remember it means one flight in ten to twenty ends in loss, a failure rate no airline or automaker could survive. New vehicles do worse: a rocket's first several flights are its own infant-mortality period, which is why the maiden flight of any launcher is watched with held breath (theme #6 — the history of rocketry is in large part a history of learning reliability the hard way).

🔗 Connection: Notice how reliability reframes the margin you met in Chapter 29, §29.4. There, margin was reserve against uncertainty — extra delta-v, extra mass capability, extra power. Here we see why that reserve is survival, not caution: the environment is unforgiving, the estimate is a distribution rather than a number, and there is no pulling over to top up. Margin, redundancy, and testing are the three tools this chapter gives you, and all three are responses to the same "no second chances" reality.

🔄 Check Your Understanding 1. Why is "the spacecraft shall be reliable" not a usable requirement? What three things must a reliability statement specify? 2. A component's reliability over a 5-year mission is quoted as $0.95$. What does that number literally mean?

Answers

1. Because "reliable" is unverifiable — there is no test that says yes or no. A usable statement gives a function (what success is), conditions (the environment), and a duration (or event), e.g. "≥ 0.95 probability of full instrument operation in the GEO radiation environment over 5 years." 2. It means: if you flew a large number of identical components under the stated conditions, about 95% would still be performing their function at the 5-year mark — equivalently, any one unit has a 5% probability of having failed by then.

32.2 Reliability math: series systems and redundancy

Here is the single most important idea in the chapter, and it is arithmetic a middle-schooler can do — which makes its consequences all the more brutal. Almost every function on a spacecraft depends on a chain of things that must all work: the power must flow and the computer must run and the thruster must fire. In reliability language, those functions are in series, and for independent components in series the reliabilities multiply.

Strategy first. We are going to treat each component's success as an independent event with its own probability, and combine them with the ordinary rules of probability. Series (all required) → multiply the reliabilities. Parallel/redundant (any one suffices) → multiply the unreliabilities and subtract from one. Everything else in reliability engineering is built from these two moves.

$$ R_{\text{series}} = \prod_{i=1}^{n} R_i = R_1 R_2 \cdots R_n. $$

Because every $R_i \le 1$, multiplying them can only drive the product down. A series system is always less reliable than its worst component — the chain is, quite literally, weaker than its weakest link. And the effect compounds viciously with the number of links.

🚪 Threshold Concept: "everything must work" is arithmetic, not attitude. Take a system of 10,000 components in series, each superb — reliability $0.9999$, meaning each fails only once in ten thousand missions. The system reliability is $$R = 0.9999^{10{,}000} = e^{10{,}000 \ln 0.9999} \approx e^{-1.0} \approx 0.37.$$ Ten thousand excellent parts in series give a system that fails almost two times in three. This is why "everything must work" is not a motivational poster; it is a mathematical death sentence for any design that piles up long chains of even very good parts. Every serious reliability strategy — cut the part count, break the series with redundancy, push each $R_i$ towards 1 with margin and testing — is an attack on this one equation. Once you have seen it, you understand why a spacecraft is designed to be as simple as it can be and no simpler: every part you add is another factor less than one.

The escape from the series trap is redundancy: providing more than one independent way to accomplish a function, so the function survives the loss of one path.

Definition (redundancy). Redundancy is the provision of two or more independent means of performing a function, arranged so that the function is still accomplished if one (or more) of them fails. In active (parallel) redundancy the units all operate at once; in standby redundancy a backup is switched in when the primary fails. Redundant units are in parallel: the function fails only if every path fails.

For $n$ independent parallel units, the system fails only when all $n$ fail, so we multiply the failure probabilities $Q_i = 1 - R_i$:

$$ R_{\text{parallel}} = 1 - \prod_{i=1}^{n}(1 - R_i), \qquad\text{and for } n \text{ identical units } R_{\text{parallel}} = 1 - (1-R)^n. $$

Redundancy works the series trap in reverse: where multiplying reliabilities drags the product down, multiplying unreliabilities drives them down, pushing the parallel reliability up toward 1. Two units that each fail 10% of the time fail together only $0.10 \times 0.10 = 1\%$ of the time — so the pair is 99% reliable. That is the whole magic of a backup, in one line.

Here are the two structures as reliability block diagrams — the picture a systems engineer draws before writing a single number:

  SERIES  (every block required):       R = R1 x R2 x R3
      --[ R1 ]---[ R2 ]---[ R3 ]-->

  PARALLEL / redundant (any one works):   R = 1 - (1 - R1)(1 - R2)
             +--[ R1 ]--+
      -------|          |------->
             +--[ R2 ]--+

Now the worked example this chapter is built around: the same spacecraft, with and without one redundant unit. It shows both moves at once and makes the payoff concrete.

Worked Example: series vs. one redundant component. A spacecraft's mission depends on three functions in series, each with its single-string reliability over the mission (let $R$ denote reliability throughout this chapter — not the gas constant of Part III):

Function	Single-string reliability
Electrical power	$R_p = 0.98$
Onboard computer / avionics	$R_c = 0.90$
Propulsion	$R_{\text{prop}} = 0.97$

No redundancy (all in series): $$R_{\text{series}} = 0.98 \times 0.90 \times 0.97 = 0.8555.$$ The system is about 85.6% reliable — a mission-loss probability of $1 - 0.8555 = 14.4\%$, roughly one chance in seven. Notice it is already below the weakest link (the $0.90$ computer): the series multiplication has pulled it down further.

Add one redundant computer. The computer is the weak link, so we harden it: fly two identical computers in active parallel, each $R_c = 0.90$. The computer block becomes $$R_{c,\text{redundant}} = 1 - (1 - 0.90)^2 = 1 - (0.10)^2 = 1 - 0.01 = 0.99.$$ The block leaps from $0.90$ to $0.99$. Re-multiply the series: $$R_{\text{system}} = 0.98 \times 0.99 \times 0.97 = 0.9411.$$ One extra box took the whole mission from 85.6% to 94.1% — cutting the mission-loss probability from $14.4\%$ to $5.9\%$, a reduction of nearly 60%, for the price of a single redundant computer.

The lesson hidden in that example is worth stating outright, because it is where engineering judgment lives: redundancy pays most where reliability is lowest. Suppose we had instead spent our one backup on the propulsion (the $0.97$ link): the propulsion block would become $1 - (0.03)^2 = 0.9991$, and the system would reach only $0.98 \times 0.90 \times 0.9991 = 0.881$ — a jump to just 88.1%, far less than the 94.1% we got by backing up the weak computer. Doubling a strong link barely moves the product; doubling the weak link transforms it. A reliability engineer's first question is always "where is the weakest string, and what would a backup there cost?" — never "what can we gold-plate?"

The links in a series chain that have no backup have a special, dangerous name.

Definition (single point of failure). A single point of failure (SPOF) is any component or function whose failure, by itself, causes loss of the mission — a series element with no redundancy. In the diagram above, before we added the second computer, every block was a single point of failure. The central goal of the failure analysis in the next section is to find every SPOF and then, one at a time, decide whether to eliminate it (add redundancy), mitigate it (add margin, screen it by test), or accept it (with eyes open, because some SPOFs — the single main engine bell, the primary structure, the one heat shield — cannot be practically duplicated).

Two caveats keep redundancy honest, and both have killed real missions. First, redundancy costs mass, and mass is the enemy (Chapter 3): a second computer, a second string of valves, a fourth reaction wheel are all kilograms that the rocket equation punishes exponentially. You cannot make everything redundant; you spend redundancy where the reliability math says it buys the most, and you make the rest reliable by other means. Second — and this is the subtle one — the parallel formula $1-(1-R)^n$ assumes the failures are independent. If a single cause can take out both units at once, the two boxes are not really in parallel at all.

Definition (common-cause failure). A common-cause failure is a single fault or condition that disables multiple redundant units simultaneously — the same software bug in both computers, the same radiation event frying both boxes, the same contaminated propellant line starving both thrusters, the same technician's error on both strings. Common-cause failures break the independence assumption, so the real reliability of "redundant" hardware can be far below the $1-(1-R)^n$ the block diagram promises. It is the reason true redundancy demands dissimilar backups, physical and electrical separation, and independent design — and, as we will see, the reason Ariane 5's two identical inertial units were, in reliability terms, only one.

💡 Intuition: Think of series reliability as a rope made of links (one weak link breaks the rope) and parallel reliability as two ropes side by side (both must break to drop the load). Redundancy is a second rope — but only if it is a different rope, anchored separately. Two ropes cut by the same falling knife are one rope wearing a disguise. That disguise — redundancy that isn't independent — is the most expensive illusion in the business.

🔄 Check Your Understanding 1. A subsystem has four identical strings in series, each $R = 0.97$. What is its reliability? Now put two such subsystems in parallel — what is the reliability of the pair? 2. A vendor advertises "fully redundant dual flight computers, 0.99 each, so 0.9999 reliability." Under what circumstance is that 0.9999 badly optimistic?

Answers

1. Series: $0.97^4 = 0.885$. Two such subsystems in parallel: $1 - (1 - 0.885)^2 = 1 - (0.115)^2 = 1 - 0.0132 = 0.987$. Redundancy at the subsystem level lifted $0.885$ to $0.987$. 2. When the two computers share a common cause — identical software with the same latent bug, a shared power supply, a shared sensor input, the same radiation vulnerability. Then a single event fails both, and the true reliability is closer to a single computer's $0.99$ (or worse) than to the independence-assuming $0.9999$.

We can put these two moves in code — the start of the reliability.py module you will complete in this chapter's Checkpoint:

def series(rs):
    """Reliability of independent components all required (in series)."""
    p = 1.0
    for r in rs:
        p *= r
    return p

def redundant(r, n):
    """Reliability that at least one of n identical parallel units survives."""
    return 1.0 - (1.0 - r) ** n

chain = [0.98, 0.90, 0.97]                             # power, computer, propulsion
print(round(series(chain), 4))                         # no redundancy
print(round(series([0.98, redundant(0.90, 2), 0.97]), 4))  # dual computer on the weak link
# Expected output:
# 0.8555
# 0.9411

32.3 Failure modes and FMEA

The reliability math of 32.2 tells you what a chain of numbers does; it does not tell you where the numbers come from, or which of a spacecraft's thousands of parts deserve a backup. For that you need a disciplined way to walk through a design and ask, of every piece, how could this fail, and what happens if it does? That procedure is the FMEA.

Definition (FMEA — Failure Modes and Effects Analysis). An FMEA is a systematic, bottom-up examination in which, for every component, the analyst lists each way it can fail (its failure modes), the effect of that failure on the subsystem and the mission, the likely cause, how the failure would be detected, and the mitigation in place. When each row is also scored for criticality, the method is called FMECA (Failure Modes, Effects, and Criticality Analysis). The output is a ranked list that directs engineering attention — and redundancy, and test effort, and margin — to the failures that matter most.

FMEA is bottom-up: it starts at the parts and works upward to the mission. Its natural complement is fault tree analysis, which is top-down — it starts from an undesired top event ("loss of attitude control") and works downward through the logic of what combinations of failures could cause it. The two are used together because they catch different things: FMEA is exhaustive about single-component failures and their consequences; fault trees are better at exposing how combinations of failures (and common-cause links) conspire. A thorough program does both.

Here is one row of an FMEA for a spacecraft's propulsion feed system, worked in full so you can see the shape of the reasoning:

That single row does an enormous amount of work. It has named a SPOF (a single latch valve that could end the mission), judged its severity (catastrophic), checked that the failure is at least detectable (so the ground knows what happened), and specified a mitigation (a redundant valve) — which, if you plug $R = 0.98$ for one valve into $1-(1-R)^2$, turns a $2\%$ mission-loss mode into a $0.04\%$ one. Multiply that discipline across every component and you have converted a vague sense that "a lot could go wrong" into a ranked, quantified, reviewable list of exactly what could go wrong and what has been done about it.

To rank the rows, most FMEAs score each failure mode on two (sometimes three) axes and combine them into a risk index. A common bottom-up form is the Risk Priority Number, $\text{RPN} = S \times O \times D$, where $S$ is severity, $O$ is the likelihood of occurrence, and $D$ is the difficulty of detection, each on a 1–10 scale (a high $D$ means hard to detect, which is bad). For a mission-level view, engineers more often use a severity × likelihood risk matrix — the tool you will use for your project:

  LIKELIHOOD                              SEVERITY  ------------------->
      |            Negligible (1)   Marginal (2)   Critical (3)   Catastrophic (4)
  Frequent (5)        MED             HIGH           HIGH            HIGH
  Probable (4)        MED             MED            HIGH            HIGH
  Occasional (3)      LOW             MED            MED             HIGH
  Remote (2)          LOW             LOW            MED             HIGH
  Improbable (1)      LOW             LOW            MED             MED

The matrix encodes an engineering value judgment that is worth making explicit: a catastrophic outcome is treated as high-risk even when it is improbable, because the cost of being wrong is total. This is why a mission will spend heavily to drive down a rare-but-catastrophic failure (a tank rupture, an uncommanded engine shutdown) while tolerating a common-but-trivial one (a redundant heater cycling more than expected). Reliability engineering is not about eliminating all risk — that is impossible — but about spending a finite budget of mass, money, and schedule where the product of severity and likelihood is largest.

📜 From History: FMEA grew up in the U.S. military and aerospace programs of the late 1950s and 1960s, and it was on the Apollo program that NASA institutionalized failure-modes analysis at scale — thousands of engineers systematically asking "how can this kill the crew?" of every component in the stack. The discipline's descendants are everywhere now, from spacecraft to surgery to car brakes. The through-line is theme #2: in systems where failure is unforgiving, you do not wait to discover failure modes in operation; you enumerate them, on paper, before you build — and you argue about the scary rows until someone has done something about them.

🧩 Productive Struggle. Before reading on: an FMEA row has severity "catastrophic" but occurrence "improbable," so its risk lands in a middle box. A schedule-pressured manager argues it should be closed as "acceptable, low likelihood." What question should the review board ask before agreeing — and what does the detection column have to do with the answer?

The board should ask: if it did happen, would we even know why — and could we do anything? A catastrophic, improbable failure that is also undetectable is far more dangerous than the risk score alone suggests, because you cannot learn from it, cannot warn of it, and cannot design the next mission against it. Low likelihood is not the same as low risk when the severity is total and the detection is poor. (This is precisely the trap that will appear in the Mars Climate Orbiter case — a slow error that was detectable, and was even noticed, but was not acted on.)

32.4 The test campaign: "test like you fly"

Analysis tells you what should work. Only test tells you what does. Every reliability number in an FMEA is, at bottom, a claim about the physical world, and the physical world has to be asked. A spacecraft therefore earns its way to the launch pad through a test campaign: a planned sequence of tests, from individual components up to the fully assembled vehicle, each designed to expose a different class of defect before space does. The campaign rests on a distinction that every space engineer must be able to state cold.

Definition (qualification testing). Qualification testing subjects a dedicated test article to environments more severe than flight — higher vibration levels, wider temperature ranges, more thermal cycles, longer durations — to prove that the design has margin against the flight environment. Because the qualification article is deliberately overstressed, it is generally not flown; it has spent some of its life proving the design is sound.

Definition (acceptance testing). Acceptance testing subjects each actual flight unit to environments at (or just slightly above) the expected flight levels, to catch manufacturing and workmanship defects — the infant-mortality failures of 32.1 — without consuming the unit's design life. Qualification asks "is the design good enough?"; acceptance asks "was this particular copy built correctly?" You need both, because a sound design can still be assembled with a cold solder joint, and a flawlessly built unit can still be of a design with too little margin.

When a program cannot afford a separate qualification article — common for one-off spacecraft and small satellites — it flies a compromise called protoflight: the flight unit itself is tested at qualification levels but for acceptance durations, accepting a little extra life consumed in exchange for not building a second copy. The choice among dedicated-qual, protoflight, and pure-acceptance is a classic cost-versus-confidence trade of the kind Chapter 29 taught you to run.

The environments a spacecraft is tested against are exactly the hazards earlier chapters taught you to respect: vibration, acoustic, and shock testing for the brutal ride to orbit (the launch loads of Chapter 23); thermal- vacuum (TVAC) testing in a chamber pumped down to space pressure and cycled between hot and cold extremes (the thermal environment of Chapter 24); electromagnetic compatibility; and end-to-end functional tests that command the spacecraft through its mission. For engines there is the ultimate test: the hot-fire, and for a stage, the full-duration static fire — a "green run" that fires the assembled propulsion system on the ground exactly as it will fire in flight. The guiding philosophy has a name, and it is the most important sentence in this section.

Definition ("test like you fly"). "Test like you fly" is the principle that a test should reproduce the flight as closely as possible — the same hardware, the same software, the same interfaces, the same configuration, the same sequence of events, in the same environment — because the failures that kill missions hide in the differences between how you tested and how you fly. Its converse, "fly like you test," forbids surprising the vehicle in flight with any condition it did not first survive on the ground. Every gap between test and flight is an untested assumption, and space is the most expensive place to discover an untested assumption.

The reason "test like you fly" is stated so insistently is that the gaps are seductive. It is cheaper to test a subsystem with a simulated input than with the real sensor; cheaper to test with flight software's predecessor than to wait for the final build; cheaper to test components separately than to assemble the whole stack and test it as one. Each shortcut saves money and schedule, and each one opens a gap where a failure can hide until flight. The N-1 case in the next section is the canonical warning: a rocket whose integrated first stage was never test-fired as a unit before flight, four times, to four failures.

There is a hard limit worth naming plainly: you cannot test reliability into a design at the end. Reliability is designed in — through simplicity, margin, redundancy, and derating — and then verified by test. Testing does not add reliability; it reveals it, and reveals the defects that must be fixed. A program that plans to "test until it passes" has misunderstood the tool. A test that a unit barely survives is not a success; it is a warning that the design margin is thin, and a thin margin over a distribution of flight conditions is a failure waiting for a bad day.

🔧 Engineering Reality: Two cultures have grown up around this truth, and both are rational responses to the same reliability problem. The traditional approach — NASA, ULA, the European and Japanese agencies — tests exhaustively on the ground and aims to fly correctly the first time, because their payloads (crews, flagship science, national-security satellites) make a failed flight unthinkable. The iterative approach — SpaceX's early Falcon and Starship programs — deliberately flies test articles to failure, fast and cheap, treating each explosion as a data point that ground tests cannot fully provide (we take this up in Chapter 38). The tension between them is real, but the disagreement is about where to buy information, not whether to: both are paying, in different currencies, for the same commodity — knowledge of how the real hardware behaves before it matters.

🔄 Check Your Understanding 1. Distinguish qualification from acceptance testing in one sentence each. Why can't one substitute for the other? 2. A team saves a month by testing the flight computer with recorded data instead of the live star tracker it will use in flight. Which principle are they violating, and what class of failure might they miss?

Answers

1. Qualification stresses a design beyond flight levels to prove it has margin; acceptance stresses each flight unit at flight levels to catch build defects. Neither substitutes because a good design can be badly built (needs acceptance) and a well-built unit can have an inadequate design (needs qual). 2. "Test like you fly." They might miss a failure in the interface between the star tracker and the computer — a timing mismatch, a data-format error, an electrical incompatibility — exactly the kind of real-hardware, real-sequence failure that recorded data cannot reproduce. (This is the seam Ariane 5 fell through.)

32.5 Case studies in failure, and what they taught

Reliability engineering learns from failure the way medicine learns from autopsy. When a mission is lost, a board convenes to do a root-cause analysis — and the lessons it extracts are paid for in hundreds of millions of dollars and, too often, in lives. Three cases below are on every space engineer's syllabus. Each failed for a reason the tools of this chapter would have caught; together they cover software, interfaces, and hardware.

Definition (root-cause analysis). Root-cause analysis (RCA) is the disciplined investigation that traces a failure past its immediate symptom to the underlying cause(s) — technical and organizational — so that the fix addresses the real defect rather than the symptom. Techniques include the "Five Whys" (ask why repeatedly until you reach something you can change), fault trees, and the fishbone (Ishikawa) diagram. A good RCA separates the proximate cause (the part that broke) from the root cause (the reason a part that could break was there, untested, in the first place) — because only fixing the root cause prevents the next, differently-shaped failure.

Ariane 5 Flight 501 (1996): a line of reused software

On its maiden flight in June 1996, the European Ariane 5 broke apart about 37 seconds after liftoff, destroying four Cluster science satellites. The proximate cause was astonishing in its smallness. The Inertial Reference System (SRI) — the box that tells the rocket which way is up — ran software inherited from the reliable Ariane 4. Deep inside it, a 64-bit floating-point number representing horizontal velocity was converted to a 16-bit signed integer. Ariane 5 was a bigger, faster rocket, and early in flight its horizontal velocity was larger than any value Ariane 4 could produce; the number overflowed the 16-bit integer, the conversion threw an exception, and the SRI — following its design — shut itself down and reported a diagnostic pattern instead of attitude data.

Now the reliability lessons cascade. There were two SRIs, for redundancy — but they ran identical software, so the backup had already suffered the same overflow a few milliseconds earlier: a textbook common-cause failure that reduced two units to one (32.2). The main computer, receiving the SRI's diagnostic bit pattern, interpreted it as flight data, commanded the engine nozzles hard over, and aerodynamic loads tore the vehicle apart. And the deepest cut: the code that overflowed was an alignment function that served no purpose after liftoff — it was left running only out of Ariane-4 heritage, computing a number Ariane 5 did not even need. The root cause was not a typo; it was an untested assumption (that horizontal velocity stays small) carried across from one vehicle to another without re-validation in the new flight environment — a "test like you fly" failure for software. Software does not wear out, but it fails the instant it meets an input its designers never tested it against. (Details are widely documented from the Board of Inquiry report; specific numbers are Tier 2.)

Mars Climate Orbiter (1999): the interface that wasn't managed

We met this loss already, in Chapter 29's history of systems engineering: the orbiter was destroyed at Mars because one team's ground software produced a thruster impulse in pound-force-seconds while the navigation software expected newton-seconds, a factor of about 4.45 that accumulated silently over months of cruise until the spacecraft entered the atmosphere far too low. Seen through this chapter's lens, two further lessons appear. First, it was a single point of failure at an unmanaged interface — no redundancy protects you from a systematic error that both "copies" share, because it is a common-cause error in the design, not a random failure in a unit. Second, and more damning, the discrepancy was detectable and was in fact detected: navigators noticed the spacecraft tracking off-course during cruise and filed concerns, but the signals were not escalated into action before arrival. A detectable failure that is not acted upon is, operationally, an undetected one. The root cause was organizational as much as technical — a communication and process failure of exactly the kind that, in the Space Shuttle program, would acquire the name normalization of deviance, a concept we take up in Chapter 37.

The Soviet N-1 (1969–1972): a cluster you cannot test

The N-1 was the Soviet Union's Moon rocket, and its first stage carried thirty NK-15 engines — an extraordinary number, chosen because no single engine large enough existed. All four N-1 launches failed, none reaching orbit. The reliability story is a compound of everything in this chapter. Thirty engines in a configuration where you need most of them working is a long series-and-voting problem: even at a generous $0.99$ per engine, the chance that all thirty run cleanly is $0.99^{30} = 0.74$ — a one-in-four chance of some engine failing on every flight, before you count the plumbing that feeds them. The designers knew this and built engine-out logic (a control system called KORD) to shut down a failed engine and its opposite for balance — a form of redundancy. But the failures were not benign: an engine that fails violently can damage its neighbors and the surrounding structure, a common-cause cascade that redundancy does not survive. And the decisive root cause was a testing choice: to save cost and schedule, the Soviets did not static-fire the assembled first stage before flight, relying on individual-engine tests and analysis instead. They violated "test like you fly" at the largest scale — the integrated stage was flown before it was ever fired as a unit — and paid with four vehicles. Contrast the American approach, which test-fired whole stages on enormous stands before they ever flew, and the lesson is stark: the integration is part of the design, and integration must be tested.

🐛 Find the Error. After Ariane 5, an engineer proposes: "The fix is a third SRI — two-out-of-three voting so a single failure is outvoted." Using this chapter's ideas, explain why a third identical SRI would not have saved Flight 501, and what the actual fix must address.

Answer

Three identical SRIs share the same software, so all three hit the same overflow on the same input at essentially the same moment — a common-cause failure. Two-out-of-three voting protects against independent, random unit failures; it does nothing against a deterministic software fault that every copy shares, because there is no disagreement to vote on: all three are wrong the same way. The real fix addresses the root cause — remove the unnecessary function, handle the exception gracefully instead of shutting down, re-validate reused code against the new vehicle's flight envelope, and (for genuine redundancy against software) use dissimilar implementations. Redundancy multiplies reliability only when the failures are independent.

The common thread across all three is that none was, at root, a failure of physics or of a single clever part. Ariane 5's chips worked; Mars Climate Orbiter's thrusters fired; the N-1's engines mostly ran. Each was a systems failure — a bad assumption, an unmanaged interface, an untested integration — the precise failures that FMEA, "test like you fly," and honest root-cause analysis exist to prevent. Reliability is lost not usually in the parts but in the seams between them.

🔄 Check Your Understanding 1. Name the single reliability concept that appears in all three case studies. 2. In one phrase each, give the root cause (not the symptom) of each failure.

Answers

1. Common-cause failure / the failure of the system at the seams rather than of an individual part (Ariane 5: identical software in both SRIs; MCO: one units error shared by the whole nav design; N-1: cascading engine damage and untested integration). 2. Ariane 5 — reused software assumption never re-validated for the new vehicle; MCO — an unmanaged unit interface and unescalated warnings; N-1 — the integrated first stage was never static-fired ("test like you fly" violated).

32.6 Margins, safety factors, and the cost of reliability

The chapters have circled a single quiet truth, and now we can say it directly: reliability is bought, and the currency is margin, mass, and money. Everything that makes a spacecraft more likely to survive — a redundant string, a thicker structure, a longer test campaign, a derated part — costs something the mission would rather spend on payload. The discipline of reliability engineering is not the pursuit of perfect safety, which is unattainable; it is the allocation of a finite reliability budget to buy the most survival per kilogram and per dollar.

The oldest form of that purchase is the factor of safety you met in Chapter 23: design a structure to withstand more load than it will ever see — typically 1.25 to 1.5 times the limit load for flight structures, higher for pressure vessels and crewed systems — so that variation in materials, loads, and workmanship never eats through to failure. The factor of safety is margin against the strength distribution overlapping the load distribution; reliability is high precisely when those two distributions barely overlap. Derating is the same idea for every non-structural part: run a resistor at a fraction of its rated voltage, a bearing below its rated speed, a valve below its rated cycles, and its effective failure rate plummets — you have moved it deep into the flat floor of the bathtub curve. Margin (the reserve of Chapter 29), factor of safety (Chapter 23), redundancy, and derating are four faces of one strategy: never let the design sit at the edge of what it can do, because the edge is where the distribution's unlucky tail reaches failure.

But margin is not free, and here the theme of the whole book returns. Every redundant unit, every extra millimeter of wall, every added test is mass, cost, and schedule — and mass, through the rocket equation, is the most expensive of all. This is the fundamental tension of the chapter: the cost of reliability rises steeply as you approach certainty. Going from 90% to 99% reliability might cost one redundant string; going from 99% to 99.9% might cost three more, plus a longer test campaign; and no amount of money buys 100%. The curve of cost against reliability bends sharply upward near the top, which forces every program to answer an uncomfortable question: how safe is safe enough?

The answer depends on what is at stake, and the industry encodes that judgment in its requirements. An uncrewed technology-demonstration CubeSat might fly happily at a design reliability of 0.7 — it is cheap, it is one of many, and its loss teaches something. A flagship science mission with a decade of build behind it and no spare is designed toward 0.9 or higher, its every SPOF scrutinized. And a crewed vehicle is held to a different order of stringency entirely: NASA's Commercial Crew Program, for instance, set a required probability of loss of crew on the order of 1 in 270 for ascent and entry — a reliability target that drives redundancy, testing, and abort systems far beyond anything an uncrewed cargo flight would carry (the specific figure is Tier 2; the point is the order of magnitude). Human-rating a vehicle is, in reliability terms, buying down the loss-of-crew probability to a number a society will accept — and accepting that it will never be zero.

🔧 Engineering Reality: The most dangerous failure of reliability engineering is not a wrong number; it is a cultural drift in which repeated success erodes the respect for margin. If a joint has leaked a little on the last ten flights and nothing bad happened, the pressure to treat the leak as "normal" — to quietly consume the margin the designers left — becomes immense. This is the organizational failure mode that destroyed both Space Shuttles, and it has a name we will study in Chapter 37. For now, hold the principle: margin is not there to be spent when the schedule is tight; it is there to be spent by the physics you did not model. A program that treats its reserves as slack has already begun to fail, it just does not know it yet.

🔄 Check Your Understanding 1. Why does the cost of buying reliability rise so steeply as reliability approaches 1? 2. Give two distinct "faces" of margin from this section and say what distribution each one holds apart from failure.

Answers

1. Because the cheap improvements are used first: the initial gains come from a single redundant string or a modest margin, but each further "nine" of reliability requires backing up already-good components (where redundancy pays little, per 32.2), longer testing, and tighter screening — with diminishing return per dollar — while 100% remains unreachable at any price. 2. E.g., factor of safety holds the strength distribution apart from the load distribution (structures); derating holds a part's operating stress well below its rated capability, pushing its failure-rate distribution down; delta-v/mass margin holds the current best estimate apart from the design limit. Each keeps the design off the edge where the unlucky tail reaches failure.

Mission Design Checkpoint: add a risk assessment, and build reliability.py

This is a major increment to your Mission Design Review — the one that turns an optimistic design into an honest one. Until now your MDR has assumed everything works. Now you ask, systematically, what if it doesn't?

The design — a risk assessment for your MDR. Add a new section to your MDR with three parts:

1. A reliability block diagram. Sketch your mission's critical functions as series and parallel blocks — power, avionics, propulsion, comms, attitude control, payload. Mark every block that has no backup: those are your single points of failure.
2. An FMEA-lite table. For your five highest-risk items, fill in the columns of 32.3: item, failure mode, effect, detection, and a risk score. Use Severity (1–4) × Likelihood (1–5) and place each on the risk matrix. Sort by risk. For each HIGH-risk row, state your mitigation — redundancy, margin, or a test that would screen it out.
3. A reliability estimate and allocation. Assign a single-string reliability to each block, multiply the series chain, and see whether it meets a target you set (say $R \ge 0.90$ for the mission). If it falls short, add redundancy to the weakest block first and re-compute — exactly the move of 32.2 — and record what mass that redundancy costs against your Chapter 23 mass budget. That trade (mass for reliability) is the heart of your risk assessment, and it flows straight into the driving-requirement discussion of Chapter 29.

The code. Create astrotools/reliability.py, the reliability calculator for your MDR:

from math import comb

def series(rs):
    """Reliability of independent components all required (in series)."""
    p = 1.0
    for r in rs:
        p *= r
    return p

def parallel(rs):
    """Reliability of independent redundant units (system works if >= 1 works)."""
    q = 1.0
    for r in rs:
        q *= (1.0 - r)
    return 1.0 - q

def redundant(r, n):
    """n identical parallel units: probability at least one of n survives."""
    return 1.0 - (1.0 - r) ** n

def k_of_n(r, n, k):
    """Probability at least k of n identical units survive (e.g., engine-out)."""
    return sum(comb(n, i) * r**i * (1 - r)**(n - i) for i in range(k, n + 1))

if __name__ == "__main__":
    chain = [0.98, 0.90, 0.97]                  # power, computer, propulsion (32.2)
    print(round(series(chain), 4))              # no redundancy
    hardened = [0.98, redundant(0.90, 2), 0.97] # dual computer on the weak link
    print(round(series(hardened), 4))
    print(round(k_of_n(0.99, 9, 8), 4))         # Falcon 9: >= 8 of 9 engines
    # Expected output:
    # 0.8555
    # 0.9411
    # 0.9966

How it feeds the capstone. Your risk assessment is the last major section of the MDR you will present in Chapter 40; reliability.py is the tool that produces its numbers, and the full astrotools package is reproduced in Appendix I. A mission design that closes on mass, power, and delta-v but has an unexamined single point of failure has not actually closed — it has only looked closed until the review board asks the question this chapter taught you to ask first.

Summary

Reliability is the discipline of making a system work when it must, with no second chance. Carry these forward:

Key relationships: series $R=\prod R_i$; parallel $R=1-\prod(1-R_i)$; identical redundancy $R=1-(1-R)^n$; exponential $R(t)=e^{-\lambda t}$. astrotools addition: reliability.py with series, parallel, redundant, k_of_n.

Spaced Review

Retrieval strengthens memory. Answer from memory before checking, then look back at the cited chapter.

1. (Ch. 29) Reliability engineering leans on margin. In Chapter 29 you learned margin is "bought down" as a program matures — 25–30% mass margin at MDR, near zero at launch. In one sentence, why is it legitimate to reduce margin over time rather than hold it constant?
2. (Ch. 29) What is a single driving requirement, and why might a crewed mission's loss-of-crew target be its driving requirement in the sense of this chapter?
3. (Ch. 7) A re-entry heat shield is a classic single point of failure. Roughly how much kinetic energy per kilogram must it dispose of on return from LEO, and why does "no second chances" apply with special force to re-entry?
4. (Ch. 7) The re-entry corridor has two failure boundaries. Name them, and explain why a corridor is itself a kind of margin.
5. (Ch. 29 synthesis) Your MDR "closes" on mass, power, and delta-v but has an unexamined SPOF. Using this chapter, argue in one sentence why it has not really closed.

Answers

1. Because margin covers uncertainty, and uncertainty falls as estimates are replaced by built-and- measured hardware — you buy down margin by buying knowledge, so near launch, when the masses are known, little reserve is needed. 2. The driving requirement is the one requirement the coupled design is most sensitive to; for a crewed vehicle the loss-of-crew probability drives redundancy, abort systems, and testing across every subsystem, rippling into mass and cost more than any other single requirement. 3. About $30\ \text{MJ/kg}$ (from an orbital speed of ~7.8 km/s); "no second chances" bites hardest because the shield must work correctly the first time, continuously, for the whole descent, with no possibility of repair or retry — a textbook catastrophic SPOF. 4. Too steep (excess deceleration and heating, vehicle burns up or crushes) and too shallow (skips back out of the atmosphere); the corridor is a margin because it is the band of entry angles the vehicle can tolerate, and staying inside it with room to spare is exactly holding margin against navigation and control error. 5. Because a design with a catastrophic single point of failure has an unbudgeted probability of total loss — it balances its resource budgets but not its risk budget, so it can look closed while carrying an unexamined path to mission failure that the review board's first hard question would expose.

What's Next

You now have the fourth great budget of mission design — after mass, power, and delta-v comes risk — and the tools to compute it: series and parallel reliability, FMEA, the test campaign, and the hard-won lessons of real failures. You have also met the central tension that the rest of the book keeps returning to: reliability is bought with mass and money, and neither is unlimited.

That tension is about to be resolved in a surprising direction. For decades the answer to "how do you make a spacecraft reliable?" was "make it big, redundant, exquisitely tested, and expensive — and fly few of them." But there is another answer, born of cheap electronics and cheaper launch: make spacecraft small, build many, accept that some will fail, and get your reliability from the constellation rather than the individual. In Chapter 33 we turn to CubeSats, small satellites, and the mega-constellations of thousands of spacecraft — where a single satellite may be designed to a reliability that would horrify an old-guard engineer, and the system is nonetheless robust because there are a thousand of them and the ground can simply route around the dead ones. It is theme #5, reusability's cousin — the democratization of space — meeting theme #2 head-on, and rewriting the economics of "everything must work."