Case Study 2: The Green Build That Tested Nothing

"Every pull request for nine weeks passed CI in eleven seconds. We thought we had made it fast."

Executive Summary

Kestrel moved to slim CI in June 2026 and the build time fell from 22 minutes to about three. Six weeks later it fell again, to eleven seconds, and nobody investigated a build getting faster.

The production manifest had stopped being uploaded. A refactor of the nightly workflow moved the dbt build step into a matrix job, and the artifact upload — which ran in an if: success() block attached to the old step — silently stopped running.

dbt build --select state:modified+ against a missing manifest selects nothing. dbt exited 0. The job went green. For nine weeks, every pull request was tested by a command that built zero models and passed.

It was found when a model with a syntax error reached production, and the postmortem's first question — "how did CI pass?" — had an eleven-second answer.

Skills applied: slim CI's operational requirements (§27.4); the fallback that was specified and not implemented; and the general shape of a control that fails open.

Background

The setup, from §27.4:

# nightly.yml -- produces the artifact
- run: dbt build --target prod
- uses: actions/upload-artifact@v4
  if: success()
  with: {name: manifest, path: target/manifest.json}

# ci.yml -- consumes it
- uses: actions/download-artifact@v4
  with: {name: manifest, path: prod-manifest/}
  continue-on-error: true            # ←
- run: dbt build --select state:modified+ --defer --state prod-manifest/

continue-on-error: true on the download is the whole incident. It was added deliberately, for a reasonable stated reason — "the first run after a manifest expires shouldn't block everyone" — and it converts a missing artifact from a loud failure into a silent one.

The refactor, six weeks later, split the nightly build across a matrix of targets to parallelize it. The dbt build step moved; the upload step's if: success() now referred to a step that no longer existed in that job, and GitHub Actions evaluated it against the job's status rather than erroring.

No warning. No failed workflow. The artifact simply stopped being produced.

The Problem

week 1-6    CI: 3.1 min   models built: 12 (median)   ✅
week 7      CI: 0:11      models built: 0             ✅
week 8      CI: 0:11      models built: 0             ✅
   ⋮
week 15     CI: 0:11      models built: 0             ✅   ← 63 merged PRs

Nine weeks. Sixty-three pull requests. Zero models built in CI.

The log said, every time:

Downloading artifact manifest... not found
Warning: continue-on-error
11:04:32  Running with dbt=1.9.1
11:04:38  Nothing to do. Try checking your model configs and model specification args
11:04:38  Done. PASS=0 WARN=0 ERROR=0 SKIP=0 TOTAL=0
Process completed with exit code 0.

PASS=0 ... TOTAL=0 and exit code 0. Everything needed to diagnose this was in every log, every time, and nobody reads a green log.

⚠️ Failure Mode — a control that fails open, and gets faster when it fails

Two properties combined here, and either alone would have been survivable.

It failed open. state:modified against a missing manifest selects nothing, and dbt correctly treats "nothing selected" as success — there is nothing wrong with a run that had no work. The failure mode of the artifact became the success of the job.

And it got faster. This is the part that made nine weeks possible. A control that gets slower when it breaks is investigated within a day. One that gets faster is experienced as an improvement, and the team's Slack has a message from week 7 saying "CI is really quick now 🎉".

The general shape, and it is worth carrying beyond CI: a check whose failure is indistinguishable from having nothing to check.

  • An empty test suite passes.
  • A WHERE clause matching no rows returns no violations.
  • A state:modified with no state selects nothing.
  • A monitoring query against a table that no longer exists — which, depending on your tool, may report zero rather than error.

The fix is the same in every case: assert that the check did something. Not that it passed — that it ran.

```bash

The four lines that make this class impossible.

n=$(jq '[.results[] | select(.resource_type=="model")] | length' target/run_results.json) if [ "$n" -lt 1 ]; then echo "CI built $n models. That is not a pass." >&2; exit 1 fi ```

A build that tests nothing must be a failure, and expressing that takes one comparison.

The Analysis

Step 1: why did nobody notice? The postmortem asked this precisely, and the answers are all individually reasonable:

The duration alert did not fire. Chapter 25 §25.4's rule alerts on r > 2.0 and r < 0.5 — and 11 seconds against a 3.1-minute median is r = 0.06, which is well below 0.5.

Because the rule was not applied to CI. It was implemented for the nightly DAG's tasks, from run_records, and CI runs did not write run records. The measurement existed and its coverage did not include the thing that broke.

The PR comment (§27.13) reported the truth and nobody read it. Looking back:

📊 Data impact
   models modified          0        ← for sixty-three consecutive PRs
   downstream models        0
   exposures affected       0
   deploy shape             ADDITIVE

"models modified: 0" on a pull request that modified a model. It was in every comment, on every PR, for nine weeks. Nobody read it because it was always the same, which is the same mechanism as Chapter 24 Case Study 2's forty-one restart alerts and Chapter 25 Case Study 2's 89%.

Step 2: what got through? Sixty-three pull requests, re-tested retroactively against the restored manifest:

63 PRs merged with no CI coverage
├── 58  would have passed anyway
├──  3  would have failed on a dbt test  (caught later by the nightly build)
├──  1  would have failed on a compile error  ← reached production
└──  1  would have failed the grain test      ← reached production, found
                                                by Ch. 23's coverage check

Two reached production. The compile error broke the nightly build the morning after its merge — a four-hour incident, and the one that triggered the investigation. The grain violation ran for six days before the grain test on the affected mart caught it, because the mart was rebuilt weekly.

Step 3: was the specified fallback ever implemented? §27.4 specifies exactly this:

if [ ! -f ./prod-manifest/manifest.json ]; then
  dbt build            # slower, and it actually tests something
fi

It was in the design document and not in the workflow. The engineer who wrote the workflow had implemented continue-on-error — the first half of the requirement, "do not block on a missing manifest" — and not the second half, "and then do something else instead."

🔎 Read the Plan — "do not fail" and "do something else" are two requirements

The design said: "a missing manifest should not block CI; fall back to a full build."

The implementation did the first clause. And a review of the workflow diff would have shown continue-on-error: true, which matches the sentence's first half perfectly and is exactly the kind of partial implementation that survives review.

The general pattern: a requirement of the form "do not X; instead Y" is two requirements, and the negative one is easier, is what the reviewer's eye lands on, and is the one that gets implemented.

  • "Do not page overnight; batch into a morning window." → paging disabled, no window built.
  • "Do not block the load on a bad row; quarantine it." → Chapter 23 Case Study 1, exactly.
  • "Do not fail on a missing manifest; fall back to a full build." → this.

The mitigation is to write the requirement with the positive clause first, and to test the negative path:

"When the manifest is missing, CI runs a full build. It must never run zero models."

One sentence, testable, and the failure mode is now the thing being asserted rather than the thing being avoided.

The Decision

Five changes.

One: assert that CI built something. The four lines from the ⚠️ callout, and the primary fix — it makes the entire class impossible regardless of cause.

Two: implement the fallback, positively:

- name: fetch production manifest
  id: manifest
  continue-on-error: true
  uses: actions/download-artifact@v4
  with: {name: manifest, path: prod-manifest/}

- name: build
  run: |
    if [ -f prod-manifest/manifest.json ]; then
      dbt build --select state:modified+ --defer --state prod-manifest/
    else
      echo "::warning::no production manifest; running a FULL build"
      dbt build
    fi

Three: the nightly job asserts its own artifact. The producer verifies what it produced, rather than the consumer discovering it did not:

- run: test -f target/manifest.json || (echo "no manifest produced" && exit 1)
- uses: actions/upload-artifact@v4
  with: {name: manifest, path: target/manifest.json}   # no `if:`

The removal of if: success() is deliberate. It was doing nothing useful — the step only runs if prior steps succeeded anyway — and it was the thing that silently changed meaning during the refactor.

Four: CI runs write run records (Chapter 25 §25.4), so §25.4's ratio rule covers them. The measurement existed; only its coverage was missing.

Five: a weekly artifact-age check. The manifest's timestamp, alerted above 48 hours — which catches a stale manifest as well as a missing one, and a stale manifest is the subtler version of the same failure.

📐 Design Decision — should a missing manifest fail CI instead?

Proposed, and rejected, and the argument is a good example of choosing between two safe-sounding options.

For failing: it is unambiguous, it cannot fail open, and it is one line.

Against, and this won: the failure would land on whoever happened to open the next pull request, who has no context, cannot fix the nightly job, and is blocked. A control that punishes an uninvolved person for an infrastructure problem gets an exception carved into it within a fortnight — and the exception will be continue-on-error: true, which is where this incident started.

The fallback is better because it keeps the developer moving while still testing. It is slower — 22 minutes instead of three — and the slowness is itself the signal: a full build in CI is visible, annoying, and prompts exactly the question "why is CI slow today?"

The general principle: when a control can fail, prefer the degraded mode that is loud to the one that is blocking. A blocked person routes around; an inconvenienced person complains, and a complaint is a detection.

What Happened

During After
CI duration 11 s 3.1 min (or 22 min on fallback)
Models built in CI 0 12 median, never 0
PRs merged without coverage 63 0
Defects reaching production 2 —
Fallback implemented no yes, and tested

The two defects were fixed — the compile error the following morning, the grain violation after six days — and re-running the sixty-three retroactively found nothing else.

Two findings from the same review:

The --defer production read grant had also stopped working, for the same nine weeks and without anyone noticing, because a build of zero models makes no queries. A credential rotation in week 8 had not been propagated to the CI role, and the failure would have been immediate and obvious in any week that CI actually ran.

So a second, independent breakage was hidden by the first. The team's note: "we had two broken things and one symptom, and the symptom was that everything was fine."

And the PR comment got a highlight. models modified: 0 now renders as a warning line rather than a number:

📊 Data impact
   ⚠️  models modified          0  — CI built NOTHING. This is not a pass.

Which is the same information, and the difference is entirely presentational, and it is the change the team expects to matter most.

Lessons

  1. A check whose failure is indistinguishable from having nothing to check will fail open. An empty test suite passes; a state:modified with no state selects nothing; PASS=0 TOTAL=0 exits 0.

  2. Assert that the check ran, not that it passed. Four lines, and it makes the class impossible.

  3. A control that gets faster when it breaks is experienced as an improvement. Nine weeks, and a Slack message celebrating it.

  4. The duration rule that would have caught it was implemented and did not cover CI. The measurement existed; only its coverage was missing.

  5. "Do not X; instead Y" is two requirements, and the negative one is easier, is what a reviewer's eye lands on, and is the one that gets implemented alone. Write the positive clause first and test the negative path.

  6. models modified: 0 was in every PR comment for nine weeks. Always the same, so never read — Chapter 24 Case Study 2 and Chapter 25 Case Study 2's mechanism, a third time.

  7. Prefer a loud degraded mode to a blocking one. A blocked person routes around it; an inconvenienced person complains, and a complaint is a detection.

  8. Have the producer verify its own artifact, rather than the consumer discovering its absence.

  9. if: success() on an upload step does nothing useful and silently changes meaning when the job is restructured.

  10. Check artifact age, not just presence. A stale manifest is the subtler version of a missing one.

  11. Two broken things, one symptom, and the symptom was that everything was fine. A control that is not running hides every other control that depends on it.

Questions for Discussion

  1. continue-on-error: true was added for a stated, reasonable reason. What review question would have surfaced the missing second half?

  2. The team celebrated CI getting faster. What would make an unexplained improvement as suspicious as an unexplained regression, without making people cynical about genuine wins?

  3. Kestrel chose a loud fallback over a blocking failure. Construct the case where blocking is right.

  4. The --defer credential had also expired, hidden by the first failure. How would you find controls whose health is only observable when something else is working?

  5. models modified: 0 was correct and unread for nine weeks. Is highlighting it a real fix or a cosmetic one? What would make you confident either way?

  6. Retroactive re-testing found 58 of 63 PRs would have passed anyway. Does that make this a smaller incident than it looks, or is that the wrong way to size it?

  7. This chapter's two case studies are a deploy that shipped code without data, and a check that ran without checking. Both were "half of a two-part thing." How many two-part things are in your pipeline, and which halves are you sure about?