Case Study 1: Exit Code 137

"There was no error. There was no traceback. There was a number, and the number was 137."

Executive Summary

build_supplier_report — a pandas job that had run every morning for two years without a code change — began failing on some days and then on all of them.

The log said:

[2026-06-09 04:12:07] Task exited with return code 137

And nothing else. No exception, no stack trace, no partial output. Three days were spent investigating the orchestrator, the container runtime, and the node pool, on the reasonable assumption that a job which had not changed and now failed intermittently was an infrastructure problem.

It was not. 137 is 128 + 9: the process was killed by SIGKILL, by the kernel's OOM killer, because it exceeded its container's 4 GB memory limit. The job read a rolling 90-day window that had grown from 118 MB of Parquet to 147 MB, and pandas' 28× memory multiplier (§22.7) turned 147 MB into 4,145 MB against a 4,096 MB limit.

The data grew 3% and the job stopped working. The fix — selecting the eight columns it actually used, then moving it to DuckDB — took the peak from 4,145 MB to 426 MB.

Skills applied: the memory multiplier (§22.7); reading an OOM (Chapter 21 §21.9, on one machine); column projection; the notebook-to-pipeline checklist (§22.12).

Background

The job. A supplier-facing report: for each supplier, ninety days of demand by product, with a handful of derived rates. Written in a notebook in 2024, promoted to a scheduled job in a week, and untouched since.

import pandas as pd

df = pd.read_parquet(f"{LAKE}/gold/agg_customer_daily/")   # everything
sup = pd.read_parquet(f"{LAKE}/gold/dim_supplier/")

df = df[df.activity_date >= (pd.Timestamp.today() - pd.Timedelta(days=90))]
j = df.merge(sup, on="supplier_id", how="inner")
out = (j.groupby(["supplier_id", "product_id"], as_index=False)
        .agg(units=("units", "sum"), revenue_cents=("revenue_cents", "sum"),
             days_active=("activity_date", "nunique")))
out.to_parquet(f"{LAKE}/gold/supplier_report/", index=False)

Two things about that code, and neither is a mistake anyone would flag in review:

It reads the whole table and then filters. pandas has no optimizer (§22.2), so the 90-day filter runs after every row has been loaded and decompressed.

It reads all 34 columns and uses 8. There is no projection pushdown in a read_parquet without a columns= argument.

The environment. The orchestrator's worker container, 4 GB memory limit, shared with eleven other small jobs.

The growth. gold.agg_customer_daily is one row per active customer per day — about 62,000 rows a day, 5,580,000 in a 90-day window, 24.9 bytes per row compressed.

2024-06    90-day window   118 MB Parquet  →  ~3,328 MB peak RSS
2025-06                    131 MB          →  ~3,694 MB
2026-04                    139 MB          →  ~3,920 MB
2026-06                    147 MB          →  ~4,145 MB   ← over 4,096

The Problem

It failed intermittently for six weeks before it failed every day, which is what made it look like infrastructure.

2026-05-02  ✓ 00:04:41
2026-05-03  ✗ 137
2026-05-04  ✓ 00:04:52
2026-05-05  ✓ 00:04:47
2026-05-06  ✗ 137
...
2026-06-09  ✗ 137
2026-06-10  ✗ 137          ← every day from here

The intermittency had a real cause and it was not randomness. Peak memory depends on the data, and the number of active customers varies by day of week — a busy Monday's window is larger than a quiet Sunday's. The job sat just under the limit and crossed it on heavy days first.

Three days went into the wrong investigation, and the reasoning was sound at every step: the code had not changed, the failure was intermittent, and it produced no application-level error. Every one of those points at the platform.

⚠️ Failure Mode — 137 is the least informative failure in this book

128 + 9 = 137. Signal 9 is SIGKILL. SIGKILL cannot be caught, so the process cannot log, cannot flush, and cannot leave a traceback. You get a number.

What sends it, in order of likelihood for a data job:

• The kernel OOM killer, because the process exceeded a cgroup memory limit. This is almost always it.
• A container runtime enforcing the same limit directly.
• An orchestrator timeout escalating from SIGTERM to SIGKILL.
• A human, or a node draining.

The three-line diagnostic, and it takes about a minute:

bash kubectl describe pod <pod> | grep -A3 "Last State" # look for OOMKilled dmesg -T | grep -i "killed process" # the kernel's own record cat /sys/fs/cgroup/memory.peak # what it actually used

If Reason: OOMKilled appears, stop investigating the platform. It is not a node problem, a scheduler problem, or a flake, and the intermittency is data-dependent rather than random.

And the general lesson: a failure with no traceback is a failure that happened outside your process's control, which is a much narrower set of causes than a failure with one. The absence of information is itself information, and here it points almost directly at the answer.

The Analysis

Step 1: confirm it is memory. Reason: OOMKilled, and memory.peak at 4,294,967,296 — the limit exactly, which is what a hard cgroup limit looks like.

Step 2: measure the multiplier, rather than assuming it. Locally, on the same day's data:

Parquet on disk                      147 MB
pandas, as written                 4,145 MB     28.2x

Step 3: find where it goes. The job reads 34 columns and uses 8:

columns read      34      including 6 free-text description fields
columns used       8

COLS = ["supplier_id", "product_id", "activity_date",
        "units", "revenue_cents", "returns_cents", "region", "channel"]
pd.read_parquet(path, columns=COLS)        # 1,180 MB peak, 8.0x

Selecting eight columns instead of thirty-four took the peak from 4,145 MB to 1,180 MB — a 3.5× reduction from one keyword argument, and the six free-text columns account for most of it, because a description field is where Parquet's dictionary encoding wins hardest and an in-memory string array wins least.

Step 4: measure the alternatives, on the same 147 MB:

Polars-lazy uses more memory than pandas-with-projection here, which is worth pausing on: the multipliers in §22.7 are for §22.6's workload, and this workload is different — fewer columns, more groups. A multiplier is a property of a query, not of an engine, and the honest use of §22.7's table is as a starting estimate to be replaced by a measurement.

The Decision

Three options were on the table.

Raise the limit to 8 GB. One line of YAML, works today.

Add columns= to the read. One line of Python, takes the peak to 1,180 MB.

Move it to DuckDB. Twenty lines of SQL, takes it to 426 MB.

📐 Design Decision — the 8 GB option was rejected, and the reason is not thrift

Raising the limit costs almost nothing on this cluster. It was rejected anyway, and the argument is worth having in full because "just give it more memory" is correct often enough to be a habit.

One: it is a fixed-term fix with an unknown expiry. The job crossed 4 GB after two years of 3% annual growth. At the same rate it crosses 8 GB in about twenty-four years — but the growth rate is not the constraint; a schema change is. One new column on agg_customer_daily, and a job reading all thirty-four is back at the limit next quarter with no warning.

Two: the container is shared with eleven other jobs. Raising one job's limit changes what fits alongside it, and the eleven neighbours did not participate in the decision.

Three: the job did not need the memory. It read 34 columns to use 8. Granting the request would have institutionalized reading 26 columns for no reason, forever, in a job nobody would look at again.

The test worth applying: "does this job need what it is asking for?" If the answer is no, raising the limit is not resourcing — it is deferring, with interest.

And the honest counterweight: if the answer had been yes — if the job genuinely needed 5 GB to do necessary work — raising the limit would have been correct and the twenty lines of DuckDB would have been a waste of a morning. The mistake is not raising limits. It is raising them without asking.

They did both of the others. columns= shipped that afternoon, because it was one line and restored service. The DuckDB rewrite shipped the following week.

COPY (
  SELECT supplier_id, product_id,
         SUM(units)                    AS units,
         SUM(revenue_cents)::BIGINT    AS revenue_cents,   -- §22.9
         COUNT(DISTINCT activity_date) AS days_active
    FROM read_parquet('gold/agg_customer_daily/**/*.parquet')
    JOIN read_parquet('gold/dim_supplier.parquet') USING (supplier_id)
   WHERE activity_date >= current_date - INTERVAL 90 DAY
   GROUP BY 1, 2
) TO 'gold/supplier_report' (FORMAT PARQUET, PARTITION_BY (supplier_id));

The filter is inside the query, so DuckDB pushes it into the Parquet read and skips the row groups that cannot match. The projection is implicit — the query names its columns, so there is no 34-versus-8 decision to get wrong.

And a memory assertion, because the underlying problem was that nothing measured this:

# Fails the build if any job's peak exceeds 60% of its container limit.
# Not 95%: the point is to find out before a busy Monday does.
assert peak_rss_mb < 0.60 * container_limit_mb, (
    "peak %.0f MB is %.0f%% of the %d MB limit"
    % (peak_rss_mb, 100 * peak_rss_mb / container_limit_mb, container_limit_mb))

What Happened

The audit of the other eleven jobs on the same worker found three at over 60% of the limit, one of them at 91%. None had failed yet. That one — a monthly reconciliation — would have failed in approximately five weeks, on a month-end, which is the worst possible time for a report nobody can debug.

Two of the three were fixed with columns= alone.

The 137 was added to the runbook, with the three-line diagnostic and one instruction in bold: before investigating the platform, check Reason: OOMKilled. The three days spent were not recoverable; the next occurrence took eleven minutes.

🏭 From the Pipeline — the memory nobody measures

Every team measures job duration. Almost none measure job memory, and the asymmetry has a reason: duration is in every scheduler's UI by default and memory is not.

The consequence is that memory is discovered, not managed. A job's memory profile is unknown until the day it exceeds a limit, at which point the discovery arrives as a SIGKILL with no traceback.

Publish peak RSS per task run, as a first-class metric, next to duration. It costs one field from a cgroup file. Then:

• Alert at 60% of the limit, not 90%, because the gap between them is where a busy Monday lives.
• Trend it. A job whose peak grows 3% a quarter is a job with a date on it, and you can compute the date.

Chapter 21 Case Study 2's version of this was duration; this is the same argument for memory, and the same sentence covers both: the margin is what you are protecting, and a pass/fail check cannot see it.

Lessons

1. Exit code 137 is SIGKILL, and SIGKILL leaves no traceback because it cannot be caught. In a data job it is almost always the OOM killer.

2. The absence of a traceback is information. It narrows the cause to something outside your process's control, which is a much smaller set.

3. Check Reason: OOMKilled before investigating the platform. Three days versus eleven minutes.

4. An OOM that depends on the data looks intermittent and is not random. Busy days cross the limit first.

5. pandas reads every column unless you name them. 34 columns for 8 used, and the six free-text fields dominated — the exact columns where Parquet's encoding wins most and memory wins least. columns= was a 3.5× reduction from one keyword.

6. pandas has no optimizer, so a filter after a read is a filter after the whole read. The 90-day window was applied to two years of loaded rows.

7. A memory multiplier is a property of a query, not of an engine. Polars-lazy used more than pandas-with-projection on this workload while using far less on §22.6's. Use the table as a starting estimate; measure your own.

8. "Just give it more memory" is correct often enough to be a habit. The test is "does this job need what it is asking for?" — and the mistake is not raising limits, it is raising them without asking.

9. Raising a shared container's limit is a decision eleven other jobs did not participate in.

10. Publish peak memory next to duration, and alert at 60% of the limit. The gap between 60% and 90% is where a busy Monday lives.

11. A job whose memory grows 3% a quarter has a date on it, and you can compute the date. Three other jobs on the same worker did; one was five weeks out.

Questions for Discussion

1. Three days went into the platform because the code had not changed and the failure was intermittent. Both facts were true. What would have redirected the investigation on day one, and what would that have cost to have in place?

2. The 60% alert threshold is arbitrary. Defend a different number. What is the cost of 40%, and of 80%?

3. The team declined to raise the limit partly because the job did not need the memory. How would you apply that test to a request where the answer is genuinely unclear?

4. Polars-lazy used more memory than pandas here and far less in §22.6. What does that do to the usefulness of §22.7's table, and how should a book present a number like that?

5. The job was promoted from a notebook in a week and untouched for two years. §22.12's checklist has four lines. Which of the four would have caught this, and would you have written it?

6. Duration is in every scheduler's UI and memory is not. Why do you think that is, and what would it take to change in your own tooling?

7. One of the other eleven jobs would have failed at month-end in five weeks. How many jobs in your systems have a computable failure date, and would you rather know?