Case Study 2: Six Thousand Invisible Products

"Nothing failed. Six thousand products just stopped appearing in 'similar items,' and absence does not raise an exception."

Executive Summary

Kestrel upgraded the embedding model behind its similarity search from 768 to 1,536 dimensions. The re-embedding job processed 41,000 of 47,000 products, failed on one malformed description, was retried, and completed the rest.

The index now held vectors from two model versions. Because the dimensions differed, the store rejected the mismatched ones outright — so 6,000 products were silently absent from every similarity search while remaining perfectly present in the catalogue, in search, and on their own pages.

It was found five weeks later when a merchandiser asked why a bestselling jacket never appeared under "similar items."

This case study is about derived data with a versioned generator, which is a shape that recurs well beyond vector stores — and about the fact that absence is the hardest failure to detect, because nothing errors and no count goes down that anyone is watching.

Skills applied: vector stores as a pipeline (§12.7); derived data and coverage assertions (§12.7); silent failure (Chapter 1 §1.2); version flags with the data (Chapter 6, Case Study 1).

Background

The feature. "Similar items" on every product page, and a "customers who liked this" module. Both run a nearest-neighbour query against product embeddings in pgvector, then join to products for price, stock, and imagery.

The pipeline, as it existed:

products (Postgres)
   │  nightly, changed products only
   ▼
embed_products.py  ──▶  embedding model API  ──▶  product_embeddings
                                                   (product_id, embedding vector(768))

47,000 products. About 400 change on a typical day.

The upgrade. A newer embedding model, 1,536 dimensions, measurably better on the team's relevance benchmark. The plan was sound: alter the column, re-embed everything, done.

ALTER TABLE product_embeddings ALTER COLUMN embedding TYPE vector(1536);

python embed_products.py --all --model text-embedding-v3

The Problem

The job ran for about four hours and then:

2025-08-14 02:41:07 INFO  embedded 41,000 / 47,000
2025-08-14 02:41:09 ERROR product 31882: model API returned 400
                          (input contains invalid UTF-8 sequence)
Traceback (most recent call last):
  ...
ValueError: embedding request failed for product 31882

The on-call engineer saw the failure, saw that 41,000 had succeeded, and did the obvious thing:

python embed_products.py --all --model text-embedding-v3 --skip-errors

It completed. --skip-errors skipped product 31882 and, because the job's "changed products" logic had already marked the first 41,000 as done in a checkpoint table, it also skipped them. The second run embedded roughly 5,999 products.

Total: 41,000 + 5,999 = 46,999 of 47,000. Which looks correct.

It was not, and the reason took two days to find.

What actually happened

The ALTER COLUMN had succeeded — but pgvector cannot convert existing 768-dimensional vectors to 1,536 dimensions, and the specific sequence the team used had left the old rows in place with their original vectors in a way the subsequent writes did not fully overwrite. The first run's 41,000 rows were updated correctly. The 5,999 from the second run were correct. The 6,000 products that had been embedded before the upgrade and were not in either run's working set retained 768- dimensional vectors.

Queries specifying a 1,536-dimensional query vector simply did not match them — no error, no warning, and SELECT COUNT(*) FROM product_embeddings returned 47,000, which is why every check anyone thought to run said the table was fine.

⚠️ Failure Mode — Absence does not raise an exception

This is the hardest class of data failure to detect, and it is worth stating in general form.

A wrong value can be caught by a range check. A missing row can be caught by a count. An item that is silently excluded from a result set is caught by neither, because the result set is supposed to be a subset — that is what a similarity search is.

Three properties made this invisible for five weeks:

  1. No error. Every query succeeded and returned plausible results.
  2. No count changed. The table had 47,000 rows throughout.
  3. Nobody is watching for what did not appear. A recommendation module with ten items looks identical whether the eleventh-best candidate was excluded for relevance or for a dimension mismatch.

The general defense is a coverage assertion: not "does this table have rows," but "does every entity that should be represented actually appear in the derived structure, in a usable form."

sql -- The nine words of SQL that would have caught it on day one. SELECT COUNT(*) FROM products p WHERE p.active AND NOT EXISTS ( SELECT 1 FROM product_embeddings e WHERE e.product_id = p.product_id AND e.model_version = :current_model -- the column that did not exist ); -- expect 0

The check requires one thing the schema did not have: the model version stored with the vector. Without it, there is no way to ask "is this row usable by the current query path."

The Analysis

Detection, five weeks late. A merchandiser asked why the Ridgeline Field Jacket — consistently in the top twenty by revenue — never appeared under "similar items" for anything.

Day 1: is it a relevance problem? The natural first hypothesis. The team looked at the ANN index parameters, the distance metric, and the candidate pool size. All fine.

Day 2: is the product in the index? It was — SELECT * FROM product_embeddings WHERE product_id = ... returned a row.

The answer came from one query that nobody had thought to run:

SELECT vector_dims(embedding) AS dims, COUNT(*)
  FROM product_embeddings
 GROUP BY 1;

 dims | count
------+--------
  768 |  6,001
 1536 | 40,999

Two dimensionalities in one column, which pgvector permits when the column type was altered in a way that did not force a rewrite of every row.

🔎 Read the Plan — Group by the thing you assume is constant

The diagnostic that solved this in ten seconds is a pattern worth internalizing:

GROUP BY the property you believe is uniform, and count.

sql SELECT vector_dims(embedding), COUNT(*) FROM product_embeddings GROUP BY 1; SELECT model_version, COUNT(*) FROM product_embeddings GROUP BY 1; SELECT pg_typeof(price), COUNT(*) FROM staging_prices GROUP BY 1; SELECT length(sku), COUNT(*) FROM products GROUP BY 1; SELECT substring(email, '@.*'),COUNT(*) FROM customers GROUP BY 1 ORDER BY 2;

Every one of these asks "is this actually as uniform as I think?" and the answer is surprisingly often no. It takes seconds and it finds the class of problem where the schema permits variation you did not intend — which no type check catches, because the type is satisfied.

This is the same instrument as Chapter 1's "is order_item_id unique?" — the dumbest available question about the data, asked first.

The Decision

Four changes, and the first is the one that generalizes furthest.

1. Version the index, not the rows.

# Build the new index ALONGSIDE. Switch readers only when it is complete.
CREATE TABLE product_embeddings_v3 (
    product_id     BIGINT PRIMARY KEY,
    embedding      vector(1536) NOT NULL,
    model_version  TEXT         NOT NULL,     -- stored WITH the vector
    embedded_at    TIMESTAMPTZ  NOT NULL,
    source_hash    TEXT         NOT NULL      -- hash of the text embedded
);

Populate fully, assert coverage, then flip the reader — a view, or a configuration value. Same pattern as Chapter 9 §9.6's compaction swap and Chapter 10 §10.3's atomic commit, and for the same reason: a partial state must never be reachable by a reader.

2. model_version and source_hash stored with every vector. The version makes the coverage check expressible. The source hash makes incremental re-embedding correct: re-embed when the hash of the source text changes, which is exact, rather than when updated_at moves, which fires on irrelevant column changes and misses direct SQL updates (Chapter 2 §2.2).

3. A coverage assertion, run after every embedding job and nightly:

-- fails the job, not just an alert
SELECT COUNT(*) AS missing
  FROM products p
 WHERE p.active
   AND NOT EXISTS (SELECT 1 FROM product_embeddings_v3 e
                    WHERE e.product_id = p.product_id
                      AND e.model_version = :current_model);
-- assert missing = 0

SELECT COUNT(DISTINCT model_version) FROM product_embeddings_v3;
-- assert = 1

4. --skip-errors was removed and replaced with --quarantine, which writes failures to a table with the reason and still fails the job. The original flag's problem was not that it skipped errors; it was that it made the job succeed while incomplete.

📐 Design Decision — Fail the job, or complete with a quarantine?

The real argument was about change 4, and both positions have force.

Complete-with-quarantine: one bad product should not block 46,999 good ones. An embedding refresh that fails entirely because of one malformed description is fragile, and the on-call engineer at 02:41 will reach for --skip-errors anyway.

Fail the job: an incomplete derived structure is silently wrong, and this incident is what that costs.

The resolution keeps both: the job processes everything it can, quarantines what it cannot with a reason, and then exits non-zero if the quarantine is non-empty. The data is as complete as possible; the job is unambiguously red; the coverage assertion tells you exactly what is missing.

What this costs: a red job every time one product has a bad description, which is roughly monthly, and a red job that people learn to ignore is worse than no job. The mitigation is that the failure names the specific product and the reason, so clearing it is a two-minute task rather than an investigation.

The general rule: a job may be partially successful; it may not be quietly partially successful.

What Happened

The v3 index was rebuilt over one night with the new schema and the coverage assertion. It failed the first time — on 214 products, all of which had been embedded from a description field that was null, producing an empty-string embedding request the API rejected.

Those 214 had been missing from similarity search since the feature launched two years earlier. Nobody had ever noticed, because absence does not raise an exception.

Since then:

  • The coverage assertion has failed nine times. Seven were new products with missing descriptions — a content problem, routed to merchandising rather than to engineering. Two were genuine pipeline failures.
  • The GROUP BY uniformity check was added to the platform's standard diagnostics and has since found two unrelated problems: a sku column with two distinct length patterns after an acquisition, and a staging table with mixed timestamp precisions.
  • The source-hash approach reduced nightly embedding volume by about 70%, because updated_at had been firing on stock-level changes that do not affect the embedded text.

That last one is worth noting: the fix for a correctness problem produced a cost improvement nobody had asked for, because updated_at had been a poor proxy for "the thing I care about changed" all along.

Lessons

  1. Absence does not raise an exception. No error, no count change, and nobody watching for what did not appear. It is the hardest data failure to detect.

  2. A coverage assertion is the general defense — not "does the table have rows," but "does every entity that should be represented appear, in a usable form."

  3. Store the version with the derived data. Without model_version, the coverage question is inexpressible.

  4. Version the index, not the rows. Build alongside, assert, flip. Same pattern as compaction and as atomic commit, for the same reason: a partial state must not be reachable.

  5. GROUP BY the property you assume is uniform. Ten seconds, and it finds the class of problem where the schema permits variation you did not intend.

  6. A job may be partially successful; it may not be quietly partially successful. Quarantine with a reason, and exit non-zero.

  7. --skip-errors made a job succeed while incomplete, which is the actual defect. Skipping was fine.

  8. A hash of the source is a better change signal than updated_at — exact rather than approximate, and here it cut work by 70% as a side effect of fixing correctness.

Questions for Discussion

  1. The on-call engineer used --skip-errors at 02:41 and it was the reasonable thing to do. What should the flag have done instead, and how would the engineer have known?

  2. 214 products had been missing since launch and nobody noticed for two years. What does that suggest about how much you should trust "nobody has complained" as evidence?

  3. The coverage assertion now fails roughly monthly for content reasons and routes to merchandising. Is a job that is red monthly sustainable? What would you change?

  4. source_hash is exact where updated_at is approximate. Where else in this book would a content hash be a better change signal, and what does it cost?

  5. The team flipped readers via a view. What are the failure modes of that switch, and how would you test it before the flip?

  6. This incident and Chapter 6's promotion_attribution_method share a shape — derived data whose provenance was not recorded. Name a third instance from this book, and state the general rule.

  7. Fixing correctness cut embedding volume 70%. How would you look for other places where a correctness fix has an unclaimed efficiency benefit?