Case Study 2: Evaluating and Improving RAG Quality

Chapter 31: Retrieval-Augmented Generation (RAG)

Overview

Company: MedAssist Health, a digital health startup providing AI-powered clinical decision support to 200 physicians. Challenge: The RAG system powering their clinical reference tool exhibited inconsistent answer quality -- high accuracy for drug interaction queries but poor performance for complex diagnostic reasoning and treatment guideline synthesis. Goal: Systematically evaluate RAG quality, identify failure modes, and implement targeted improvements to achieve 90%+ faithfulness and 85%+ answer relevance across all query types.

Problem Analysis

MedAssist's initial RAG system was built rapidly during a prototype phase. While physicians found it useful for simple lookups, several quality issues emerged after three months of production use:

1. Hallucinated dosages: In 4% of responses, the system generated medication dosages not present in retrieved documents.
2. Incomplete answers: Complex queries requiring synthesis across multiple guidelines received partial answers 23% of the time.
3. Irrelevant retrieval: 18% of retrieved chunks were topically related but did not contain the information needed to answer the query.
4. Contradictory sources: When retrieved chunks contained conflicting information (e.g., different editions of guidelines), the system did not flag the contradiction.

A manual audit of 500 queries revealed the following quality distribution:

The 7% dangerous response rate was unacceptable for a clinical tool.

Evaluation Framework

RAGAS-Inspired Metrics

The team implemented a comprehensive evaluation framework measuring four dimensions:

1. Faithfulness: Does the answer contain only information supported by retrieved context?

$$\text{Faithfulness} = \frac{\text{Number of claims supported by context}}{\text{Total number of claims in answer}}$$

2. Answer Relevance: Does the answer address the question asked?

$$\text{Answer Relevance} = \text{mean}(\cos(\text{emb}(q), \text{emb}(a_i)))$$

where $a_i$ are synthetic questions generated from the answer and $q$ is the original query.

3. Context Precision: Are the retrieved chunks relevant to the query?

$$\text{Context Precision} = \frac{1}{K} \sum_{k=1}^{K} \frac{\text{relevant chunks at rank} \leq k}{k} \cdot \mathbb{1}[\text{chunk}_k \text{ is relevant}]$$

4. Context Recall: Does the retrieved context contain all information needed to answer?

$$\text{Context Recall} = \frac{\text{Number of ground truth claims covered by context}}{\text{Total ground truth claims}}$$

Evaluation Dataset

The team curated 300 query-answer pairs across three difficulty tiers:

Baseline Results

The results confirmed the pattern: good performance on simple queries but significant degradation on moderate and complex queries.

Improvement Iterations

Iteration 1: Improved Chunking

Problem identified: The original chunking strategy used fixed 512-token chunks, which frequently split clinical guidelines mid-paragraph, separating dosage information from its associated conditions.

Solution: Recursive chunking with section-aware splitting. Chunks respect heading boundaries and include the section hierarchy as metadata.

Iteration 2: Hybrid Search with Reranking

Problem identified: Dense retrieval missed queries containing specific drug names, ICD codes, and medical abbreviations that required exact lexical matching.

Solution: Hybrid search combining dense retrieval (weight 0.7) with BM25 keyword search (weight 0.3), followed by a cross-encoder reranker.

Iteration 3: Query Decomposition

Problem identified: Complex queries required information from multiple clinical topics (e.g., drug interactions + renal dosing + contraindications). A single embedding could not capture all facets.

Solution: For queries classified as complex, decompose into sub-queries using an LLM, retrieve for each sub-query independently, and merge results with deduplication.

Iteration 4: Faithfulness Guardrails

Problem identified: Even with better retrieval, the LLM occasionally generated unsupported claims, especially about dosages and contraindications.

Solution: A post-generation verification step that extracts factual claims from the response and checks each against the retrieved context using NLI (natural language inference). Unsupported claims are flagged or removed.

Final Results

Latency Impact

The additional latency from query decomposition and faithfulness checking was acceptable for the clinical use case, where correctness outweighs speed.

Key Lessons

1. Evaluation before optimization. Without a structured evaluation framework, the team had been making changes based on anecdotal feedback. The RAGAS-inspired metrics provided objective guidance on where to focus effort.

2. Chunking matters more than model choice. Switching from fixed to section-aware chunking produced a larger quality improvement than upgrading the embedding model from MiniLM to BGE-large.

3. Hybrid search is essential for technical domains. Medical terminology, drug names, and codes require exact lexical matching that pure dense retrieval misses. The 70/30 dense/BM25 split was found through grid search.

4. Query decomposition unlocks complex reasoning. The single largest improvement on complex queries came from breaking them into sub-queries. This is equivalent to giving the retriever multiple "searches" for a single user question.

5. Post-generation verification is a safety requirement. For high-stakes domains, relying solely on the LLM to be faithful is insufficient. An independent verification step using NLI reduced dangerous responses by 90%.

6. Evaluation is continuous, not one-time. The team established a weekly evaluation pipeline on a rotating subset of queries, catching regressions early as the document corpus evolved.

Code Reference

The complete implementation is available in code/case-study-code.py.