Post
5 RAG Failure Modes in Production
March 15, 2024· 1 min read
The Problem
Most RAG systems look great in a notebook. In production, they drift, hallucinate citations, or return irrelevant chunks. Here are five failure modes I see repeatedly.
1. Stale or Wrong Chunks Retrieved
Your index is updated nightly, but users ask about today’s data. Or the top-k retrieval returns the wrong section because the query is ambiguous. Fix: Define a freshness SLA and add retrieval metrics (e.g. chunk relevance score, recency) to your eval harness.
2. Context Overflow / Truncation
You pack 10 chunks into the context window; the model only “sees” the first 3 and invents the rest. Fix: Log token usage per request and add evals that check whether the model’s answer is grounded in the retrieved text.
3. No Guardrails on Citations
The model cites document X, page Y, but the quote doesn’t exist or is out of context. Fix: Use an LLM-as-judge or rule-based check: “Is this citation present and accurate in the retrieved content?”
4. Prompt Drift
A small change in the system prompt (or a new teammate editing it) tanks accuracy. Fix: Version prompts and run a regression suite before deploy. Observability (e.g. Langfuse) lets you compare runs.
5. Single-Point Failure in the Pipeline
One failing service (embedding, vector DB, LLM) brings down the whole flow. Fix: Timeouts, fallbacks, and health checks per stage. Log latency and error rate per component.
Summary
RAG in production needs: freshness guarantees, retrieval and answer evals, citation checks, prompt versioning, and per-stage observability. Get these in place before scaling traffic.