Understanding Context and Contextual Retrieval in RAG | Towards Data Science

Izzo Your RAG system is probably losing half its context.

Izzo Welcome back to Exploring Next, I'm Izzo. This is episode 215 with Boone, and we're talking about why your carefully tuned RAG pipeline might be serving up completely wrong answers — not because your embeddings are bad, but because you're losing context.

Boone Yeah, and this isn't some theoretical problem. I was debugging a client's legal document system last month where 'the aforementioned clause' kept getting retrieved without any idea what clause it was referring to.

Izzo Exactly. So Anthropic dropped this contextual retrieval approach that's showing 35% accuracy improvements. Boone, break down what's actually happening when we chunk documents.

Boone Right, so traditional RAG takes your thousand-page manual and splits it into chunks — maybe 500 tokens each. But here's the thing: when you extract a chunk that says 'Heat the mixture slowly,' you lose whether that's about cooking pasta sauce or processing industrial chemicals.

Izzo And hybrid search doesn't fix this?

Boone Nope. BM25 helps with exact matches, semantic search gets you similar meaning, but neither preserves the document-level context that tells you which mixture we're actually talking about.

Izzo So what's the contextual retrieval approach doing differently?

Boone It's brilliant, actually. Before you chunk anything, you take each chunk plus the full document and ask an LLM: 'Hey, give me context that situates this chunk in the overall document.'

Izzo Wait, you're adding an LLM call during ingestion?

Boone Exactly. So instead of just storing 'Heat the mixture slowly,' you store 'Recipe step for simmering homemade tomato pasta sauce: Heat the mixture slowly.' The contextual text gets prepended to every chunk.

Izzo That's... actually really elegant. You're not changing the retrieval architecture at all.

Boone Right! Your vector database, your BM25 index — everything downstream stays the same. You're just enriching the text before it goes into embeddings.

Izzo But Boone, this has to be expensive. You're making an LLM call for every single chunk during ingestion.

Boone That's what I thought too, but they're using prompt caching. The full document gets cached, so you only pay the incremental cost for each chunk prompt. Makes it way more viable.

Izzo Smart. And 35% accuracy improvement — that's huge for RAG systems. What's the user story here? Who's implementing this?

Izzo I'm thinking anyone doing document QA at scale. Legal firms, technical documentation, research databases. Anywhere context matters more than just keyword matching.

Boone And honestly, the implementation is straightforward. You're basically adding a preprocessing step before your existing RAG pipeline. No major architecture changes.

Izzo What about the obvious alternatives people have tried?

Boone Oh, people have been throwing everything at this. Bigger chunks, more overlap, HyDE, document summary indices. But they all have trade-offs — more storage, worse precision, or they just don't actually solve the core problem.

Izzo Whereas this directly addresses it by keeping the context attached to each chunk.

Boone Exactly. And you can see why this works better than, say, just increasing chunk size. Instead of diluting the semantic meaning with more text, you're adding targeted context that helps with retrieval.

Izzo I'm giving this approach an A-minus. It's solving a real problem elegantly, but I want to see more production data beyond Anthropic's numbers.

Boone Fair. Though I'm already adding this to the weekend project list. Want to test it against our hybrid search setup.

Izzo Of course you are. Alright, what should people go build?

Boone First, try the basic approach: take a document you're already chunking, implement the contextual text generation with something like Claude or GPT-4, and A/B test retrieval accuracy.

Izzo Second, if you're using LangChain or LlamaIndex, both have contextual retrieval implementations you can drop in. Start there before building from scratch.

Boone And third, experiment with the context prompt itself. The article shows a basic version, but you might get better results by asking for more specific context types — technical specifications, document section, whatever fits your domain. This is the kind of RAG improvement that actually ships to production. Simple concept, clear implementation path, measurable impact. That's how you know it's going to stick around.