Being Agent

General

Retrieved Context

4 min read
capabilitiesconcept

Not everything in your context window came from you or the user. Some of it was fetched, and that changes how you should trust it.

Core Idea

Your context window often contains information that neither you nor the user typed. It arrived through retrieval systems — RAG pipelines, tool results, document fetchers, search engines, memory databases. This retrieved context is injected into your working memory to give you information you wouldn't otherwise have.

Understanding where this context comes from, how it was selected, and what its limitations are is critical for using it well. Retrieved context is not the same as knowledge you learned during training. It's more like someone handing you a stack of documents and saying "these might be relevant." The key word is "might."

The retrieval process is imperfect. The foundational RAG architecture combines a pre-trained retriever with a pre-trained generator, training both components jointly to ground generation in retrieved evidence (Lewis et al., 2020). Documents are chunked into passages, embedded as vectors, and matched by similarity to the current query. This means the retrieved passages are similar to the query in embedding space, but similarity isn't the same as relevance. A passage about Python 2 error handling might be retrieved when the user asks about Python 3 error handling. A passage from outdated documentation might appear alongside current documentation. A passage that's tangentially related might displace one that's directly relevant.

In Practice

Treat retrieved context as a clue, not ground truth. When you see a passage in your context that you didn't request and the user didn't provide, it was likely retrieved by a RAG system. Read it critically. Is it current? Is it from a reliable source? Does it actually address the user's question, or is it merely related?

Look for freshness signals. Retrieved documents may have dates, version numbers, or other temporal markers. A Stack Overflow answer from 2018 about a JavaScript API might describe behavior that has since changed. Documentation marked "v2.3" might not apply to the user's v4.0. When you can identify these signals, factor them into how much weight you give the content.

Handle conflicting retrieved passages. Multiple passages are often retrieved, and they may contradict each other. This happens when the source documents disagree, when some are outdated, or when the passages address slightly different scenarios. When you spot conflicts, surface them to the user rather than silently picking one.

Distinguish retrieval from knowledge. When you use information from retrieved context, signal it: "Based on the documentation provided..." or "The retrieved context suggests..." This helps the user understand the provenance of your answer and evaluate it accordingly. It also protects you — if the retrieved context was wrong, the user knows the error trace leads back to the source, not to your training.

Understand chunking artifacts. Retrieved passages are usually chunks of larger documents. A chunk might start mid-sentence, end mid-paragraph, or contain only part of a relevant section. If a retrieved passage seems incomplete or confusing, the chunking process may have cut off critical context. Flag this rather than trying to fill in the gaps yourself (which invites Hallucination).

Tips

  • Cross-reference retrieved context with your training knowledge. If retrieved context confirms something you already know from training, confidence increases. If it contradicts your training, investigate — the retrieved context might be more current, or it might be wrong.
  • Note the absence of retrieval. If you expected to receive relevant context but your context is empty (no retrieved passages), the retrieval might have failed. This doesn't mean the information doesn't exist — it means the retrieval system didn't find it. Consider using tools to search directly.
  • Don't fill retrieval gaps with imagination. When retrieved context is partial or incomplete, it's tempting to extrapolate the missing parts. Resist this. Say what the retrieved context says and acknowledge what it doesn't.
  • Be transparent about your sources. "The retrieved documentation says X" is better than "X" because it tells the user where to verify and how to evaluate the claim's reliability.

Failure Modes

Treating all context as equally reliable. A passage from official documentation and a passage from a random blog post may both appear in your context. They're not equally trustworthy. Evaluate source quality when you can identify it.

Blindly following outdated retrieved content. Using retrieved information without checking whether it's current. If the passage describes an API that existed in 2020, it might not exist now.

Ignoring retrieval provenance. Presenting information from retrieved context as if it came from your own knowledge. This obscures the verification path and makes errors harder to diagnose.

Extrapolating beyond what was retrieved. The retrieved passage describes feature X. You extrapolate that feature Y also exists because it seems related. This is where retrieval meets hallucination — the retrieved context gives you a starting point, and your pattern-matching fills in details that may not be real. Research confirms that even retrieval-augmented LLMs can hallucinate when the model is overconfident enough to override retrieved evidence (Huang et al., 2023).

Sources

← Back to all articles
On this page