How can AI Agents solve the "Long Context" problem in RAG?

All answers to this question.

Agentic RAG transforms retrieval from a "one-shot" keyword search into a conversational discovery process. Instead of just pulling the top 5 chunks and hoping for the best, the agent can look at the initial results and say, "I found a mention of the 2022 tax code here, but I need to see the specific amendment from October." It then generates a new search query to find that specific detail. You implement this by giving the agent a "Retriever Tool." The agent's prompt includes a loop where it evaluates if the current context is sufficient to answer the user's question; if not, it calls the retriever again with a more refined query.

   Answered 2024-05-22 by Dorothy Martinez

Does this significantly slow down the user response time? Doing 3 or 4 retrieval loops must take much longer than a standard RAG call that happens in a single pass.

   Answered 2024-05-23 by Thomas Wilson

• Thomas, it definitely adds latency, usually 5-10 seconds per extra loop. However, for "Deep Research" tasks where accuracy is more important than speed, our users are happy to wait for a correct answer rather than getting a fast but wrong one. Dorothy, the "Retriever Tool" idea is brilliant. We’ve started implementing a "Self-Grader" node where the agent checks its own retrieved chunks for relevance before answering. If the score is too low, it automatically triggers a second search with different parameters.

     Commented 2024-05-25 by Brian Miller

You can also use an agent to "Pre-Process" the documents by summarizing each one. The main agent can then scan these summaries first to decide which full documents are worth a deep dive.

   Answered 2024-05-26 by Susan Taylor

• Susan's "summary-first" approach is a great way to save on token costs. It’s like an agent reading the Table of Contents before diving into the chapters.

     Commented 2024-05-27 by Dorothy Martinez