How to handle multi-stage retrieval in LlamaIndex to reduce AI hallucinations?

All answers to this question.

To solve hallucinations, you definitely want to use the "Node Postprocessor" module. Step one is initializing your base retriever to pull a larger initial set, say similarity_top_k=20. Step two is adding a CohereRerank or a LLMRerank module to your query engine. This forces the system to look closer at the semantic relevance of the retrieved chunks before the final LLM synthesis. I implemented this for a legal discovery tool in late 2023, and we saw a dramatic drop in "hallucinated" clauses because the second-stage re-ranker was much better at understanding the nuance of the query than a simple vector search.

   Answered 2025-01-10 by Karen Mitchell

Does adding a re-ranking step significantly increase the latency of the response? My users are very sensitive to waiting more than 2 seconds for an answer.

   Answered 2025-01-15 by David Harrison

• David, it does add a bit of overhead, usually around 300-500ms. However, if accuracy is your priority, it’s a necessary trade-off. You can mitigate this by using a smaller, faster cross-encoder model for re-ranking locally instead of calling an external API. In many production cases, a slightly slower but correct answer is always better than a fast, incorrect one that loses user trust.

     Commented 2025-01-17 by Jason Meyers

I highly recommend checking out "Small-to-Big" retrieval. You search small chunks for better match accuracy but send larger parent chunks to the LLM for context.

   Answered 2025-01-19 by Christopher Bell

• I agree, Christopher. That specific feature in LlamaIndex is arguably one of its biggest advantages for maintaining context without cluttering the search.

     Commented 2025-01-20 by Karen Mitchell