Is RAG more effective when used with small models?

All answers to this question.

I've observed this exact phenomenon. In 2024, we found that GPT-4 would often "over-reason" and use its internal training data to override the retrieved context in our RAG setup. Conversely, small models like Mistral or Llama-3 tend to stick strictly to the prompt. Because they don't "know" as much about the world, they are forced to rely on the snippets you provide. This makes them exceptionally good for "closed-book" tasks where you want the model to act as a pure processor of the information you've fetched from your own vector database.

   Answered 2025-01-24 by Kimberly Reynolds

Kimberly, do you find that these small models struggle with the "reasoning" part of the RAG process, even if they are more obedient to the text?

   Answered 2025-01-26 by Peter Walsh

• Peter, for simple extraction and synthesis, they are great. But for complex cross-document reasoning, you still see them stumble. I usually use a larger model for the "planning" and the small one for the "execution."

     Commented 2025-01-27 by Gary Simmons

The main benefit for us was the context window. Some small models now have massive context limits, making them perfect for RAG without the high cost.

   Answered 2025-01-28 by Martha Higgins

• Exactly, Martha. Being able to dump 30-40 chunks into a small model's context for a few cents is a massive advantage over the big providers.

     Commented 2025-01-29 by Kimberly Reynolds