Optimizing RAG performance: Should I focus on the retrieval prompt or the embedding model?

All answers to this question.

If your retrieval is already accurate (meaning the right context is being injected), then the bottleneck is almost certainly your "Synthesis Prompt." Most developers provide too much raw text and tell the AI to "summarize." Instead, use "Information Extraction" prompts. Tell the model to first "Identify key entities," then "Filter out irrelevant noise," and finally "Formulate the response using the following schema." I spent months on this in 2023; the prompt structure usually has a much higher ROI than fine-tuning embeddings unless your domain is extremely niche.

   Answered 2025-12-13 by Brenda Collins

Does the size of the chunks you are retrieving impact how you should structure that synthesis prompt? I’ve noticed the model gets confused if the context window has 10+ small chunks.

   Answered 2025-12-15 by Victor Stone

• Victor, absolutely. "Lost in the Middle" is a real phenomenon where LLMs ignore the middle part of long contexts. I handle this by prompting the model to "Re-rank the retrieved chunks by relevance before answering." This internal step makes the model focus on the most important bits first, even if the total context is quite large.

     Commented 2025-12-16 by Brian Kelly

Try adding a "Step-Back" prompt. Ask the AI to identify the underlying concept of the user's query before it looks at the retrieved documents to answer.

   Answered 2025-12-17 by Diana Prince

• That "Step-Back" technique is underrated. It helps the AI maintain the "big picture" intent instead of getting bogged down in the specific wording of a single document chunk.

     Commented 2025-12-18 by Brenda Collins