How to implement a two-stage RAG pipeline with Haystack and a Cross-Encoder?

All answers to this question.

This "Retriever-Ranker" pattern is a classic search best practice. In Haystack, you simply add a TransformersSimilarityRanker (for local models) or a CohereRanker (for API-based) into your pipeline after your retriever. The retriever might pull 20 documents based on fast vector similarity, and then the ranker—which is slower but much more accurate—re-evaluates those 20 and outputs only the top 3. We implemented this for a customer support bot in 2023 and found it reduced our token usage by 60% while increasing the "faithfulness" of the answers because the LLM wasn't getting distracted by irrelevant context.

   Answered 2025-11-24 by Elizabeth Moore

Is it better to use a local BERT-based ranker or pay for an external re-ranking API like Cohere? I’m worried about the latency impact of another API call.

   Answered 2025-11-27 by Jeffrey Clark

• Jeffrey, it’s a trade-off. A local Cross-Encoder like cross-encoder/ms-marco-MiniLM-L-6-v2 is incredibly fast (under 100ms for 20 chunks) and can run on the same GPU as your embeddings. If you're building a powerful AI search engine using Haystack for high-volume traffic, keeping the ranker local is usually the better move for latency. Only go with an API ranker if you need the absolute state-of-the-art accuracy of a massive model and your users can tolerate an extra 400-600ms of wait time.

     Commented 2025-11-29 by Justin Robinson

The Ranker is also great for filtering out documents that are completely irrelevant by setting a minimum score threshold before they hit the LLM.

   Answered 2025-12-01 by Cynthia Lewis

• Exactly, Cynthia. If no document meets the threshold, you can have the agent simply say "I don't know" rather than hallucinating an answer.

     Commented 2025-12-02 by Brandon Lee