How does Haystack handle large-scale PDF indexing for AI search?

All answers to this question.

The standard PDF converter in Haystack is good for plain text, but for complex tables, it often loses the structural context. For our engineering documents, we ended up using the AzureConverter or the PyMuPDF integration which handles layout much better. If you have "image-only" PDFs, you'll definitely need to use the OCRPreProcessor. Another pro tip: for tables, try converting them into Markdown format during the ingestion phase. LLMs and retrieval models find it much easier to understand structured Markdown than a jumbled mess of text that used to be a table.

   Answered 2025-09-05 by Brenda Simmons

Have you looked at the new MarkdownConverter in Haystack 2.0? It’s designed specifically to help with these structural issues when you use tools like Unstructured.io as a frontend for your PDF parsing.

   Answered 2025-09-10 by Scott Peterson

• Scott, I haven't tried the MarkdownConverter yet! Does it require a separate license for Unstructured.io, or can it be run locally? We have some very sensitive data in these PDFs, so I’d prefer a solution that doesn't send the raw files to an external API for parsing if possible.

     Commented 2025-09-25 by George Russell

For 50k documents, make sure you use the Batch mode for indexing. It saves so much time on the embedding generation compared to processing files one by one.

   Answered 2025-09-28 by Carol Alexander

• Great advice, Carol. Batching is essential for scale. We also found that using a local GPU for the embedding phase cut our indexing time by 70%.

     Commented 2025-09-29 by Ronald Gray