Can Haystack be used for multi-modal search with images and text together?

All answers to this question.

Yes, it’s entirely possible. In Haystack 2.0, you can use the MultiModalRetriever or create a custom component using a model like CLIP. The workflow involves creating a document store where each document contains both a text description and an image path or blob. During indexing, you use CLIP to generate embeddings for both. When a user uploads a photo, you embed that photo and search the vector space. The framework handles the "tensors" just as it does for text. We built a prototype for a fashion brand recently, and the ability to find "dresses with this specific floral pattern" by uploading a screenshot was incredibly impressive to the client.

   Answered 2025-03-07 by Patricia Adams

How do you handle the metadata for the images? If I want to filter by "price" or "brand" while also doing an image search, is that possible in the same pipeline?

   Answered 2025-03-10 by Steven Wright

• Steven, absolutely. You can build a powerful AI search engine using Haystack that uses "Metadata Filtering." Most retrievers in the library allow you to pass a filters dictionary. So, your search would first filter the document store for "Brand: Nike" and "Price < 100", and then perform the vector similarity search on only those filtered images. This hybrid approach—combining structured metadata and unstructured image data—is where the library really shines for e-commerce.

     Commented 2025-03-12 by Robert Collins

Make sure you use a modern vector database like Qdrant or Milvus for this, as they have much better native support for handling large multi-modal vectors.

   Answered 2025-03-15 by Jennifer Harris

• Good advice, Jennifer. Using a specialized vector DB makes the scaling process for millions of product images much more manageable.

     Commented 2025-03-16 by Sarah Jenkins