Can I use Instructor to extract entities from multi-modal inputs like images?

All answers to this question.

Absolutely, and it’s a game-changer for document processing. You just pass the image URL or base64 data in the messages array as you normally would with the OpenAI SDK. Since the library patches the client, you still just add response_model=InvoiceSchema. The model looks at the image and fills out the Pydantic fields. I’ve used this for medical forms and it’s surprisingly accurate at catching table data that traditional OCR often messes up. The best part is that you can add validation logic to check if the extracted total amount equals the sum of the line items.

   Answered 2025-01-06 by Heather Walker

How does the validation retry logic work with vision—does it resend the entire image for every retry, and does that double the cost?

   Answered 2025-01-07 by Justin Clark

• Justin, it does resend the image because the model needs the context to fix its mistake. This can get expensive if your image is high-resolution. My tip is to downscale the image to the minimum required for legibility before sending. Also, keep your schema simple to minimize the chance of a retry being needed in the first place.

     Commented 2025-01-08 by Deborah Hall

Going from raw pixels to a validated Python object in one step feels like magic every time I run it.

   Answered 2025-01-09 by Amy Young

• It really does, Amy. It’s significantly faster to develop than the old Tesseract-based pipelines we used to build.

     Commented 2025-01-10 by Heather Walker