What is the best way to stream partial structured data using Instructor?

All answers to this question.

You should look into the Partial type provided by the library. When you wrap your model in Partial[MyModel], the library uses a specialized parser that can handle incomplete JSON strings. You then use stream=True in your completion call and iterate through the generator. Each iteration yields a Pydantic object with whatever fields have been populated so far. I used this for a travel itinerary generator where the flight details appeared instantly while the hotel descriptions were still being "typed" by the model. It makes the app feel incredibly fast.

   Answered 2025-05-26 by Cynthia Roberts

Can we use this streaming feature with local models like Llama 3 running through Ollama, or is it restricted to the OpenAI API?

   Answered 2025-05-27 by Patrick Harris

• Patrick, it definitely works with local models! As long as the provider supports the OpenAI-compatible streaming protocol, the library can patch it. I’ve personally tested it with Ollama and it works great for extracting structured data from local inference without the latency of a full round-trip.

     Commented 2025-05-28 by Matthew Wilson

The user experience improvement is massive when you aren't staring at a loading spinner for ten seconds.

   Answered 2025-05-29 by Brian Anderson

• Spot on, Brian. For long extractions, streaming partial results is the only way to keep users engaged.

     Commented 2025-05-30 by Cynthia Roberts