Can DSPy be used with non-OpenAI models like Anthropic or local LLMs?

All answers to this question.

This is actually where DSPy shines the most. Because the prompts are generated by the compiler rather than being hard-coded by you, the framework is naturally model-agnostic. You can define your program logic once and then compile it for Claude 3.5 Sonnet, and then re-compile it for a local Llama 3 instance. The compiler will pick different few-shot examples and different phrasing that work best for each specific model's architecture. We are currently running a hybrid setup where we use DSPy to optimize our local models to perform nearly as well as the top-tier cloud models for our specific summarization tasks.

   Answered 2025-04-20 by Rachel Foster

When you re-compile for a smaller local model in DSPy, do you find that the instructions become much longer to compensate for the lower reasoning capability?

   Answered 2025-04-25 by Jeffrey Reed

• Often, yes! The optimizer might realize that the smaller model needs more explicit, step-by-step guidance or more diverse examples to reach the same conclusion. That’s the beauty of DSPy—it does that tedious trial-and-error for you. You just provide the metric and the data, and the framework figures out the best way to talk to that specific model.

     Commented 2025-04-28 by Kevin Parker

It works great with vLLM for high-throughput serving. The separation of the "program" from the "parameters" makes it easy to swap backends in seconds.

   Answered 2025-05-05 by Melissa Gardner

• That flexibility is key for enterprise. Not being locked into one API vendor gives you so much more leverage when it comes to cost and privacy.

     Commented 2025-05-07 by Rachel Foster