How to effectively test AI agents built with the Pydantic AI library?

All answers to this question.

The framework comes with a TestModel class that is a lifesaver for unit tests. It allows you to simulate LLM responses without actually hitting an API and spending money. You can pre-define what the model "returns" and then verify that your tools were called with the right arguments and that your dependencies were handled correctly. For the non-deterministic part, we use "Evals" where we run the agent against a set of 50 gold-standard queries and use another LLM to grade the responses based on the Pydantic models we defined.

   Answered 2025-01-15 by Cynthia Nelson

Does TestModel support testing the retry logic? Like, if the first "mock" response is invalid JSON, will Pydantic AI correctly trigger a second call to the mock model?

   Answered 2025-01-20 by Gary Lopez

• Yes, it does! You can provide a sequence of responses to the TestModel. This is exactly how we tested our "self-healing" logic—we mock a failure first, then a success, and assert that the final output is correct. It makes the whole system much more robust.

     Commented 2025-01-22 by Paul Adams

Since it’s all standard Python code, you can use pytest for everything. No need for specialized runners, which makes it fit perfectly into any standard CI/CD pipeline.

   Answered 2025-02-02 by Donna Rivera

• That’s the best part! It doesn't feel like "AI code," it just feels like "code." It’s refreshing to use standard tools for once.

     Commented 2025-02-05 by Brandon Wood