How to evaluate agent performance using AgentOps metrics?

All answers to this question.

The best way to evaluate agents is to look at the "Trajectory Evaluation." Instead of just looking at the final answer, AgentOps allows you to score each step. For example, did the agent select the right tool? Did it use the tool correctly? You should track "p95 latency" to see if the agent is spending too much time "thinking" or if a specific API is slowing it down. We also use "Implicit Feedback" metrics, like whether the user actually followed the agent's instructions or if they had to ask a human for help anyway. By correlating these traces with your business outcomes, you can identify exactly where the agent is failing and refine the prompts for that specific step.

   Answered 2025-08-17 by Deborah Fisher

Are you currently collecting human-in-the-loop feedback during the testing phase, or are you relying entirely on automated scores? Sometimes a "perfect" technical trajectory still feels robotic or unhelpful to a human.

   Answered 2025-08-20 by Brian Walsh

• Brian, we’ve been relying on automated scores, but I think you’re right—we’re missing the "human" element. We are planning to add a dashboard where our QA team can review the traces and provide "Golden" labels. This will help us create a better evaluation harness so we can compare the agent's performance across different versions of the LLM and see if we are actually improving the user experience.

     Commented 2025-08-23 by Amy Coleman

Focus on the "Cost per Success." If an agent takes 10 steps to solve a problem that should take 2, it might be successful but it's not efficient. AgentOps helps find that balance.

   Answered 2025-08-25 by Gregory Lane

• Spot on, Gregory. Efficiency is a huge part of quality management. An agent that is 100% accurate but costs $5 per query isn't a viable solution for most businesses.

     Commented 2025-08-27 by Deborah Fisher