How do I optimize Agentic AI for cost-efficiency when using high-token LLMs?

All answers to this question.

This is a huge issue in production Agentic AI right now. The best strategy is "Semantic Routing." You use a very fast, cheap model (or even a simple vector search) to classify the complexity of the task. If it’s a simple data lookup, route it to a 7B parameter model. If it requires deep reasoning or multi-step planning, only then do you send it to GPT-4. Also, look into "Prompt Caching"—many providers are now offering discounts for reused context. Reducing the length of your system prompts and using more efficient "few-shot" examples can also save a ton of money.

   Answered 2024-08-17 by Susan Walker

Have you considered fine-tuning a smaller model on your specific agent's "thinking" patterns so it can eventually replace the expensive model for 80% of your tasks?

   Answered 2024-08-19 by William Scott

• William, that’s exactly what we are experimenting with now. We are using GPT-4 to generate "Gold Standard" logs of how the agent should think, and then we plan to use those logs to fine-tune a smaller Llama model. It's a bit of an upfront investment in data collection, but the long-term ROI on API costs could be massive if we can get a 7B model to perform like a 70B one for our specific niche.

     Commented 2024-08-20 by Kevin Lewis

Don't forget to implement "max iteration" limits. Sometimes agents get into a reasoning loop and can burn $10 worth of tokens in a few seconds if they aren't stopped.

   Answered 2024-08-21 by Margaret Hall

• Margaret is spot on. Without "Recursion Limits," your agents can become a bottomless pit for your budget. Always set a hard cap on how many turns an agent can take.

     Commented 2024-08-22 by Susan Walker