Why are Small LLMs (SLMs) becoming the preferred choice for on-device mobile AI applications?

All answers to this question.

In my recent project, we switched to Small LLMs (SLMs) specifically to avoid the high latency of API calls. For a fitness app, you really need sub-100ms response times, which you just can't get reliably with a cloud-based LLM over mobile data. Models like Phi-3 are surprisingly capable of following instructions if you provide clear system prompts. The main trade-off is the context window; you can't feed it a 50-page manual, but for localized feedback based on sensor data, it’s remarkably efficient and uses very little battery compared to larger models.

   Answered 2025-11-14 by Megan Fletcher

How are you handling the quantization process to ensure the model actually fits within the limited RAM of a mid-range device?

   Answered 2025-11-15 by Jordan Pierce

• That’s a crucial step, Jordan. We usually go with 4-bit quantization using bitsandbytes or AutoGPTQ. It reduces the memory footprint by nearly 70% with only a minor hit to accuracy. For a posture correction app, that small accuracy trade-off is worth the massive gain in inference speed, especially since the model is only handling narrow, domain-specific language rather than general knowledge.

     Commented 2025-11-16 by Lawrence Brooks

SLMs are perfect for this because they offer privacy. Users don't want their workout videos or personal health data sent to a third-party server.

   Answered 2025-11-17 by Austin Meyer

• Excellent point, Austin. Privacy is a huge selling point in 2024, and keeping everything local is the best way to ensure compliance with health data regulations.

     Commented 2025-11-18 by Tyler Harrison