Using Kubernetes HPA for scaling AI inference services based on custom GPU metrics?

All answers to this question.

Yes, this is definitely possible, but it requires the "Prometheus Adapter." Standard HPA only knows about the metrics the Kubelet provides (CPU/RAM). You need to export GPU metrics using something like NVIDIA DCGM Exporter, which feeds data into Prometheus. Then, the Prometheus Adapter makes those metrics available to the Kubernetes Custom Metrics API. Once that's set up, your HPA manifest can target gpu_memory_utilization instead of CPU. It’s a bit of a setup process, but it’s the industry standard for production-grade AI inference scaling to ensure low latency under load.

   Answered 2025-11-13 by Lisa Montgomery

Is there a risk of "flapping" where the pods scale up and down too quickly? Since AI models take a long time to load into VRAM, wouldn't a rapid scale-down event kill a pod that just finished its setup?

   Answered 2025-11-15 by Ryan Howell

• You hit on a major pain point, Ryan. To solve this, you need to tune the behavior section in your HPA spec. You can set a "stabilization window" for scale-down (e.g., 300 seconds). This tells Kubernetes to wait and see if the load stays low before terminating a pod. This prevents the "thrashing" where you spend more time loading the 10GB model into memory than actually serving requests. It makes the whole system much more stable.

     Commented 2025-11-17 by Lisa Montgomery

We use a similar setup but scale based on "Request Per Second" via our Ingress controller. Sometimes that’s a better proxy for load than the hardware metrics themselves.

   Answered 2025-11-18 by Paul Henderson

• That's a valid alternative, Paul. RPS is often a "leading" indicator, whereas GPU heat or memory is a "lagging" indicator. Combining both gives the most robust scaling.

     Commented 2025-11-20 by Sandra Lewis