Can Kubernetes manage "Spot" instances for AI training to reduce cloud costs?

All answers to this question.

Yes, and this is where Kubernetes really shines for cost-saving. You use "Node Affinity" to prefer Spot nodes for training jobs. To handle preemption, you need two things: a "Termination Handler" and model checkpointing in your code. The termination handler catches the 2-minute warning from the cloud provider and sends a SIGTERM to your pod. Your code should catch that signal, save the current weights to a persistent volume (like S3 or an Azure Disk), and exit. Kubernetes then sees the pod failed and, thanks to the "Job" controller, automatically reschedules it on a new available node to resume

   Answered 2025-05-21 by Laura Griffin

What happens if there are no more Spot instances available in that region? Does the training just stop forever, or can Kubernetes failover to a full-priced "On-Demand" node automatically?

   Answered 2025-05-23 by Gary Peterson

• You can set up "Node Groups" with different priorities. You define a Spot group as priority 1 and an On-Demand group as priority 2. If the Cluster Autoscaler fails to find a Spot node after a certain timeout, it will fall back to provisioning an On-Demand node. This ensures your deadline is met even if the Spot market is tight. It’s the perfect balance between saving money and ensuring the job actually gets finished.

     Commented 2025-05-25 by Laura Griffin

We use the "Karpenter" autoscaler for this. it’s much faster at picking the cheapest instance type available that fits the GPU requirements of the pod.

   Answered 2025-05-26 by Brian Castro

• I've heard great things about Karpenter. Its ability to "right-size" the node to the pod's exact request is a game changer for AI cost management.

     Commented 2025-05-28 by Deborah King