How to handle multi-tenant AI workloads on a single Kubernetes cluster?

All answers to this question.

ResourceQuotas are the first line of defense, but they are "blunt." They limit the total number of GPUs a namespace can request. For better sharing, you should look into NVIDIA Multi-Instance GPU (MIG) or "GPU Time-Slicing." MIG allows you to physically partition one high-end GPU into up to seven isolated instances. Kubernetes can then treat each slice as a separate resource. If you use Time-Slicing, multiple pods can share a GPU by taking turns on the compute cores. Combined with "PriorityClasses," you can ensure that "Production Inference" always beats "Experimental Training" if the cluster runs out of space.

   Answered 2025-02-07 by Betty Cooper

If I use MIG, is the memory also isolated? I'm worried that if one user's pod has a memory leak or an out-of-memory (OOM) error, it might crash the other "slices" on that same physical card.

   Answered 2025-02-09 by Jason Myers

• Jason, that’s the beauty of MIG—it provides hardware-level isolation for both compute and memory. If a pod in one MIG partition crashes or OOMs, it has zero impact on the other partitions. It’s much safer than simple library-level sharing. However, keep in mind that you can't change the MIG profile dynamically without "draining" the node, so you have to plan your partition sizes (like 1g.5gb vs 3g.20gb) based on your team's typical model sizes.

     Commented 2025-02-11 by Betty Cooper

Taints and Tolerations are also key. We taint our GPU nodes so that basic web-app pods don't accidentally get scheduled there and waste expensive GPU-node "headroom."

   Answered 2025-02-12 by Mark Wagner

• Great point, Mark. There is nothing more frustrating than seeing a simple Nginx pod sitting on a node that could be running a $30k GPU for a training job.

     Commented 2025-02-14 by Steven Reynolds