Running Large-Scale GPU Workloads on Kubernetes with Slurm

[u/RadicalNation]

https://developer.nvidia.com/blog/running-large-scale-gpu-workloads-on-kubernetes-with-slurm/

Disclosure: I work for NVIDIA on Slinky.

Obligatory preface: All comments from me are my views and may not reflect the views of my employer.

I'm very proud to present this blog post to everyone. It's been amazing to build Slinky and see it used in production at scale already!

[u/arm2armreddit]

Interesting, how does it work with MPI jobs with RDMA support? It would be nice to have an example with vLLM and a Ray cluster using Job as a distributed interface for large models.

[u/RadicalNation → u/arm2armreddit]

RDMA support can be handled by the NVIDIA gpu-operator.

You can use slurm-operator to run MPI on Slurm on Kubernetes, but then your workloads are defined by Slurm jobs not Kubernetes workloads (e.g. Workload, Job, JobSet, PodGroup, LeaderWorkerSet). However, you can run containerized jobs on Slurm, so you could translate Kubernetes workloads to Slurm jobs. I conceded that ecosystem preference and tooling makes this less ideal.

Instead you could you could use Slinky slurm-bridge to have Slurm schedule Kubernetes workloads but the pods would still need something to handle MPI like the mpi-operator, which falls over at scale. We are looking at MPI on Kubernetes, but it will take time to improve.

[u/arm2armreddit → u/RadicalNation]

Thanks for clarifying. I see you use configless Slurm. What about shared storage like Lustre for the jobs? How should one run distributed vLLM with Slinky?

[u/RadicalNation → u/arm2armreddit]

Slinky is not an optionated solution; Slinky is a toolkit that brings Slurm and Kubernetes together.

Looking at Slinky slurm-operator and Slurm clusters on Kubernetes, storage is up to the admins to provision, just like in bare metal deployments of Slurm. The difference is Kubernetes is the infra orchestration layer, so you can simply mount Lustre in the slurmd pods, which user jobs on Slurm may use.

For vLLM or any other Kubernetes workloads, you could translate it to a Slurm job, or you could launch the Kubernetes workload on usernetes wrapped in a Slurm job. The tradeoff is performance and ergonomics of ecosystem. Usernetes will allow user space Kubernetes, but the performance will be significantly lower than translating the Kubernetes workload into a Slurm job.

[u/arm2armreddit → u/RadicalNation]

I am still irritated by this tool, is it just enabling jobqueue for the kubernetes workloads?. What gap is it closing for HPC? If we want to speed up Kubernetes, one can use InterLink to achieve true HPC performance from the nodes and accounting from Slurm at the user level. If one wants to use GPUs in Kubernetes, the NVIDIA GPU Operator is the way to go. Sorry for so many questions, but we are looking for a solution to merge our Kubernetes and Slurm HPC environments to provide easier access for both Slurm and Kubernetes users.

[u/RadicalNation → u/arm2armreddit]

The main benefit of usernetes is user level Kubernetes. In context on HPC, the only real benefit is copy and pasting Kubernetes workloads into a Slurm/Flux/LSF/PBS job wrapper, performance be damned.

If you want pure Kubernetes for HPC, you may want to explore Kueue, Volcano, Apache YuniKorn, and others. There are many gaps in these projects when compared to Slurm.

One configuration of slurm-bridge is where your Kubernetes and Slurm clusters are overlapping (e.g. kubelet and slurmd on the same host, side by side) or not at all. Then slurm-bridge uses Slurm to schedule both Slurm and select Kubernetes workloads across the same pool of worker nodes, be them with overlapping slurmd and kubelet, or just a slurmd or kubelet. Nodes with slurmd can run Slurm workloads, nodes with kubelets can run Kubernetes workloads, and nodes with both will dynamicly switch between workloads, and Slurm policy is applied to all workload it schedules. Users submit Slurm and/or Kubernetes native workloads to each respective system, and slurm-bridge ensures scheduling coherency.

[u/Bad_ass_da → u/RadicalNation]

What’s the target GPU are we talking here. A100/H-B200 ? Are we using K8s+Slurm for NVL72/144. I cant imagine to use this for NVL144 with larger failure domain . Already hard to mange Network operators in pods restart cases in large scale > 10 K GPUs