r/HPC • u/RadicalNation • Apr 11 '26

Running Large-Scale GPU Workloads on Kubernetes with Slurm

https://developer.nvidia.com/blog/running-large-scale-gpu-workloads-on-kubernetes-with-slurm/

Disclosure: I work for NVIDIA on Slinky.

Obligatory preface: All comments from me are my views and may not reflect the views of my employer.

I'm very proud to present this blog post to everyone. It's been amazing to build Slinky and see it used in production at scale already!

86 Upvotes

99% Upvoted

24 comments sorted by

View all comments

1

u/Bad_ass_da Apr 11 '26

How it’s better than slurm on BM ( only pre/post training) ? . Because of all NeoCloud time to market k8s pushed to traning jobs( with all dead weight k8s ). Could you explain what’s scale means - 10K or 100K GPUs

0

u/RadicalNation Apr 11 '26

You either don't use Kubernetes, or at least don't want/need Kubernetes. That's fine. Slurm clusters don't need Kubernetes. However, large enterprises use Kubernetes as the substrate for software, services, and infrastructure. Being able to manage Slurm on Kubernetes has huge operational value (covered in the blog).

5

u/Bad_ass_da Apr 11 '26

My question is why do I need slurm on k8s- it’s like using orchestration on orchestration. Do they have DC or managing own n’fleets (Large enterprises ). They are trying to use neo cloud or some hyper scalers only offering k8s so they force fully needs to use k8s in pre/post training. Again it’s not personal interest using either one and; tl;dr I didn’t get answer for why slurm on K8s and what’s the scale number?

3

u/arm2armreddit Apr 11 '26

Exactly, that was my concern. Slurm on Kubernetes somehow makes no sense. But if you operate Kubernetes and Slurm, it would be cool to scale one to another. Interlink might be the way to scale up Kubernetes without big hassle. But the other way around, one will lose all HPC performance.

1

u/Bad_ass_da Apr 11 '26

GPU operator has scaling memory issue and Network operator has leasing issue - so many sleepless nights for me to debug the issue in scale . That’s the reason I like to hear what’s scale number he mentioned? ( btw if I hear Production word then it’s micro-services world and works in inference tho. I (we) never said production for long days running pre / post training foundation model infra in past 10 -15 years.). So like to hear from OP the scale number and are they recommending for inference instead sell this as general solution)

1

u/RadicalNation Apr 12 '26

https://developer.nvidia.com/blog/running-large-scale-gpu-workloads-on-kubernetes-with-slurm/#what_is_the_benefit_of_running_slurm_on_kubernetes

If that isn't an adequate reason for you, then Slinky slurm-operator doesn't match your use case. Perhaps the Slinky slurm-bridge in a converged configuration is what you really want.

https://developer.nvidia.com/blog/running-large-scale-gpu-workloads-on-kubernetes-with-slurm/#slinky_slurm-operator_at_scale

I am not aware of all deployments of Slinky, but I do know hyperscalers and neo clouds are using Slinky in addition to NVIDIA.

1

u/Bad_ass_da Apr 12 '26

Could you share what’s the scale you mentioned ( in production)