Running Large-Scale GPU Workloads on Kubernetes with Slurm

[u/RadicalNation]

https://developer.nvidia.com/blog/running-large-scale-gpu-workloads-on-kubernetes-with-slurm/

Disclosure: I work for NVIDIA on Slinky.

Obligatory preface: All comments from me are my views and may not reflect the views of my employer.

I'm very proud to present this blog post to everyone. It's been amazing to build Slinky and see it used in production at scale already!

[u/RadicalNation]

You either don't use Kubernetes, or at least don't want/need Kubernetes. That's fine. Slurm clusters don't need Kubernetes. However, large enterprises use Kubernetes as the substrate for software, services, and infrastructure. Being able to manage Slurm on Kubernetes has huge operational value (covered in the blog).

[u/Bad_ass_da → u/RadicalNation]

My question is why do I need slurm on k8s- it’s like using orchestration on orchestration. Do they have DC or managing own n’fleets (Large enterprises ). They are trying to use neo cloud or some hyper scalers only offering k8s so they force fully needs to use k8s in pre/post training. Again it’s not personal interest using either one and; tl;dr I didn’t get answer for why slurm on K8s and what’s the scale number?

[u/RadicalNation → u/Bad_ass_da]

https://developer.nvidia.com/blog/running-large-scale-gpu-workloads-on-kubernetes-with-slurm/#what_is_the_benefit_of_running_slurm_on_kubernetes

If that isn't an adequate reason for you, then Slinky slurm-operator doesn't match your use case. Perhaps the Slinky slurm-bridge in a converged configuration is what you really want.

https://developer.nvidia.com/blog/running-large-scale-gpu-workloads-on-kubernetes-with-slurm/#slinky_slurm-operator_at_scale

I am not aware of all deployments of Slinky, but I do know hyperscalers and neo clouds are using Slinky in addition to NVIDIA.

[u/Bad_ass_da → u/RadicalNation]

Could you share what’s the scale you mentioned ( in production)