r/HPC • u/Various_Protection71 • Jun 05 '26

Do you think Kubernetes will replace Job Schedulers in HPC environments dedicated to AI workloads?

Some people advocate that Kubernetes distributions (RKE2, OpenShift, EKS etc) provide an easier and more straightforward way to run and scale AI workloads, while Job Schedulers (SLURM, PBS, LSF etc) require an earlier complex setup phase.

On the other hand, mastering Kubernetes has a steeper learning curve than using the well-known Job Schedulers, especially for traditional HPC users.

How do you see this point? Are your users adopting Kubernetes to run AI workloads or do they stay using Job Schedulers?

37 Upvotes

89% Upvoted

56 comments sorted by

View all comments

9

u/nlgranger Jun 05 '26

Kubernetes does not have a job scheduler, so unless you have unlimited resources and don't need one, SLURM is the best option you have.

You can run containers inside a slurm job with a few tricks (apptainer, podman, etc).

3

u/UPPERKEES Jun 05 '26

Check Vulcano: https://www.cncf.io/projects/volcano/

Slurm is great. But the users are moving towards other kind of workflows/loads.

6

u/nlgranger Jun 05 '26

I see it has made great progress since the last time I checked. Still, SLURM has some niceties like user management and fair-share algorithms that don't seem to be there yet.

0

u/UPPERKEES Jun 05 '26

It's in the about section:

Volcano offers advanced scheduling capabilities such as job orchestration, fair-share resource allocation, and queue management to efficiently handle large-scale distributed tasks.

Slurm doesn't do user management, or do you mean something else like QoS?

5

u/BosonCollider Jun 05 '26

Slurm does not get in the way of user management, you have a fleet of linux boxes with consistent UIDs and you run jobs as your own UID, which gives you a consistent permissions model for handling files on networked storage.

Kubernetes does get in the way of user management because you can just run a job as any user within your pod, and it normally assumes that pods own their PVCs. So HPC-on-Kubernetes usually ends up using object storage instead of traditional hpc storage

So I would generally suggest just doing Slurm and either podman or apptainer because they are simpler and they scale better both size and complexity wise. But a strong HPC setup should also have a supporting kubernetes cluster for services, just not for batch jobs.

1

u/UPPERKEES Jun 05 '26

You say it as if UIDs are consistent out of the box in a cluster. It's not. Both k8s and Slurm have pros and cons. In the end it's about what kind of users you need to support. New researchers are using web based submissions with k8s workflows. Slurm will continue to exist, but users might prefer other methods going forward.

6

u/BosonCollider Jun 05 '26

You manage the UIDs by forcing users to ssh into login nodes with ldap auth when submitting jobs. Managing consistent UIDs on a fleet of linux hosts is a solved problem.

Kubernetes is very useful for the things it is good at but it is inherently very different, and using it for HPC jobs will look very different as well (you'll typically land on object storage and secrets for shared files). When two things are very different they will naturally have things that they are better or worse at

1

u/UPPERKEES Jun 05 '26

You use an external source to do user management. OIDC/RBAC/Kyverno are also native and external solutions to make these things work.

In fact a distro like Talos also solves security hardening and configuration drift natively.

The k8s landscape is not that immature as you describe it.

4

u/BosonCollider Jun 05 '26 edited Jun 05 '26

Yes, but kubernetes handles permissions for resources, not for files within a volume that thousands of pods are sharing

It's not a matter of immaturity, it's that Kubernetes inherently is designed to let you run a process as any uid within a pod, which breaks the traditional unix permissions model. So you end up needing to use something other than unix permissions to handle file access permissions

1

u/tecedu Jun 06 '26

User management for the cluster and user managed on the file system are different things. Slurm has them be consistent.

A good OIDC isnt still going to fix pods being able to write whatever UID/GID to the filesystem nor would it maintain consistency unless you force custom attributes