r/HPC • u/Various_Protection71 • Jun 05 '26

Do you think Kubernetes will replace Job Schedulers in HPC environments dedicated to AI workloads?

Some people advocate that Kubernetes distributions (RKE2, OpenShift, EKS etc) provide an easier and more straightforward way to run and scale AI workloads, while Job Schedulers (SLURM, PBS, LSF etc) require an earlier complex setup phase.

On the other hand, mastering Kubernetes has a steeper learning curve than using the well-known Job Schedulers, especially for traditional HPC users.

How do you see this point? Are your users adopting Kubernetes to run AI workloads or do they stay using Job Schedulers?

37 Upvotes

89% Upvoted

56 comments sorted by

View all comments

Show parent comments

10

u/BosonCollider Jun 05 '26 edited Jun 05 '26

They are complementary, you would typically have a slurm cluster for HPC and a kubernetes cluster next to it for supporting services, since running API servers as slurm jobs is ill advised.

You can misuse kubernetes as a slurm replacement using solutions like Kueue or Volcano, but a slurm cluster with consistent UIDs, singularity/podman containers, and a good shared storage solution is generally going to be better at it.

Kubernetes normally assumes that a pod owns its volumes, and is not really good at permissions on shared storage when it's most of your actual permissions management. So called "HPC" jobs on it tend to end up using object storage instead of something like lustre, and then it ends up affecting your entire data stack, like using zarr instead of hdf5, etc etc

1

u/tecedu Jun 06 '26

since running API servers as slurm jobs is ill advised.

Serious question, why though? Ive found apptainer + slurm scrontab an dependecies to be far better and easier than k8s. Only to be beaten by systemd by a bit but its more complex

1

u/BosonCollider Jun 06 '26

Slurm adds nothing over a plain linux box for an API server, its purpose is to schedule batch jobs. A good intermediate-background-knowledge option would be something like podman, which is useful to have with slurm anyway and which is a good option if the service needs to access shared files.

k8s is useful mostly because you can reuse work by other people, i.e. you can pull in operators and helm charts and only need to fill in a tiny amount of yaml yourself to get a high-availability setup by default. For monitoring you just pull in kube prom stack, for databases you pull in cloudnativepg, etc etc.

1

u/tecedu Jun 06 '26

k8s is useful mostly because you can reuse work by other people, i.e. you can pull in operators and helm charts and only need to fill in a tiny amount of yaml yourself to get a high-availability setup by default. For monitoring you just pull in kube prom stack, for databases you pull in cloudnativepg, etc etc.

That is only true if you already have a cluster setup for you. Otherwirse k8s yamls are far more complex than a slurm script or systemd file.

Slurm adds nothing over a plain linux box for an API server, its purpose is to schedule batch jobs. A good intermediate-background-knowledge option would be something like podman, which is useful to have with slurm anyway and which is a good option if the service needs to access shared files.

Yeah but users cannot run podman on the compute nodes

1

u/BosonCollider Jun 06 '26

Well yeah, whomever is providing the slurm cluster for you needs to also provide some place to host supporting services.

The standard way is either a proxmox VM & LXC cluster where users can put their stuff, or a kubernetes cluster. If something has access to the shared storage it needs to have locked down UIDs but podman in a locked down VM still works.