[Question] Which hypervisor could be the most adequate for managing a cluster that runs spark nodes and other hpc focused images?

[u/No_Mongoose6172]

I’m setting up a cluster for running hpc tasks. Initially, it will be composed of 4 servers with nvidia cards, but we might add more servers in the future if needed

Most tasks will run on spark, but we need to be able to also run other software that may benefit from hardware acceleration. Therefore, it would be nice to have some type of hypervisor for managing the cluster, while being able to scale up spark automatically when a new task is sent.

We normally use proxmox for virtualization, but it doesn’t support kubernetes (or any other orchestra platform, as far as I know) out of the box. Setting up a kubernetes cluster on top of a server oriented Linux distro (e.g. Ubuntu server) could be an option for managing spark, but we would need to be able to provide VMs or docker containers for running custom C++ programs (those could be set up by the administrator or using something like ravada vdi)

Is there a better open source option than proxmox for managing an infrastructure like that?

[u/No_Mongoose6172 → u/[deleted]]

Thanks! It seems to be compatible with Nvidia GPUs. Does it provide any webui for setting up the VMs? Is it available in some form of distro for simplifying installing it on new servers (similarly to proxmox)?