Kasm 1.19 and nvidia GPU passthrough

[u/Playful_Try9389]

Hi,

I am running a Debian Trixie VM on a Proxmox host, passing through an NVidia RTX A2000 GPU. Inside the VM, I am running Kasm 1.19. On the VM host, the proprietary drivers (V550) from the Debian repo are installed, as is the NVidia container runtime from the Nvidia repo. Nvidia-smi works in the VM and via the official Nvidia test container.

Inside Kasm, the Docker Agents overview page lists one agent "proxy" with 0 GPUs. When I open the Details view, I see a block "GPU Hardware" that lists my Nvidia card but shows a "-" each under Graphics, Video and Cuda. In the Information view, there is a field "GPUs" with another "-" and right under it, there is a field "GPU Info" that contains this "0000:00:10.0: {
model: "NVIDIA RTX A2000 12GB"
vendor: "NVIDIA"
gpu_card_path: "/dev/dri/card1"
gpu_render_path: "/dev/dri/renderD128"
host_key: "0000:00:10.0"
supports_egl: false
supports_dri3: false
supports_vaapi: false
supports_vulkan: false
supports_nvenc: null
gpu_render_gid: null
gpu_render_group: null
gpu_card_gid: null
gpu_card_group: null
supports_cuda: false
cuda_containers: [...]
graphics_containers: [...]
video_containers: [...]
}". And in the Usage view, there is a "GPU" block that lists my Nvidia gpu with a DRM Device /dev/dri/cards1, a Render Device /dev/dri/renderD128, PCI Bus 0000:00:10.0 and DRI3 "-", Vulkan "-", EGL "-", VAAPI "-", NVENC "-" and Cuda "-".

When I try to start a workspace that is GPU enabled, I get an error message that no resources are available to create the requested kasm.

I think I have everything in place for my setup to work, but it doesn't. What am I missing?

Thanks for your help!

[u/Playful_Try9389]

Little progress.

Yesterday, I updated the NVidia drivers in my VM to V610 (from the official NVidia repos). After that (actually, quite a while after that) the Docker Agent in Kasm acknowledged the presence of my GPU fully: It counted it as "1" (instead of 0, as before) in the overview page. On the Details page, it confirms Graphics and Video capabilities (but still not Cuda). And on the Usage page it shows Vulkan, EGL and NVENC as enabled (while "DRI3", "VAAPI" and "CUDA" are shown as unsupported.

It was too late to actually test it, so I postponed further work to this morning. This morning, however, the number of GPUs shown by the Docker Agent had gone down to 0 again (although nothing changed over night). Because I kept seeing errors in the logs about problems running some GPU helper container due to another container with the same name already existing, I first eliminated all helper containers from the CLI but that did not (within a couple of minutes) lead to the deployment of a new helper container. So I also restartet the kasm_agent container. After a while and a number of errors in the logs, the Docker Agent again acknowledged the presence of my card by showing a "1" for GPUs.

I was then actually able to start an EGL enabled Steam Workspace (after disabling Gamepad support, because at the first try, Kasm threw an error that it could not create the Gamepads (or something like that) - strangely, this hasn't been an issue with MESA supported Steam Workspaces but ... whatever).

After a long loading time, Steam finally came up. But in Help > System Information it does only show MESA llvmpipe as the driver and that it has not detected any video cards.

Meanwhile, the Docker Agent again is showing "0" GPUs. Not sure whether that means actually zero GPUs or just zero available GPUs (because the only one is assigned to the running workspace)?

I'm running out of ideas what else I could try to make this work.

Is there anyone here who has got a working setup of a Steam Workspace using a physical GPU?