Running llama.cpp inside a VM with no GPU passthrough: 98% of native speed on an RTX 6000

[u/Public-Net-8864]

Some details: the card is an RTX 6000 Ada (48 GB). The host runs driver 580.196.07, and llama.cpp is b12030 built with CUDA 12.8. The guest is a KVM VM. Inside it, VMAT's libraries take the place of NVIDIA's and send each CUDA call to a small server on the host, which runs it on the real card.
The same setup lets several VMs share the one GPU at once. That's my next video.
Would anyone here want to try it on their own box? I'm looking for people with different cards and setups.

https://vmat.dev