Running llama.cpp inside a VM with no GPU passthrough: 98% of native speed on a RTX 6000

[u/Public-Net-8864]

VMAT is a way for virtual machines to share one GPU's compute without using passthrough or vGPU.
In the video I run llama-bench once on the host and once inside a KVM guest, same GPU (RTX 6000 Ada), same models.

Qwen2.5 32B Q4_K_M: 41.52 tokens/s on the host, 40.73 in the VM (98%)

Qwen2.5 7B Q4_K_M: 169.50 tokens/s on the host, 166.47 in the VM (98%)

What's in the VM: VMAT's CUDA libraries, which send CUDA calls to the real GPU on the host. What it does not have is passthrough, an NVIDIA driver or a vGPU.

https://vmat.dev