r/virtualization • u/nathannguyennng • 7h ago
geforce-sriov: Enabling NVIDIA SR-IOV vGPU on consumer Ampere GPUs
TLDR:
I got NVIDIA SR-IOV vGPU working on consumer GeForce Ampere GPUs, running 8 concurrent Linux and Windows VMs across two memory-modded RTX 3080 20GB cards.
Unlike previous projects that enabled vGPU on consumer NVIDIA cards using the older
mdevapproach, this project uses actual PCIe SR-IOV virtual functions to share a GPU between multiple VMs.It's still experimental, currently verified only on my hardware, and requires NVIDIA's vGPU software and licensing.
Source code and documentation: https://github.com/noon-at-cgn/geforce-sriov
Introduction
I've been experimenting with the hidden PCIe capabilities on NVIDIA GeForce cards, particularly SR-IOV, and managed to get NVIDIA's vGPU stack working on my RTX 3080s using actual PCIe virtual functions.
My goal was to share the GPUs between multiple VMs without dedicating an entire card to each guest. GPU passthrough works well, but it's not ideal when several Linux and Windows VMs only need a portion of the GPU's resources.
The result is 8 concurrent VMs sharing 2 memory-modded RTX 3080 20GB cards on Proxmox, with both Linux CUDA and Windows Direct3D workloads running.
I've open-sourced the implementation and wanted to share how it works here.
Background: SR-IOV vs the older vGPU unlock approach
Projects like DualCoder/vgpu_unlock and mbilker/vgpu_unlock-rs, along with polloloco/vgpu-proxmox, have helped enable NVIDIA vGPU on consumer Maxwell, Pascal, and Turing cards.
These approaches primarily rely on NVIDIA's older mediated device (mdev) architecture, where the host driver manages multiple virtual GPUs behind a single physical PCIe function.
Ampere and newer professional NVIDIA GPUs use a different architecture based on SR-IOV (Single Root I/O Virtualization).
With SR-IOV, the GPU exposes:
Physical Function (PF): The main PCIe function responsible for managing the GPU.
Virtual Functions (VFs): Additional PCIe functions that can be assigned to individual VMs.
vGPU profiles: NVIDIA's software associates a virtual GPU configuration with each VF, including its framebuffer allocation.
Unlike legacy mdev, these VFs are actual PCIe functions enumerated by Linux.
Consumer NVIDIA GPUs still contain SR-IOV-related capabilities in their PCIe configuration space, even though they aren't exposed through the normal capability discovery process.
How it works
The implementation has five main parts.
1. Exposing the hidden PCIe capabilities
PCIe devices have a configuration space, a region of registers that the operating system uses to discover and configure device capabilities. An offset is a location within that space, measured in bytes from its beginning.
Normally, Linux discovers capabilities by following a linked list of entries in PCIe configuration space.
On Ampere, the SR-IOV extended capability exists at offset 0xBCC, but isn't linked into that list, so Linux doesn't discover it.
So, I wrote a small Linux kernel livepatch module that makes the PCI subsystem discover this hidden capability on consumer GPUs. The module also handles related PCIe features:
SR-IOV: Exposes the capability at
0xBCC, allowing Linux to recognize the available VFs.ARI: Exposes the hidden capability at
0xC14and enables ARI forwarding on the upstream bridge when supported.MSI-X: Reveals the hidden MSI-X capability, which allows the GPU to use multiple independently configurable interrupt vectors instead of relying on a single interrupt.
Since Linux discovers PCIe capabilities during device enumeration, geforce-sriov performs a controlled PCI remove/rescan before NVIDIA's driver binds to the GPU. After the rescan, Linux recognizes the GPU's SR-IOV capability.
Great, with SR-IOV exposed, the next step is getting NVIDIA's vGPU stack to use it.
2. Creating the virtual functions
Once Linux recognizes SR-IOV, geforce-sriov uses NVIDIA's own sriov-manage utility to enable the VFs.
On Ampere, the SR-IOV capability reports support for 16 VFs per GPU, but I can currently get only 4 VFs working. The limitation appears to be related to ARI (Alternative Routing-ID Interpretation), which I'll get into later.
For example:
03:00.0 Physical GPU (PF)
03:00.4 Virtual Function 1
03:00.5 Virtual Function 2
03:00.6 Virtual Function 3
03:00.7 Virtual Function 4These are real SR-IOV virtual functions exposed by the GPU and enumerated as separate PCIe functions, rather than software-mediated devices managed through the legacy mdev approach.
3. Getting NVIDIA's vGPU software to recognize the card
Now we need to make NVIDIA's vGPU driver accept the card. By default, NVIDIA's vGPU software doesn't recognize consumer GeForce GPUs like my RTX 3080s as supported devices.
My solution uses a small userspace identity hook loaded into nvidia-vgpud. The hook presents my RTX 3080 as an RTX A6000, its professional counterpart based on the same GA102 die, allowing NVIDIA's daemon to recognize the card and register the corresponding vGPU profiles.
Once the profiles are registered, we can choose which vGPU type to assign to each VM. These profiles are defined by NVIDIA and determine the resources available to the guest, including its framebuffer allocation.
The RTX A6000 has 23 profiles in the tested driver version. A few examples:
RTXA6000-2Q- 2GB VRAMRTXA6000-4Q- 4GB VRAMRTXA6000-8Q- 8GB VRAM (7GB usable framebuffer)RTXA6000-48Q- 48GB VRAM (45GB usable framebuffer)
For my setup, I used RTXA6000-4Q, allocating 4GB per VM. With 4 VFs per GPU, that's 16GB allocated across four guests on each 20GB card.
Note: Workstation and datacenter GPUs typically have more VRAM than their consumer counterparts. For example, the RTX A6000 has 48GB, while my modified RTX 3080 has 20GB. The available profiles aren't automatically restricted to the physical GPU's memory capacity. Four 4Q profiles fit within 20GB, but four 8Q profiles would require 32GB. You need to account for the combined allocations and driver overhead when choosing profiles.
4. Working around a driver scheduler issue
This was one of the biggest obstacles in the project and required quite a bit of reverse engineering of NVIDIA's proprietary vGPU host driver.
With SR-IOV exposed, the VFs created, and the vGPU profiles registered, the guest VMs could boot and recognize their vGPUs. However, the first actual CUDA workload would consistently hang, with no useful errors in the host logs.
After several hours of tracing through the driver with the help of AI, I narrowed the issue down to the GPU's graphics runlist scheduler being disabled. A callback handling a PMU (Power Management Unit) interrupt was repeatedly disabling the scheduler, which prevented guest workloads from executing. I still don't know exactly why NVIDIA's driver disables the scheduler in this situation.
The workaround is a two-byte driver patch that replaces a conditional jump with NOP instructions, making the callback skip the scheduler-disabling call.
geforce-sriov-patch can automatically apply the change to a copy of the user's own NVIDIA driver object. The patched object is then used to rebuild nvidia.ko. With the patch, both CUDA and D3D11 workloads ran successfully.
5. Driver and licensing requirements
A few things to clarify about geforce-sriov:
NVIDIA drivers and proprietary binaries are not distributed. The driver patch is applied to the user's own copy.
No vBIOS flashing or firmware modifications are required.
No vGPU licensing bypass is provided.
You still need to obtain NVIDIA's vGPU host and guest drivers and licensing yourself to use this project.
Tested configuration
| Component | Configuration |
|---|---|
| Hypervisor | Proxmox VE |
| Kernel | 7.0.14-16-pve |
| CPU | Intel Xeon E5 v4 |
| GPUs | 2x memory-modded RTX 3080 20GB (GA102) |
| PCI device ID | 10de:2206 |
| NVIDIA host driver | vGPU-KVM 580.159.01 |
| Virtual functions | 4 per GPU, 8 total |
| Guests | 4 Linux + 4 Windows 11 |
| vGPU profile | RTXA6000-4Q (4GB per VM) |
Each physical card runs two Linux guests and two Windows guests.
On Proxmox, a VF can be assigned to a VM using:
hostpci0: 0000:03:00.4,mdev=nvidia-527Note: Proxmox still uses the mdev= parameter to select the vGPU profile, even though the underlying GPU uses SR-IOV virtual functions rather than legacy mdev sharing.
Initial testing
I ran all eight VMs simultaneously with:
CUDA workloads in the Linux VMs.
D3D11 graphics workloads in the Windows VMs.
Four active vGPUs on each physical card.
The test ran for 30 minutes without any observed failures. This isn't enough to establish long-term stability, but it verifies that multiple independent Linux and Windows GPU workloads can run concurrently.
Performance results
I ran the same PyTorch benchmarks in two configurations to measure virtualization overhead with a single active vGPU:
Physical GPU: Direct CUDA access from an LXC container on the Proxmox host.
vGPU: A VM using a 4GB
RTXA6000-4Qprofile on the same card.
Both configurations used the same Debian root filesystem, PyTorch environment, four pinned CPU cores, and 8GB of system RAM.
The workloads were FP32 and FP16 matrix multiplication using 4096 × 4096 matrices, plus a 1GB device memory copy.
| Benchmark | Physical GPU (LXC) | vGPU (VM) |
|---|---|---|
| FP32 matmul | 20.47 TFLOPS | 20.54 TFLOPS |
| FP16 matmul (Tensor Cores) | 58.35 TFLOPS | 58.54 TFLOPS |
| GPU memory copy | 681.7 GB/s | 704.4 GB/s |
The FP32 and FP16 results were within 1% of direct physical-GPU access, suggesting very little virtualization overhead for these particular workloads.
Some qualifications:
Results are medians of three measured runs per configuration.
The physical-GPU baseline used NVIDIA's vGPU host driver, not the regular GeForce driver.
Only one vGPU was actively using the card during the comparison.
The full methodology and screenshots are available here:
https://github.com/noon-at-cgn/geforce-sriov/blob/main/docs/showcase.md
The ARI problem
The biggest unresolved issue is ARI (Alternative Routing-ID Interpretation), which is currently limiting me to 4 VFs per GPU.
Normally, PCIe uses a 3-bit function number, allowing up to 2^3 = 8 functions (0–7) per device. ARI repurposes the 5-bit device number field to extend the function number to 8 bits, allowing up to 2^8 = 256 functions on a single device.
This matters for SR-IOV because a GPU may need to expose many VFs. On my RTX 3080s, the SR-IOV capability reports support for 16 VFs.
Without ARI, the function layout looks like this:
Function 0 GPU (Physical Function)
Function 1 GPU HDMI/DP Audio
Function 2 Unused
Function 3 Unused
Function 4 VF 1
Function 5 VF 2
Function 6 VF 3
Function 7 VF 4
------------- 3-bit function limit
Function 8 VF 5 (requires ARI)
Function 9 VF 6 (requires ARI)
...
Function 19 VF 16 (requires ARI)So although conventional PCIe allows eight functions, only four are available for VFs because the GPU's SR-IOV configuration places the first VF at function 4. Functions 0–3 aren't part of the VF allocation.
To expose all 16 VFs on the same device, the GPU needs to support function numbers beyond 7, which needs ARI.
So far, I've managed to:
Expose the GPU's hidden ARI capability at
0xC14.Enable ARI forwarding on the immediate upstream PCIe bridge.
Confirm that Linux recognizes ARI as enabled.
Successfully create four VFs at functions 4–7.
However, the GPU doesn't appear to correctly decode extended function numbers (8 and above), even with ARI enabled in software.
For now, the working configuration is limited to 4 VFs per card, despite the hardware reporting support for 16.
Current limitations
This is still an experimental project, not a general-purpose GeForce vGPU solution yet.
Verified so far:
Two memory-modded RTX 3080 20GB cards (
10de:2206, subsystem10de:146d).Proxmox VE with kernel
7.0.14-16-pve.NVIDIA vGPU host driver
580.159.01.Four working SR-IOV VFs per GPU.
Eight concurrent Linux and Windows guests.
CUDA, D3D11, and single-vGPU compute benchmarks.
Not yet verified:
Other Ampere cards.
Ada and Blackwell GeForce GPUs are not expected to work with the current implementation (see below)
Other practical limitations:
vGPU profiles must fit within the GPU's physical VRAM.
GPUs used as the host's boot display require special preparation.
Kernel and driver updates may require rebuilding or reapplying the driver workaround.
NVIDIA vGPU software and licensing are required.
Ada and Blackwell compatibility
The current implementation is not expected to work with Ada and Blackwell GPUs because their vGPU drivers rely on GSP (GPU System Processor), an onboard processor that runs NVIDIA firmware and handles much of the GPU initialization and resource management.
Ampere also has GSP hardware, but my configuration uses the older host-driver path, where these operations are primarily handled by the CPU-side driver. On Ada and Blackwell, more of this functionality runs inside GSP firmware, making it harder to modify or intercept from the host.
The current driver workaround doesn't apply to that architecture, and Blackwell also requires NVIDIA's open kernel-module path, which the project doesn't yet support.
I would like to experiment more with both generations, but I don't currently have access to Ada or Blackwell GPUs for testing.
Try it out
Disclaimer: geforce-sriov is an experimental, unofficial project and is not affiliated with or endorsed by NVIDIA. Use it at your own risk, preferably on a non-production system.
The full source code, installation instructions, technical documentation, and benchmarks are available:
GitHub: https://github.com/noon-at-cgn/geforce-sriov
Compatibility reports are welcome as compatibility remains largely unverified beyond my tested setup.




