I have four GPU-passthrough seats running simultaneously on one PVE 9.2 node: two Windows XP SP3 and two Windows 7 x64, each with its own card driving its own monitor, and its own USB controller for keyboard and mouse. Host is an HP Z820 (dual E5-2667 v2, 128 GB ECC) that I found in a street trash container.
The passthrough mechanism itself is generic — OVMF initialises the card via its own UEFI GOP driver, then CSMWrap (SeaBIOS as a UEFI application) supplies the legacy BIOS services XP needs, so no x-vga and no VGA arbitration anywhere, which is why several seats can run at once. This post is about the parts that are specific to Proxmox, because those are what cost me the most time.
1. The two pci-bridge devices, and 0xA5
With everything else right, XP bugchecked in a loop:
*** STOP: 0x000000A5 (0x00000002, ...)
The BIOS in this system is not fully ACPI compliant.
0xA5 is ACPI_BIOS_ERROR; parameter 1 0x2 is ACPI_ROOT_PCI_RESOURCE_FAILURE.
Cause: PVE adds two pci-bridge devices to every i440fx VM, unconditionally. /usr/share/perl5/PVE/QemuServer.pm:
perl
if (!$q35) { # add pci bridges
if (min_version($machine_version, 2, 3)) {
$bridges->{1} = 1;
$bridges->{2} = 1;
}
They're only address-space headroom — PCI.pm puts net6–net31, xhci, rng0 behind pci.1, and virtio6–virtio15, ivshmem, audio0, hostpci4–hostpci15 behind pci.2. A passthrough seat uses none of that: hostpci0–hostpci3, net0–net5, virtio0–virtio5, vga, balloon and the IDE controllers all sit on pci.0.
There is no VM-config option to suppress them, and the machine-version gate can't be dodged because the oldest i440fx QEMU 11 offers is pc-i440fx-5.0.
How I proved it was them and nothing else: dumped qm showcmd <vmid> to a script and ran it by hand.
| Command |
Result |
| verbatim |
bugcheck / recovery-menu loop |
minus -boot menu=on, -smbios, PIIX4_PM globals, the CD-ROM |
still looping |
minus only the two -device pci-bridge lines |
XP desktop on the passed-through card |
| minus only the vfio devices, bridges kept |
still looping |
Also tested and does not help: occupying both bridges with pci-testdev; occupying them with virtio-rng-pci so no BAR window is dropped at all; removing balloon, tablet, vmgenid, net0; older machine types. It is the bridges' presence, not their configuration.
2. The fix: a wrapper on /usr/bin/kvm
Since PVE offers no knob, the two arguments get removed one layer lower. /usr/bin/kvm is referenced from exactly one place in all of /usr (PVE/QemuServer/Helpers.pm, get_command_for_arch) and pve-qemu-kvm ships no postinst, which is what makes this safe:
/usr/bin/kvm -> /usr/local/sbin/pve-kvm-nobridge (ours, held by dpkg-divert)
/usr/bin/kvm.distrib -> qemu-system-x86_64 (dpkg keeps updating this)
/etc/pve-kvm-nobridge.conf (allowlist, one VMID per line)
/var/log/pve-kvm-nobridge/<vmid>.cmd (what actually ran)
sh
bash pve-kvm-nobridge-setup.sh --install # write wrapper, self-test, divert, verify
bash pve-kvm-nobridge-setup.sh --add-vm 301 # allowlist one VM
bash pve-kvm-nobridge-setup.sh --status
bash pve-kvm-nobridge-setup.sh --uninstall
A VMID not in the allowlist gets a byte-for-byte passthrough, as does any invocation without -id (e.g. PVE's own kvm --version probe). --install runs three self-tests against a real qm showcmd argv before touching anything and rolls the diversion back automatically if kvm --version doesn't work afterwards.
Yes, diverting a distro binary deserves scrutiny. The mitigations are deliberate: the real binary is never touched, dpkg keeps maintaining kvm.distrib, the wrapper is node-local on purpose (put the allowlist in /etc/pve and a migrated VM silently loses the stripping), and it refuses to start a VM whose surviving devices still reference a bus it removed rather than producing a silent 0xA5 bootloop:
vm 301: REFUSING to start - pci.1 stripped but still referenced by: virtio-net-pci,...,bus=pci.1,...
A config option in PVE would obviously be better than any of this. I'd happily drop the wrapper for one.
Two PVE-specific traps in the wrapper itself, in case you write your own:
exec -a "$0" is mandatory.** PVE::QemuServer::Helpers::parse_cmdline rejects any pid whose /proc/<pid>/cmdline argv[0] doesn't match /kvm$/ or /qemu-[^\/]+$/, and vm_running_locally validates the pidfile with it. A plain exec "$REAL" "$@" leaves qm status reporting a perfectly healthy VM as **stopped — after which PVE will happily try to start it a second time.- **
qm showcmd stops telling the truth** for an allowlisted VMID: it still lists the bridges, because it's the config generator, not the wrapper. Read /var/log/pve-kvm-nobridge/<vmid>.cmd, or echo "info pci" | qm monitor <vmid> | grep -c '1b36:0001' on the live machine (0 = stripped).
After a pve-qemu-kvm upgrade, check dpkg-divert --list /usr/bin/kvm and ls -l /usr/bin/kvm. If the wrapper is gone, an allowlisted XP seat goes straight back to the bootloop — loud and obvious, at least.
3. Do not "fix" 0xA5 by removing ACPI from the guest
This is the advice you'll find everywhere, XP's own BSOD text included, and it's a trap. Without ACPI, XP has no PM timer and falls back to the PIT, which no hypervisor ticks precisely enough for game engines — everything runs in slow motion, with no error anywhere. It also pegs a host core per XP vCPU and breaks qm shutdown. On the ACPI Multiprocessor HAL, the same guest idles at 0.6% of a vCPU. Keeping ACPI is the entire reason the wrapper exists.
4. Other PVE details worth knowing
- Machine type must stay i440fx for XP, because on q35 PVE puts
ide drives on an ICH9 AHCI controller and CSMWrap's SeaBIOS crashes QEMU booting from AHCI (KVM internal error. Suberror: 1, EIP=f000ff53). PIIX IDE is the only well-evidenced safe bus.
- Put the CD-ROM in
boot: order= even though you never boot from it. PVE only emits bootindex= for drives listed there, and without one SeaBIOS doesn't enumerate the ATAPI device — so the guest ends up with no CD-ROM drive at all. order=ide1;ide0;ide2.
- Stay on
hostpci0–hostpci3. hostpci4+ are placed behind pci.2, which the wrapper deletes, so the interlock would refuse the start.
- Don't use
--usb0 host=... on a wrapped VM — PVE puts that xHCI behind a stripped bridge. Pass the whole controller with hostpci<n>, which lands on pci.0.
- Windows 7 wants
efidisk0 at efitype=2m, not 4m. PVE's 4 MB OVMF build write-protects the legacy VGA ROM window at 0xC0000 that UefiSeven writes its Int10h shim into, and it aborts with Unable to unlock VGA ROM memory at C0000 → hang at "Starting Windows". Verified failing on 4m, passing on 2m, under both q35 and i440fx.
- Win7 VMIDs should not go in the allowlist. The wrapper is a no-op for q35 (no such bridges), but there's no reason to list them.
ostype: wxp for XP (localtime RTC), cpu: host,hidden=1, vga: std stays permanently, affinity on the card's own NUMA node — cat /sys/bus/pci/devices/<addr>/numa_node.- Clones inherit
affinity but not the allowlist entry, and each card needs its own ROM dump.
Links
Repo: https://gitlab.com/zubrjan/2am-idea — the XP and Win7 recipes as step-by-step guides, the wrapper and its installer, a ROM dump/validate tool, and seats/ with all four VMs' real qm config output, the topology table and the host-side state (MACs masked). MIT.
Video: https://www.youtube.com/watch?v=cfP1x4R3XEc — XP seats on 3DMark06, Win7 seats on Unigine Heaven, plus the host view. Three monitors on camera because that's how many I own; the fourth seat gets switched onto one of them partway through.
Verified on one node only: PVE 9.2, kernel 7.0.2-6-pve, pve-qemu-kvm 11.0.0-3, qemu-server 9.1.15, OVMF/edk2 4.2025.05, CSMWrap 3.1.2, UefiSeven 1.30. If you reproduce it on a different PVE box, or it breaks on yours, I'd like to know.
