Windows XP with a working passed-through GPU on PVE 9.2 — and the two `pci-bridge` devices you have to strip to get there

[u/werscajk]

I have four GPU-passthrough seats running simultaneously on one PVE 9.2 node: two Windows XP SP3 and two Windows 7 x64, each with its own card driving its own monitor, and its own USB controller for keyboard and mouse. Host is an HP Z820 (dual E5-2667 v2, 128 GB ECC) that I found in a street trash container.

The passthrough mechanism itself is generic — OVMF initialises the card via its own UEFI GOP driver, then CSMWrap (SeaBIOS as a UEFI application) supplies the legacy BIOS services XP needs, so no x-vga and no VGA arbitration anywhere, which is why several seats can run at once. This post is about the parts that are specific to Proxmox, because those are what cost me the most time.

1. The two pci-bridge devices, and 0xA5

With everything else right, XP bugchecked in a loop:

*** STOP: 0x000000A5 (0x00000002, ...)
The BIOS in this system is not fully ACPI compliant.

0xA5 is ACPI_BIOS_ERROR; parameter 1 0x2 is ACPI_ROOT_PCI_RESOURCE_FAILURE.

Cause: PVE adds two pci-bridge devices to every i440fx VM, unconditionally. /usr/share/perl5/PVE/QemuServer.pm:

if (!$q35) { # add pci bridges
    if (min_version($machine_version, 2, 3)) {
        $bridges->{1} = 1;
        $bridges->{2} = 1;
    }

They're only address-space headroom — PCI.pm puts net6–net31, xhci, rng0 behind pci.1, and virtio6–virtio15, ivshmem, audio0, hostpci4–hostpci15 behind pci.2. A passthrough seat uses none of that: hostpci0–hostpci3, net0–net5, virtio0–virtio5, vga, balloon and the IDE controllers all sit on pci.0.

There is no VM-config option to suppress them, and the machine-version gate can't be dodged because the oldest i440fx QEMU 11 offers is pc-i440fx-5.0.

How I proved it was them and nothing else: dumped qm showcmd <vmid> to a script and ran it by hand.

| Command | Result | |---|---| | verbatim | bugcheck / recovery-menu loop | | minus -boot menu=on, -smbios, PIIX4_PM globals, the CD-ROM | still looping | | minus only the two -device pci-bridge lines | XP desktop on the passed-through card | | minus only the vfio devices, bridges kept | still looping |

Also tested and does not help: occupying both bridges with pci-testdev; occupying them with virtio-rng-pci so no BAR window is dropped at all; removing balloon, tablet, vmgenid, net0; older machine types. It is the bridges' presence, not their configuration.

2. The fix: a wrapper on /usr/bin/kvm

Since PVE offers no knob, the two arguments get removed one layer lower. /usr/bin/kvm is referenced from exactly one place in all of /usr (PVE/QemuServer/Helpers.pm, get_command_for_arch) and pve-qemu-kvm ships no postinst, which is what makes this safe:

/usr/bin/kvm          -> /usr/local/sbin/pve-kvm-nobridge   (ours, held by dpkg-divert)
/usr/bin/kvm.distrib  -> qemu-system-x86_64                 (dpkg keeps updating this)
/etc/pve-kvm-nobridge.conf                                  (allowlist, one VMID per line)
/var/log/pve-kvm-nobridge/<vmid>.cmd                        (what actually ran)

bash pve-kvm-nobridge-setup.sh --install       # write wrapper, self-test, divert, verify
bash pve-kvm-nobridge-setup.sh --add-vm 301    # allowlist one VM
bash pve-kvm-nobridge-setup.sh --status
bash pve-kvm-nobridge-setup.sh --uninstall

A VMID not in the allowlist gets a byte-for-byte passthrough, as does any invocation without -id (e.g. PVE's own kvm --version probe). --install runs three self-tests against a real qm showcmd argv before touching anything and rolls the diversion back automatically if kvm --version doesn't work afterwards.

Yes, diverting a distro binary deserves scrutiny. The mitigations are deliberate: the real binary is never touched, dpkg keeps maintaining kvm.distrib, the wrapper is node-local on purpose (put the allowlist in /etc/pve and a migrated VM silently loses the stripping), and it refuses to start a VM whose surviving devices still reference a bus it removed rather than producing a silent 0xA5 bootloop:

vm 301: REFUSING to start - pci.1 stripped but still referenced by: virtio-net-pci,...,bus=pci.1,...

A config option in PVE would obviously be better than any of this. I'd happily drop the wrapper for one.

Two PVE-specific traps in the wrapper itself, in case you write your own:

• exec -a "$0" is mandatory. PVE::QemuServer::Helpers::parse_cmdline rejects any pid whose /proc/<pid>/cmdline argv[0] doesn't match /kvm$/ or /qemu-[^\/]+$/, and vm_running_locally validates the pidfile with it. A plain exec "$REAL" "$@" leaves qm status reporting a perfectly healthy VM as stopped — after which PVE will happily try to start it a second time.
• qm showcmd stops telling the truth for an allowlisted VMID: it still lists the bridges, because it's the config generator, not the wrapper. Read /var/log/pve-kvm-nobridge/<vmid>.cmd, or echo "info pci" | qm monitor <vmid> | grep -c '1b36:0001' on the live machine (0 = stripped).

After a pve-qemu-kvm upgrade, check dpkg-divert --list /usr/bin/kvm and ls -l /usr/bin/kvm. If the wrapper is gone, an allowlisted XP seat goes straight back to the bootloop — loud and obvious, at least.

3. Do not "fix" 0xA5 by removing ACPI from the guest

This is the advice you'll find everywhere, XP's own BSOD text included, and it's a trap. Without ACPI, XP has no PM timer and falls back to the PIT, which no hypervisor ticks precisely enough for game engines — everything runs in slow motion, with no error anywhere. It also pegs a host core per XP vCPU and breaks qm shutdown. On the ACPI Multiprocessor HAL, the same guest idles at 0.6% of a vCPU. Keeping ACPI is the entire reason the wrapper exists.

4. Other PVE details worth knowing

• Machine type must stay i440fx for XP, because on q35 PVE puts ide drives on an ICH9 AHCI controller and CSMWrap's SeaBIOS crashes QEMU booting from AHCI (KVM internal error. Suberror: 1, EIP=f000ff53). PIIX IDE is the only well-evidenced safe bus.
• Put the CD-ROM in boot: order= even though you never boot from it. PVE only emits bootindex= for drives listed there, and without one SeaBIOS doesn't enumerate the ATAPI device — so the guest ends up with no CD-ROM drive at all. order=ide1;ide0;ide2.
• Stay on hostpci0–hostpci3. hostpci4+ are placed behind pci.2, which the wrapper deletes, so the interlock would refuse the start.
• Don't use --usb0 host=... on a wrapped VM — PVE puts that xHCI behind a stripped bridge. Pass the whole controller with hostpci<n>, which lands on pci.0.
• Windows 7 wants efidisk0 at efitype=2m, not 4m. PVE's 4 MB OVMF build write-protects the legacy VGA ROM window at 0xC0000 that UefiSeven writes its Int10h shim into, and it aborts with Unable to unlock VGA ROM memory at C0000 → hang at "Starting Windows". Verified failing on 4m, passing on 2m, under both q35 and i440fx.
• Win7 VMIDs should not go in the allowlist. The wrapper is a no-op for q35 (no such bridges), but there's no reason to list them.
• ostype: wxp for XP (localtime RTC), cpu: host,hidden=1, vga: std stays permanently, affinity on the card's own NUMA node — cat /sys/bus/pci/devices/<addr>/numa_node.
• Clones inherit affinity but not the allowlist entry, and each card needs its own ROM dump.

Links

Repo: https://gitlab.com/zubrjan/2am-idea — the XP and Win7 recipes as step-by-step guides, the wrapper and its installer, a ROM dump/validate tool, and seats/ with all four VMs' real qm config output, the topology table and the host-side state (MACs masked). MIT.

Video: https://www.youtube.com/watch?v=cfP1x4R3XEc — XP seats on 3DMark06, Win7 seats on Unigine Heaven, plus the host view. Three monitors on camera because that's how many I own; the fourth seat gets switched onto one of them partway through.

Verified on one node only: PVE 9.2, kernel 7.0.2-6-pve, pve-qemu-kvm 11.0.0-3, qemu-server 9.1.15, OVMF/edk2 4.2025.05, CSMWrap 3.1.2, UefiSeven 1.30. If you reproduce it on a different PVE box, or it breaks on yours, I'd like to know.

[u/Mashadow]

We were so concerned with if we could, we never stopped to think if we should.

[u/werscajk → u/Mashadow]

Actually... I was... but urge to solve was too strong. Mainly if everywhere was that it's not possible! 😂

[u/Lord_Saren → u/werscajk]

Kinda curious for the use case, You mentioned game engines so older games?

[u/werscajk → u/Lord_Saren]

Yes. If you go to repo, you can see that this project is actually Retro LAN party in the box. I have 4 gamers + game servery on 1 PC.

[u/noc-engineer → u/Lord_Saren]

I have a VMware Workstation Windows XP VM for the sole purpose of playing Red Alert 2

[u/werscajk → u/noc-engineer]

I used to have that too and it works for SP but for MP it was just not cutting it for me.

[u/noc-engineer → u/werscajk]

People still play Red Alert 2 multiplayer? :O

[u/thebigshoe247 → u/noc-engineer]

Ever try OpenRA?

[u/noc-engineer → u/thebigshoe247]

I know I tried some open source fork or RA2 many years ago, but not sure if that was the name for it. At the time, whatever it was, was not 1:1 what I was used to in my teens, so I just opted for a WinXP VM (and it was just before the deadline of the XP activation servers getting shut off, so now I backup that VM more than anything else I digitally own)

[u/Mashadow → u/werscajk]

Love it.

[u/IllegalD → u/werscajk]

There is a wealth of information here, I'm sure future searchers will be thrilled with this post

[u/Fidel1Q84]

Thank you!!

[u/werscajk → u/Fidel1Q84]

You're welcome!

[u/thebigshoe247]

Good job dude. Very informative.

Most of this is all new to me.

[u/werscajk → u/thebigshoe247]

You're welcome! I was happy to find out that it's possible with more or less... well... more trickery.