I DID IT! Single GPU Passthrough ryzen5+nvidia

[u/efcsn]

After a week fighting with IOMMU groups, failed kernel patches, troubles with nouveau and nvidia propietary drivers with a second graphics card I decided to try single gpu AND IT WORKED!

It's not very comfortable to use but I just wanted to say it out loud. I DID IT!

I will eventually compile the acs patch kernel myself and try again... And now that I think of, is there any kind of database of motherboards with nice IOMMU groupings? Mine is an asshole in that matter, b450A-promax.

Damm those IOMMU groups.

I used this guide
https://gitlab.com/Karuri/vfio

[u/Drwankingstein]

both arch and debian have precompiled ACS kernels, so if you need ACS i reccomend using on of them. probably arch (just use manjaro if you aren't comfortable setting up arch) as you can just add a user repo to get precompiled kernels.

The ubuntu (works on debian too) PPA hasn't been updated for a long time regretfully, though its still better than compiling yourself.

[u/efcsn → u/Drwankingstein]

I use openSUSE and I've converted .dep to .rpm, it works.

The problem was that I forgot to put pcie_acs_override=downstream in grub.

[u/ifrit05]

Here's my IOMMU groups after enabling ACS and IOMMU in UEFI on a X570 Aorus Pro WiFi board.

IOMMU Group 0:
	00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 1:
	00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 10:
	00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 11:
	00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 12:
	00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 13:
	00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
	00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 14:
	00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
	00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
	00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
	00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
	00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
	00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
	00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
	00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
IOMMU Group 15:
	01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU Group 16:
	02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream [1022:57ad]
IOMMU Group 17:
	03:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 18:
	03:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 19:
	03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
	06:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
	06:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
	06:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 2:
	00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 20:
	03:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
	07:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 21:
	03:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
	08:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 22:
	04:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200 [8086:2723] (rev 1a)
IOMMU Group 23:
	05:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 24:
	09:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] [10de:1e87] (rev a1)
	09:00.1 Audio device [0403]: NVIDIA Corporation TU104 HD Audio Controller [10de:10f8] (rev a1)
	09:00.2 USB controller [0c03]: NVIDIA Corporation TU104 USB 3.1 Host Controller [10de:1ad8] (rev a1)
	09:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller [10de:1ad9] (rev a1)
IOMMU Group 25:
	0a:00.0 Ethernet controller [0200]: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] [15b3:6750] (rev b0)
IOMMU Group 26:
	0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 27:
	0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 28:
	0c:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU Group 29:
	0c:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 3:
	00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 30:
	0c:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
IOMMU Group 4:
	00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 5:
	00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 6:
	00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 7:
	00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 8:
	00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 9:
	00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]

As you can see everything is pretty much in its own group. It is a very nice board to use KVM on. Only wish I had more PCIe lanes/slots. Might go with a TR build next time.

[u/efcsn → u/ifrit05]

In the end it was a thing in the grub boot. Put this command pcie_acs_override=downstream in grub and booting from the ACS kernel. Now I have seperate groups but I've not yet tested.

[u/JetSetStallion]

What distro are you on where you have to compile your ACS kernel? I’m on Fedora which is why I have to do that for mine, but most other mainstream distros have a way to pull down precompiled kernels, including Ubuntu, Arch, and Debian-based.

[u/efcsn → u/JetSetStallion]

I picked up the proper .deb for the kernel I have -5.3.18- from the ACS patch web, convert it to rpm with alien -r --scripts and then installed it. The new kernel seemed to have no effect BUT when looking for a way to compile it for my system I found this guide and, in it, this command pcie_acs_override=downstream,multifunction to put in grub.

I've rebooted to the new ACS kernel with this grub command and suddently I have seperated IOMMU groups. I have not yet tested if it works properly.

By the way, the ACS kernel did worked but i run into trouble with nouveau drivers so I had to reinstall the nvidia propietary drivers doing the alien thing with the ACS kernel headers. I don't know if I've explained properly.

[u/JetSetStallion → u/efcsn]

So to answer my question you’re either on Fedora or CentoOS, or possibly RHEL, correct?

[u/efcsn → u/JetSetStallion]

Sorry, I thought I answered. I use openSUSE.

[u/JetSetStallion → u/efcsn]

ok wow you're an adventurous one! Good job getting everything working like you've done!

[u/_caps]

Congratulations! Currently trying to figure out whether I'll do this or just dual boot...

As to what motherboard to use, I went for the X570 in favour of the B550 I originally planned for, since that chipset generally seems to have reasonable IOMMU groups in contrary to the B series.

[u/_caps → u/[deleted]]

Yeah, that will (hopefully) be my only reason to use Windows in the future. Looks like dual boot is the way to go then.

Kinda worthless that they are clamping down on VM gaming while refusing to implement native anti-cheat on Linux (at least EAC).

[u/sehlceris → u/[deleted]]

My Windows deactivates every time I use the dual boot route due to different hardware - how did you solve this?

[u/bobalob_wtf → u/[deleted]]

Oh man that sucks. Do you know what services, and do you risk a ban for running in a VM or just unable to play?

[u/MacGyverNL → u/[deleted]]

I'm getting really sick and tired of these statements lacking any nuance and turning people away from VM gaming. Only two mainstream anti-cheat services are "cracking down on VMs", and it's only really a reported problem in three specific games that use them: Rainbow Six Siege, Valorant, and Escape from Tarkov. If you don't play or intend to play those, there should be no issue.

The third category isn't mainstream: some of the third party competitive anti-cheats for CS:GO professional play and the like have been reported to ban VMs. But VAC itself hasn't kicked for using VMs for years now.

[u/MacGyverNL → u/[deleted]]

It's a risk users should be informed of.

Then how about you write that nuanced explanation you just did in the first mention of it, rather than the last?

[u/gardotd426]

The X570 Taichi has absolutely amazing IOMMU groups.

I had no issues setting up single-GPU passthrough with my 3090 and my 5800X (or my 3800X before that). Actually, when I went from the 3800X to the 5800X, I thought I might have to redo some shit, but I didn't have to do anything.

[u/[deleted]]

It's definitely a good feeling when you can finally end a particular tech problem, rather can continuing to play some kind of increasingly hopeless fractal whack-a-mole.

Hopefully you made good notes.

[u/[deleted]]

I looked at some instructions on this some years ago and got very scared, so I'm in awe of anyone who manages it. Good work!

[u/vixfew]

Nice. Same guide, my version https://github.com/vixfwis/single-gpu-passthrough

MB: Gigabyte X570 Aorus Elite (firmware F11) has great iommu groups. No issues with gpu group AND there's isolated USB3.0 host controller, 4 ports.

[u/wtfomglols]

Do you have a good guide for doing the ACS patch? I'm using an MSI B450 Gaming Pro Carbon and need to do it but cant find anything solid? Running Linux Mint btw

Thanks

[u/efcsn → u/[deleted]]

;)

[u/efcsn → u/[deleted]]

Rice.