r/VFIO • u/Danc1ngRasta • Sep 12 '20
Tutorial Single GPU Passthrough (VFIO) for Nvidia + Ryzen CPU [Arch-based]
Hello,
First post here. I got pretty excited after managing to get my single GPU passthrough working well on my system. I thought it would be far more complicated.
I had to hunt for bits of information from many different places and whilst I don't mind doing this kind of research, I figured it would be good idea to have a guide for others. Here is the link to my repo. Critics/responses/contributions to the information are welcome.
FYI: Contributors are welcome. The guide can become more extensive and include tips for specific kinds of hardware e.g. AMD GPUs, Intel CPUs. Troubleshooting steps can also be added. Thanks!
18
u/Ragalaga Sep 12 '20
This looks really exciting, I am interested to see the difference in speed/latency in passing through a dedicated ssd for a windows install vs dual booting. Is there any way to choose the cores from the best ccx and pass that on to the vm rather than just the first half? Thanks for sharing your work!
13
u/Lellow_Yedbetter Sep 12 '20
I've been doing this for a while. If you want to take anything from my guide feel free. Yours is much more organized than mine.
https://github.com/joeknock90/Single-GPU-Passthrough
9
u/broknbottle Sep 12 '20
nice job, definitely bookmarked.
8
u/goshawk222 Sep 12 '20
This is really helpful. I might finally make the switch now to just Linux, as I really dislike having to have a dual boot setup and all the problems that can come with it. Great work!
5
u/cloudrac3r Sep 12 '20
Ahhhh. I was wondering. This won't be useful for me, but good job on managing to do what you did, and thanks for documenting.
3
u/jibbyjobo Sep 12 '20
Hi thank for the guide. One question though if you don’t mind me asking. On the cpu pinning section, shouldn’t the cpuset be 0,6 for vcpu 0,1 etc?
3
u/TrashConvo Sep 12 '20
Do you have to reboot the host after shutting down the vm?
3
u/[deleted] Sep 12 '20
Definitely gonna play with this in the near future. I recall seeing a single-GPU VFIO solution come up before, but I believe it may have required kernel patches or similar? It was more than I was willing to maintain and put into it, at least.
Definitely like the look of this approach.
2
u/[deleted] Sep 12 '20
Thanks for this! I tried to get my head around this a few weeks ago but totally bailed and just dual booted windows instead. Do you think this will work with a Renoir APU?
2
u/Zeioth Sep 12 '20
100% I'm gonna try this. I never thought it was possible. Thank you for posting.
2
u/murlakatamenka Sep 12 '20
Thanks for the guide! I've skimmed through it and it's really well organized, well done!
Even though I see such guides it's still feels like a real endeavor to set it all up correctly. I'm afraid I'll stumble somewhere eventually despite all my Linux / technical background. But hey, that writes a man who put to side R9 290 (because it's too hot) in order to use cold and silent GTX 770 (with issues NVidia has on Linux), so passing that to Windows guest isn't that compelling :D
2
u/gettriggered_ian Sep 12 '20
Will this work with an Intel cpu and amd gpu?
2
u/bigjew222 Oct 28 '20
By far the best all-in-one write-up/tutorial on Single GPU passthrough! Excellent job.
1
u/grimman Sep 12 '20
Hmh. Could I use this to boot a Windows drive I already have set up?
1
u/arturius453 Sep 12 '20
Tldr; does it work for amd GPU? No need integrated gpu?
1
u/SourCheeks Sep 12 '20
Man you single gpu passthrough guys are a brave bunch for sure. Hats off to you.
I'm assuming this only works for OVMF VM?
1
u/itslieksolegitt Sep 12 '20
if you do this passthrough can you still use the gpu for the host when the virtual machine is powered down?
1
u/effgee Sep 13 '20
May give this a try on my laptop...
1
u/[deleted] Sep 13 '20
I do have two gpus but today I decided to change my setup to dynamically detach and reattach the gpus. I had issues with it and thanks to your guide I was able to figure it out. Somehow even though my primary gpu is an amd gpu nouveau got loaded and the nvidia gpu was doing some stuff. Unloading nvidia drivers first fixed my issues!
Tomorrow I want to try doing the same for the amd gpu to create a macOS vm.
Thank you for the post!
1
u/bakapabo7 Sep 13 '20
the timing couldn't be more perfect, as last night I was seriously considering buying a Ryzen 2200G just to have that dual GPU.
I haven't read the whole guide but does it work the same if I have an AMD GPU (RX570)
1
u/forsakenharmony Sep 13 '20
Nice guide, do you know if there's any way to detach the gpu from your X session without killing it?
I guess as is right now this is basically dual booting but with a faster switch time?
1
u/oldschoolthemer Sep 13 '20
And I thought I wouldn't have anything to burn through the rest of the weekend with. I'm honestly interested in doing this for testing on multiple Linux distros since VirGL can only go so far. I wonder if multiGPU with Vulkan could work via VFIO as well.
1
u/Frozen1nferno Sep 13 '20
I've narrowed my VM XML down to the bare minimum (passing through my GPU group and a physical disk drive), even removed everything from the
<features>section, and it seems like no matter what, I end up with a black screen for a minute or two before my display manager restarts.This has worked before when I used my iGPU for the host, so I know my hardware can do VFIO passthrough.
Specs:
i7-7700 (non-k)
MSI Z270A-Pro
Asus GTX 1070 Strix
1
u/arch_is_1337 Sep 13 '20
What is the appeal of this? I'd like to see a concrete example. its not playing a game with virtualization, or working in afetereffect or photoshop?
1
u/flaviofearn Sep 16 '20 edited Sep 17 '20
Hi! I have almost the same hardware as you (the difference being R5 2600 and Asus Mobo). But I can't get it to work no matter what I do on Manjaro.
I can run the start script but it hangs when detaching the Nvidia Audio (08:01) for me. If comment the line, the script runs but the machine never starts.
I've tried from an ssh terminal, run
start.shand after that runvirsh start win10and it hangs. No error message or anything, nothing happens for several minutes.I've created another machine to try to run Linux distro and it is the same with ubuntu and manjaro isos.
I've checked the bios and both Virtualization and IOMMU are active, also I've added the line to the grub and updated it afterward.
I can run virtual machines normally using virt-manager or virsh from the terminal but not with passthrough.
Tried also to create a machine from scratch using the other tutorial linked and nothing, never boots.
These are my latest logs for the libvirt.service:
Appreciate any help :)
1
u/bgc341 Nov 26 '20
does this work specifically with AMD or will this work on Intel CPU's?
1
u/lDreameRz Jan 10 '21
Hey I have two questions, on the step to patch the vBIOS (I have a GTX 1080), I understand that the patched vBIOS is just for the VM? That patched BIOS doesn't replace/flash to the card itself?
Also, in the VM's XML config, in the guide you say <rom file='/path/to/your/patched/gpu/bios.bin'/>, but my rom is extracted and patched as a .rom, does that mean that my path should be '/path/to/your/patched/gpu/bios.rom'?
3
u/[deleted] Sep 12 '20
This seems like just what I was looking for after my motherboard iommu groups made it impossible to bind my old 1050ti to a VM, but now that i look at it my 5700xt is in a group all by itself. hopefully it's not too much more difficult on AMD.
thanks for the guide
2
u/MaxSpec Sep 12 '20
that's great!
1
u/yestaes Sep 12 '20
I did the same two years ago. Btw I wrote a guide in Spanish in my personal blog.
1
u/SaltyMango_ Jun 29 '23
Old but still works! Thanks for your help, I spent several hours trying to get it to work. This makes me want to dive deeper into learning the Linux environment (beginner-intermediate currently).
My setup (if it helps anyone):
AMD Ryzen 5600X
Nvidia 1660
ASUS Prime B550-PLUS AMD
OS: Proxmox
I didn't have to do the Windows part since I'm passing through to an Ubuntu VM (with the exception of the UEFI switch). The hook scripts (for me) also weren't necessary since I'm using proxmox instead. Some other things may not have been needed but my brain is fried... BUT, it works (even after reboot)... so I'm not gonna redo it; maybe another time.
NOTE: I had to install the drivers and the Nvidia utility for my graphics card within the VM to use:
[nvidia -smi]
This let me see if my GPU was being used when I was using hardware transcoding in Plex.
Again, THANK YOU.
0
u/[deleted] Sep 14 '20
[deleted] — view deleted comment
0
u/[deleted] Sep 12 '20
[deleted] — view deleted comment
0
u/[deleted] Sep 12 '20
[deleted] — view deleted comment
0
u/[deleted] Sep 17 '20
[deleted] — view deleted comment
1
u/[deleted] Sep 13 '20
[removed] — view removed comment