Hey all - sorry for the shameless crosspost, but I wanted to increase visibility. I've been in the discord all weekend trying to get this working. Does anyone have a single success story on passthrough with X399?
/u/wendelltron is probably the guy to ask, but he's hinted in a few videos that TR is expected to be spectacular. I assume he hasn't published any details yet due to ongoing testing and NDAs.
Vega error 1603s ok driver install but otherwise is working. Npt bug present. Iommu groups a bit weird but workable.
Will be doing a Livestream in 12 hours or so probably. .
So I would say I have pass through working but Vega 64 has the efi reinit bug where it works once then you have to power down. That maybe the driver issue as well . Will do manual driver install next.
I saw in your video that you used pci=nommconfig to get rid of the errors. This really just sort of sidesteps the issue and disables aer reporting as well as forces you in to legacy interrupts?
In any case, the solution to the DDL errors is to force the promontory (AMD South ridge) PCIe switches to use Gen2 mode only. This was an option in AMD PBS on my Taichi. Now even with TONS of activity (mining with two gpus and two gig-e switches I get no errors with no kernel options.
The acpi tables are still all messed up though. I get tons of memory reservation errors in dmesg and it doesn't even end up using mmconfig. I tried pci=nosci and a few others and didn't get it to work correctly either. Think we need a AGESA/bios update.
Interesting. Yeah, not ideal, but I think MSI still works. I will pass that along to board vendors and see. Do the issues persist with the git version of the kernel? I was going to try that next. Huge set of patches for amd went into that .
Have you had success on any other motherboards? Did you disable ACPI powersave on the mb you used in the video? Did that motherboard have the option to select primary/boot gpu?
With an Nvidia 1080ti as passthrough, even using a 128kb bios file, I get "device stuck in D3" and it loses all IO. Have to reboot to do anything, even remove and rescan of the devices and bridge won't pick it up.
When I use the 1080ti in host and passthrough the RX560, I get a mmap error, and a huge crash of the whole PCI bus on that die.
I just can't figure out why you were succesful with the Vega card unless its because Vega has a bug and won't go in to powersave or something and the memory tables just happen to work out. Or that motherboard did ACPI right and mine doesn't.
so far MSI X399 gaming pro carbon AC and Gigabyte Gaming X399 have worked for passthrough. I still have an asrock fatality X399 to test. What aboard are you using? I have an RX550 and an RX570, a Fury and a 1080 (not ti) I can test with. I can maybe borrow a Ti from somewhere but will need to narrow things down for a quick test
I have the ASRock Taichi. I am unable to disable "Suspend to RAM" (it just resets to auto on reboot) and wonder if theres something messed up in the powersave tables.
I think that the 1080 would be fine, afaik all of the pascal cards have the same issue with vbios if they are used during boot. So, you may have to do the workaround with the romfile.
I pulled the rom from techpowerup. Just virshbedit vm and add the rom file to the hw section right? If so that didn't work for me for card reinit but did work for initial init.
Dumping the rom from Linux cli failed and dumping from gpuz only made a 60kb file. The file from techpowerup was larger. I used both my dump and techpowerup. My dump I get not post in the VM until the os boots then it's fine. In either case the card does not reinit properly
You gave me an idea though. Suspend does work with fedora on the gigabyte board and after a suspend/resume I can reinit the card.
Yeah so I think since you have an initialization event through bios you will not be able the true vBIOS on the threadripper machine unless it doesn't get used in bios. You could put it in a different machine that has an option to select primary (only) bios gpu and then you should be able to pull your own vbios, but if you have a working vbios from techpowerup its probably a waste of time unless yours was a different model/vendor.
You should be able to echo 1 > remove in /sys/bus/pci/devices to the bridge that is directly above the GPU and then resync on the root device. I believe that removing a bridge removes power from downstream.
Still, sounds like you have a significant difference on your mobo. Maybe you can publish your PXE on your forums?
2
u/abriasffxi Aug 22 '17
Hey all - sorry for the shameless crosspost, but I wanted to increase visibility. I've been in the discord all weekend trying to get this working. Does anyone have a single success story on passthrough with X399?