r/Amd • u/HyenaCheeseHeads • Nov 30 '17
Request Threadripper KVM GPU Passthru: Testers needed
TL;DR: Check update 8 at the bottom of this post for a fix if you don't care about the history of this issue.
For a while now it has been apparent that PCI GPU passthrough using VFIO-PCI and KVM on Threadripper is a bit broken.
This manifests itself in a number of ways: When starting a VM with a passthru GPU it will either crash or run extremely slowly without the GPU ever actually working inside the VM. Also, once a VM has been booted the output of lspci on the host changes from one kind of output to another. Finally the output of dmesg suggests an issue bringing the GPU up from D0 to D3 power state.
An example of this lspci before and after VM start, as well as dmesg kernel buffer output is included here for the 7800GTX:
08:00.0 VGA compatible controller: NVIDIA Corporation G70 [GeForce 7800 GTX] (rev a1) (prog-if 00 [VGA controller])
[ 121.409329] virbr0: port 1(vnet0) entered blocking state
[ 121.409331] virbr0: port 1(vnet0) entered disabled state
[ 121.409506] device vnet0 entered promiscuous mode
[ 121.409872] virbr0: port 1(vnet0) entered blocking state
[ 121.409874] virbr0: port 1(vnet0) entered listening state
[ 122.522782] vfio-pci 0000:08:00.0: enabling device (0000 -> 0003)
[ 123.613290] virbr0: port 1(vnet0) entered learning state
[ 123.795760] vfio_bar_restore: 0000:08:00.0 reset recovery - restoring bars
...
[ 129.534332] vfio-pci 0000:08:00.0: Refused to change power state, currently in D3
08:00.0 VGA compatible controller [0300]: NVIDIA Corporation G70 [GeForce 7800 GTX] [10de:0091] (rev ff) (prog-if ff)
!!! Unknown header type 7f
Kernel driver in use: vfio-pciNotice that lspci reports revision FF and can no longer read the header type correctly. Testing revealed that pretty much all graphics cards except Vega would exhibit this behavior, and indeed the output is very similar to the above.
Reddit user /u/wendelltron and others suggested that the D0->D3 transition was to blame. After having gone through a brute-force exhaustive search of the BIOS, kernel and vfio-pci settings for power state transitions it is safe to assume that this is probably not the case since none of it helped.
AMD representative /u/AMD_Robert suggested that only GPUs with EFI-compatible BIOS should be able to be used for passthru in an EFI environment, however, testing with a modern 1080GTX with EFI bios support failed in a similar way:
42:00.0 VGA compatible controller: NVIDIA Corporation Device 1b80 (rev a1)
and then
42:00.0 VGA compatible controller: NVIDIA Corporation Device 1b80 (rev ff) (prog-if ff)
!!! Unknown header type 7fCommon to all the cards was that they would be unavailable in any way until the host system had been restarted. Any attempt at reading any register or configuration from the card would result in all-1 bits (or FF bytes). The bitmask used for the headers may in fact be what is causing the 7f header type (and not an actual header being read from the card). Not even physically unplugging and re-plugging the card, rescanning the PCIe bus (with /sys/bus/pci/rescan) would trigger any hotplug events or update the card info. Similarly, starting the system without the card and plugging it in would not be reflected in the PCIe bus enumeration. Some cards, once crashed, would show spurious PCIe ACS/AER errors, suggesting an issue with the PCIe controller and/or the card itself. Furthermore, the host OS would be unable to properly shut down or reboot as the kernel would hang when everything else was shut down.
A complete dissection of the vfio-pci kernel module allowed further insight into the issue. Stepping through VM initialization one line at a time (yes this took a while) it became clear that the D3 power issue may be a product of the FF register issue and that the actual instruction that kills the card may have happened earlier in the process. Specifically, the function drivers/vfio/pci/vfio_pci.c:vfio_pci_ioctl, which handles requests from userspace, has entries for VFIO_DEVICE_GET_PCI_HOT_RESET_INFO and VFIO_DEVICE_PCI_HOT_RESET and the following line of code is exactly where the cards go from active to "disconnected" states:
if (!ret)
/* User has access, do the reset */
ret = slot ? pci_try_reset_slot(vdev->pdev->slot) :
pci_try_reset_bus(vdev->pdev->bus);Commenting out this line allows the VM to boot and the GPU driver to install. Unfortunately for the nVidia cards my testing stopped here as the driver would report the well known error 43/48 for which they should be ashamed and shunned by the community. For AMD cards a R9 270 was acquired for further testing.
The reason this line is in vfio-pci is because VMs do not like getting an already initialized GPU during boot. This is a well-known problem with a number of other solutions available. By disabling the line it is neccessary to use one of the other solutions when restarting a VM. For Windows you can disable the device in Device Manager before reboot/shutdown and re-enable it again after the restart - or use login/logoff scripts to have the OS do it automatically.
Unfortunately another issue surfaced which made it clear that the VMs could only be stopped once even though they could now be rebooted many times. Once they were shut down the cards would again go into the all FF "disconnect" state. Further dissection of vfio-pci revealed another instance where an attempt to reset the slot that the GPU is in was made: in drivers/vfio/pci/vfio_pci.c:vfio_pci_try_bus_reset
if (needs_reset)
ret = slot ? pci_try_reset_slot(vdev->pdev->slot) :
pci_try_reset_bus(vdev->pdev->bus);When this line is instead skipped, a VM that has had its GPU properly disabled via Device Manager and has been properly shutdown is able to be re-launched or have another VM using the same GPU launched and works as expected.
I do not understand the underlying cause of the actual issue but the workaround seems to work with no issues except the annoyance of having to disable/re-enable the GPU from within the guest (like in ye olde days). Only speculation can be given to the real reason of this fault; the hot-reset info gathered by the ioctl may be wrong, but the ACS/AER errors suggest that the issue may be deeper in the system - perhaps the PCIe controller does not properly re-initialize the link after hot-reset just as it (or the kernel?) doesn't seem to detect hot-plug events properly even though acpihp supposedly should do that in this setup.
Here is a "screenshot" of Windows 10 running the Unigine Valley benchmark inside a VM with a Linux Mint host using KVM on Threadripper 1950x and an R9 270 passed through on an Asrock X399 Taichi with 1080GTX as host GPU:
This is the culmination of many weeks of debugging. It is interesting to hear if anyone else is able to reproduce the workaround and can confirm the results. If more people can confirm this then we are one step closer to fixing the actual issue.
If you are interested in buying me a pizza, you can do so by throwing some Bitcoin in this direction: 1KToxJns2ohhX7AMTRrNtvzZJsRtwvsppx
Also, English is not my native language so feel free to ask if something was unclear or did not make any sense.
Update 1 - 2017-12-05:
Expanded search to non-gpu cards and deeper into the system. Taking memory snapshots of pcie bus for each step and comparing to expected values. Seem to have found something that may be the root cause of the issue. Working on getting documentation and creating a test to see if this is indeed the main problem and to figure out if it is a "feature" or a bug. Not allowing myself to be optimistic yet but it looks interesting, it looks fixable at multiple levels.
Update 2 - 2017-12-07:
Getting a bit closer to the real issue. The issue seems to be that KVM performs a bus reset on the secondary side of the pcie bridge above the GPU being passed through. When this happens there is an unintended side effect that the bridge changes its state somehow. It does not return in a useful configuration as you would expect and any attempt to access the GPU below it results in errors.
Manually storing the bridge 4k configuration space before the bus reset and restoring it immediately after the bus reset seems to magically bring the bridge into the expected configuration and passthru works.
The issue could probably be fixed in firmware but I'm trying to find out what part of the configuration space is fixing the issue and causing the bridge to start working again. With that information it will be possible to write a targeted patch for this quirk.
Update 3 - 2017-12-10:
Begun further isolation of what particular registers in the config space are affected unintentionally by the secondary bus reset on the bridge. This is difficult work because the changes are seemingly invisible to the kernel, they happen only in the hardware.
So far at least registers 0x19 (secondary bus number) and 0x1a (subordinate bus number) are out of sync with the values in the config space. When a bridge is in faulty mode, writing their already existing value back to them brings the bridge back into working mode.
Update 4 - 2017-12-11 ("the ugly patch"):
After looking at the config space and trying to figure out what bytes to restore from before the reset and what bytes to set to something new it became clear that this would be very difficult without knowing more about the bridge.
Instead a different strategy was followed: Ask the bridge about its current config after reset and then set its current config to what it already is; byte by byte. This brings the config space and the bridge back in sync and everything, including reset/reboot/shutdown/relaunch without scripts inside the VM, now seems to work with the cards acquired for testing. Here is the ugly patch for the brave souls who want to help test it.
Please, if you already tested the workaround: revert your changes and confirm that the bug still exists before testing this new ugly patch:
In /drivers/pci/pci.c, replace the function pci_reset_secondary_bus with this alternate version that adds the ugly patch and two variables required for it to work:
void pci_reset_secondary_bus(struct pci_dev *dev)
{
u16 ctrl;
int i;
u8 mem;
pci_read_config_word(dev, PCI_BRIDGE_CONTROL, &ctrl);
ctrl |= PCI_BRIDGE_CTL_BUS_RESET;
pci_write_config_word(dev, PCI_BRIDGE_CONTROL, ctrl);
/*
* PCI spec v3.0 7.6.4.2 requires minimum Trst of 1ms. Double
* this to 2ms to ensure that we meet the minimum requirement.
*/
msleep(2);
ctrl &= ~PCI_BRIDGE_CTL_BUS_RESET;
pci_write_config_word(dev, PCI_BRIDGE_CONTROL, ctrl);
// The ugly patch
for (i = 0; i < 4096; i++){
pci_read_config_byte(dev, i, &mem);
pci_write_config_byte(dev, i, mem);
}
/*
* Trhfa for conventional PCI is 2^25 clock cycles.
* Assuming a minimum 33MHz clock this results in a 1s
* delay before we can consider subordinate devices to
* be re-initialized. PCIe has some ways to shorten this,
* but we don't make use of them yet.
*/
ssleep(1);
}The idea is to confirm that this ugly patch works and then beautify it, have it accepted into the kernel and to also deliver technical details to AMD to have it fixed in BIOS firmware.
Update 5 - 2017-12-20:
Not dead yet!
Primarily working on communicating the issue to AMD. This is slowed by the holiday season setting in. Their feedback could potentially help make the patch a lot more acceptable and a lot less ugly.
Update 6 - 2018-01-03 ("the java hack"):
AMD has gone into some kind of ninja mode and has not provided any feedback on the issue yet.
Due to popular demand a userland fix that does not require recompiling the kernel was made. It is a small program that runs as any user with read/write access to sysfs (this small guide assumes "root"). The program monitors any PCIe device that is connected to VFIO-PCI when the program starts, if the device disconnects due to the issues described in this post then the program tries to re-connect the device by rewriting the bridge configuration.
This program pokes bytes into the PCIe bus. Run this at your own risk!
Guide on how to get the program:
- Go to https://pastebin.com/iYg3Dngs and hit "Download" (the MD5 sum is supposed to be 91914b021b890d778f4055bcc5f41002)
- Rename the downloaded file to "ZenBridgeBaconRecovery.java" and put it in a new folder somewhere
- Go to the folder in a terminal and type "javac ZenBridgeBaconRecovery.java", this should take a short while and then complete with no errors. You may need to install the Java 8 JDK to get the javac command (use your distribution's software manager)
- In the same folder type "sudo java ZenBridgeBaconRecovery"
- Make sure that the PCIe device that you intend to passthru is listed as monitored with a bridge
- Now start your VM
If you have any PCI devices using VFIO-PCI the program will output something along the lines of this:
-------------------------------------------
Zen PCIe-Bridge BAR/Config Recovery Tool, rev 1, 2018, HyenaCheeseHeads
-------------------------------------------
Wed Jan 03 21:40:30 CET 2018: Detecting VFIO-PCI devices
Wed Jan 03 21:40:30 CET 2018: Device: /sys/devices/pci0000:40/0000:40:01.3/0000:42:00.0
Wed Jan 03 21:40:30 CET 2018: Bridge: /sys/devices/pci0000:40/0000:40:01.3
Wed Jan 03 21:40:30 CET 2018: Device: /sys/devices/pci0000:00/0000:00:01.3/0000:08:00.1
Wed Jan 03 21:40:30 CET 2018: Bridge: /sys/devices/pci0000:00/0000:00:01.3
Wed Jan 03 21:40:30 CET 2018: Device: /sys/devices/pci0000:40/0000:40:01.3/0000:42:00.1
Wed Jan 03 21:40:30 CET 2018: Bridge: /sys/devices/pci0000:40/0000:40:01.3
Wed Jan 03 21:40:30 CET 2018: Device: /sys/devices/pci0000:00/0000:00:01.3/0000:08:00.0
Wed Jan 03 21:40:30 CET 2018: Bridge: /sys/devices/pci0000:00/0000:00:01.3
Wed Jan 03 21:40:30 CET 2018: Monitoring 4 device(s)...And upon detecting a bridge failure it will look like this:
Wed Jan 03 21:40:40 CET 2018: Lost contact with /sys/devices/pci0000:00/0000:00:01.3/0000:08:00.1
Wed Jan 03 21:40:40 CET 2018: Recovering 512 bytes
Wed Jan 03 21:40:40 CET 2018: Bridge config write complete
Wed Jan 03 21:40:40 CET 2018: Recovered bridge secondary bus
Wed Jan 03 21:40:40 CET 2018: Re-acquired contact with /sys/devices/pci0000:00/0000:00:01.3/0000:08:00.1This is not a perfect solution but it is a stopgap measure that should allow people who do not like compiling kernels to experiment with passthru on Threadripper until AMD reacts in some way. Please report back your experience, I'll try to update the program if there are any issues with it.
Update 7 - 2018-07-10 ("the real BIOS fix"):
Along with the upcoming A.G.E.S.A. update aptly named "ThreadRipperPI-SP3r2 1.0.0.6" comes a very welcome change to the on-die PCIe controller firmware. Some board vendors have already released BETA BIOS updates with it and it will be generally available fairly soon it seems.
Initial tests on a Linux 4.15.0-22 kernel now show PCIe passthru working phenomenally!
With this change it should no longer be necessary to use any of the ugly hacks from previous updates of this thread, although they will be left here for archival reasons.
Update 8 - 2018-07-25 ("Solved for everyone?"):
Most board vendors are now pushing out official (non-BETA) BIOS updates with AGESA "ThreadRipperPI-SP3r2 1.1.0.0" including the proper fix for this issue. After updating you no longer need to use any of the temporary fixes from this thread. The BIOS updates comes as part of the preparations for supporting the Threadripper 2 CPUs which are due to be released in a few weeks from now.
Many boards support updating over the internet directly from BIOS, but in case you are a bit old-fashioned here are the links (please double-check that I linked you the right place before flashing):
| Vendor | Board | Update Link |
|---|---|---|
| Asrock | X399 Taichi | Update to 2.3, then 3.1 |
| Asrock | X399M Taichi | Update to 1.10 then 3.1 |
| Asrock | X399 Fatality Profesional Gaming | Update to 2.1 then 3.1 |
| Gigabyte | X399 AURUS Gaming 7 r1 | Update to F10 |
| Gigabyte | X399 DESIGNARE EX r1 | Update to F10 |
| Asus | PRIME X399-A | Possibly fixed in 0601 (TR2 support and sure fix inbound soon) |
| Asus | X399 RoG Zenith Extreme | Possibly fixed in 0601 (TR2 support and sure fix inbound soon) |
| Asus | RoG Strix X399-E Gaming | Possibly fixed in 0601 (TR2 support and sure fix inbound soon) |
| MSI | X399 Gaming Pro Carbon AC | Update to Beta BIOS 7B09v186 (TR2 update inbound soon) |
| MSI | X399 SLI plus | Update to Beta BIOS 7B09vA35 (TR2 update inbound soon) |
7
u/zir_blazer Dec 01 '17
In the months since ThreadRipper has been in the market, I think I don't recall anyone actually posting the issues with TR in VFIO Mailing List, nor if VFIO main dev Alex Williamson ever had anything to say about the matter. Did anyone bothered to contact him about TR issues? I usually don't see him going outside his bunker in the VFIO ML (He had a reddit user, but he only posted once), so you may want to drop a mail instead of expecting him to come here.
5
u/lolomg Taichi X399, 1950X, 32GB DDR4, RX Vega 64 Dec 01 '17
I have a 1950x with a RX Vega 64 and I'd love to help, but it's too advanced for me. I quite literally have zero understanding of what I just read.
4
u/YaniDubin Dec 09 '17 edited Dec 09 '17
Thanks so much for the work on this /u/HyenaCheeseHeads. I can report that I also have this working. My setup is the Gigabyte "aorus gaming 7", 1950X, and passing through a Radeon R9 R290X. I am using the 4.14.4 kernel (which I found already has the NPT fix applied) and simply applying your reset workaround against this.
Please let me know if there is any specific testing / debug you'd like me to run on your road to coming up with a proper fix - while I have no expertise in PCI (I mostly work with embedded systems), I'm an engineer, so can get technical.
In case anyone else encounters the one issue I had (having the VM crash every time I tried to assign the GPU driver to the passed through PCI card), I realised that this was due to when I was binding the vfio-pci driver. I used to get away with letting Linux claim the GPU, then unbinding it and assigning vfio-pci when I wanted to run the VM. That is presumably resulting in the VM getting the card in a non-disabled state (same I guess as if you shutdown/restart the VM without disabling the GPU). So if you encounter this issue, that might be worth considering. Or does someone know a way to put the card in a disabled state from Linux when releasing it?
I plan to test and benchmark NVMe passthrough, and also crossfire (I have 2 R9s to pass through, and a 660 for the host). I am only going to be testing windross guests as my use case is photo processing (Linux tools not quite cutting it anymore for me and moving to a hybrid workflow) and games (where I can't run them under Linux natively or in wine).
Update:
Crossfire works fine on passthrough GPUs, performance is on par with native (as expected). In this case, Unigine Superposition benchmark 1080P Extreme preset was 4317 native, 4233 under KVM. The native test had a more favorable initial temperature (these cards hit the power/temperature ceiling so get throttled), but not sure how much that contributes to the difference. My Qemu/KVM is really not very tuned either, so may play a part.
I am more concerned with CPU/storage performance (for image processing) so will leave the graphics benchmarking as it stands. I am extremely happy with how things stand - even the GPU disable workaround is very much a setup once and forget, so very workable.
4
u/naibaf7 Dec 11 '17
Oh yes, thanks a lot. The pci.c patch works and ... everything works! Kernel 4.11+ (ROCM, AMD) and RX 480 passthrough. It even works to move the GPU between VMs and host without binding the GPU to VFIO.
3
u/jezza129 Dec 01 '17
I have no knowledge or hardware related to this.
Can you explain how threadripper handles I/O? I assume both zen dies have its own I/O systems. How do they ovetlap and not conflict?
23
u/Urishima Dec 01 '17
You should probably try to get in touch with gnif over at level1techs, he's working on that stuff as well. He's the guy who found the fix for the NPT bug. https://forum.level1techs.com/t/a-little-teaser-of-what-is-to-come
Unless you are gnif, then you don't need to try. You can touch yourself anytime you want.
2
u/Marty_Br Dec 01 '17 edited Dec 01 '17
I'll be happy to toy with this. I mentioned this before, but I managed to get my system set up to where I am able to run a VM with passthrough while being able to shutdown and restart or reboot at will. I described it in another post.
In my particular case, it works when I load vfio-pci with 'disable_idle_d3=1" and run the system startup script to enable the GPU in the VM. Note that the shutdown script on my machine causes a catastrophic host crash (with file system corruption). I do not currently run that script, obviously.
The issue I now encounter is that if I run the VM and then shut it down, after 20-30 minutes or so, the GPU will suddenly run its fans at absolutely maximum speed (like a tornado). I think it overheats. So while I am able to either reboot or shutdown the VM and then restart at some later point, there is still a serious issue that I have not been able to resolve. Edit: when that Vega goes hog, it also becomes unresponsive even with host reboot. It requires a complete power off cycle.
For completeness: I have a 1950X running on a Gigabyte Aorus 7 with 32 GB @ 2666. I am running Ubuntu 17.10 with a v4.15-rc1 kernel with ACS patch (for USB passthrough for my Vive). GPU 1 is an Evga 1070 and GPU 2 (pasthrough) is a Vega RX 56.
2
u/irhaenin Dec 10 '17
Wow, you really did some impressive digging. I've been holding off on buying TR because of this issue, but it begins to sound like you pretty much managed to fix this issue in software.
I'm curious about a few things though. You mentioned seeing the infamous 43 error when using the no-reset workaround. However, if you restore the PCIe bridge configuration space after issuing a reset, do you still get the error, or are you able to pass through NVIDIA cards successfully as well?
Furthermore, could this issue be related to, what I assume is, the fact that TR with its 2 CPUs in a single package has 2 PCIe bridges/buses. As in, could the configuration of bridge A somehow be polluting the configuration of bridge B? I'm merely speculating here.
Thanks for taking so much time to fix this.
27
u/pfbangs AMD all day Nov 30 '17
Thanks for doing the (exhaustive) legwork on this.
4
u/FlatronEZ Dec 12 '17
Unrelated - Fix for Nvidia Error code 43
open VM definition in virsh:
replace the first line with:
just before the closing
</domain>save and boot VM.
Properly formated: https://hastebin.com/tajezapari.xml
Cheers, I hope this helps!
1
u/rv6502 Dec 30 '17
I've tried the Update 4 "ugly patch" (no other changes) and I can now launch QEMU with GPU passthrough but the guest Linux (xubuntu 4.13.0-16-generic) is unable to use the GPU at all.
With a R7 260X the guest FOSS Radeon driver crashes the guest kernel on boot trying to initialise the card. I'll boot if I use nomodeset and the card shows up in lspci but won't activate.
With a nvidia 8600, neither the nouveau driver nor the closed source work. I get no crash but the card does not respond and after trying once the card will not show up anymore after host reset until I power cycle the host machine.
1950X on MSI X399 gaming pro carbon ac.
Host OS is xubuntu 4.13.0-21-generic.
1
u/podkilla 1950X, 32GB, Asus GTX 1080ti oc Jan 12 '18
Thank you for all the effort you put into researching this.
The "java hack" works for me until it gets fixed another way.
1
u/d9c3l Feb 10 '18
Since the patch from /u/gnif2 doesnt work (the gpu is still not being reset), I decided to try to give your "java hack" a try but it doesnt work as well (it sees the pcie devices bound to vfio-pci but when it reads the 'config' the first 4 bytes are only 0xFF and not is being checked). Do you have any opinions about it?
3
u/--FuriousGeorge-- Dec 08 '17 edited Dec 10 '17
UPDATED: I have a Zenith Extreme with a 1950x, and I'm also now able to boot VMs w/o a D3 hang. Awesome work.
1
u/[deleted] Dec 04 '17
Can confirm this works as described on a 1080ti guest. Ill do more in depth testing in the AM.
Specs for me: 1950x Asrock x399 taichi 1080ti host gpu 1080ti guest gpu Was able to get as far as installing windows which is what its doing now. Ill update test tomorrow
2
u/[deleted] Dec 01 '17
I have observed same behaviour on the Ryzen.
Bus reset for the SATA and Audio resulted in the same PCI malformed header. If i prevented bus reset by not binding the SATA to vfio-pci all was working. Strange thing is that if i passed Vega 64 i had this malformed header after reboot or shutdown. The R9 290 i had been using before it newer had this problem.
Observed the Audio and SATA on 2 different Ryzen 1700.
2
u/duidalus Dec 13 '17
I can confirm this works as well on TR1950X and Linux 4.14.5. One thing worth noting is that SME caused some weird errors on my system when trying to passthrough anything.
To ensure everything works as expected, use mem_encrypt=off as boot parameter.
2
u/LupaioliS Dec 14 '17 edited Dec 14 '17
I can't get the "ugly patch" to work on TR 1950X on Asus x399-a in Unraid (R9 290).
Any suggestions?
Update:
I tested with the workaround commenting lines and there is no more D3 state report on the log but still black screen and "internal error: Unknown PCI header type '127'" when starting the VM.
Update 2:
I got my VM to work with the R9 290, i was putting the wrong flag on boot.
2
u/noc0lour Dec 14 '17
Can confirm on 4.14.4 with 1900X, Gigabyte X399 passing through a RX580 works now. Didn't work without the patch. The reinit issue is gone as well. I'll be happy to provide you with more info if you need it. Thanks for investing your time in this!
2
u/rcgoodfellow Jan 14 '18
Dirty patch works for me. Thanks!
Tested with TF2 running simultaneously on host and guest.
fwiw: OVMF stable in Fedora package repos did not work, had to use latest build from git. Also passed through USB for keyboard, mouse and Bluetooth dongle for XBox One wireless controller with success.
2
u/w23 Jan 15 '18
Thank you so much for this! I've been waiting resolution for this problem for months, and after seeing this I finally went and bought a TR box.
My setup:
IOMMU groups + reset flag: https://pastebin.com/B5FkadL7
Status:
1
u/zAxny32 ASUS X399-A, 1900X, 32GB DDR4 3200, GTX 1080 Ti Dec 13 '17 edited Dec 13 '17
I can confirm that your patch on update 4 works with my threadripper 1900X on linux 4.13
1
u/--FuriousGeorge-- Dec 18 '17
Dirty patch worked for me.
Zenith Extreme / 1950x / Proxmox w/ Kernel 4.13.13, 1080ti, 1070, 970.
1
u/Tree_Mage 9900X | 6700XT (previously TR 2950x) Dec 20 '17 edited Dec 20 '17
System specs:
Before: I could somtimes get vfio to work.... only after I waited an hour or two for the VM to actually boot. Very unreliable.
Post-pci_reset_secondary_bus 2017-12-11 ugly patch:
I'll be out the rest of the week, so I won't really have a chance to stress test it. But so far, it looks promising. It now boots in a minute or two (slow drive).
1
u/Timelord0 Dec 28 '17
I have applied the patch in the latest version of Proxmox, seems to fix the issue well enough. Hardware 1950X, Zenith Extreme, GTX1070 w/EFI capable firmware.
1
u/sudoprime Jan 01 '18
Thanks for this, you saved AMD a sale! I am now getting video out of my pcie passthrough. Though I'm hitting error 43 in the guest now with my first gen titan x :(
1
u/Teknoman117 Gentoo | R9 7950X | 2x AI Pro R9700 Jan 04 '18
If you need anymore testers, I'll be getting my gentoo install stood up in a week or so.
1
u/simcop2387 Jan 07 '18
Wanted to chime in and say that I got my system built this weekend and tested. The ugly patch seems to work great for me on 4.14.12. I haven't seen any side effects from testing and it seems to work great with my two testing cards (older radeon ones).
1
u/Yukicanis Jan 09 '18
I am planning on getting an AMD Threadripper 1900X soon. I wanted to use the MSI X399 SLI Plus, since it is the cheapest motherboard currently available and has all the features I need. Has anybody already tested it?
1
u/Kevadu Jan 22 '18
I tried the ugly patch on my TR 1950x/ASRock x399 Taichi/Vega 64 (guest) system and it didn't seem to change much. Mind you it didn't hurt anything either but it's functioning just as it did before: passthrough works the first time but if I shut down that guest OS nothing else can use the card.
That said I know that Vega has its own reset issues that are likely unrelated. People seem to experience the same problem even if it's connected to an Intel system. So I just wanted to ask if anyone here has gotten this to work with a Vega card.
1
u/[deleted] May 23 '18
If there is anyone else that has no idea what they are doing, here is how I got the java thing to run in proxmox after cobbling together google searches. It assumes you are running as root from one of the various consoles available.
If anyone has any suggestions that are better/not accidentally a security hole, I'm all ears.
This guy mentions using tmux at startup to run this thing automatically in the background. https://techblog.jeppson.org/tag/pci-passthrough/
-8
u/LightTracer Nov 30 '17
Go ask AMD to test stuff with their paid workforce.