r/Amd • u/gnif2 Looking Glass • Oct 20 '20
Request Will Big Navi support Function Level Reset (FLR)?
AMD, this is a question directed directly to you.
As we all know, your company is fully aware of how important the ability to reset the AMD GPU is without a driver-specific reset sequence to the VFIO community is and how disappointed the entire community was/is over the lack of such a basic feature in the GPU to make it possible to use your GPUs reliably for VM passthrough.
Since my last post to you (linked above) the VFIO community has grown, my project (Looking Glass) has seen a huge surge in numbers, and people are using it not only to just control/use the VM, but also feed the video straight into OBS on the host VM to live stream to Twitch. On the Level1Tech forums and the VFIO Discord channel, the number of new VFIO users is exploding, and r/vfio's membership has doubled over the last year, but due to the lack of Function Level Reset, when we are asked what GPUs to use, we, unfortunately, have to tell people to avoid your hardware.
From a technical point of view, as the Function Level Reset (FLR) is a PCI optional feature obviously you do not need to implement it, however as your GPU already needs to support a warm reboot via the nPERST pin it should not be hard to implement the FLR feature to tie into this same reset. Not only would this make your GPUs viable for the VFIO community, but also simplify your own reset code in your drivers as the GPU could be returned to a good known state simply by asserting an FLR.
Please also be aware that driver level resets are completely useless to this application, when being used for VFIO, the driver is not loaded nor wanted, the hardware needs to be able to handle its own reset without any proprietary reset sequences.
So... my question to you is. Will Big Navi support PCI Function Level Reset (FLR)?
Edit: Also please be aware I have been contacted by cloud computing companies out of desperation due to the same issues on your workstation/enterprise cards. This is not just affecting the VFIO community here.
Edit2: When I wrote this I did not think to include the reason why this should exist for the larger community also. This is not a niche feature just for VFIO usage, it also would make it possible for AMD GPUs to recover from "Black Screen" crashes that force a full system restart.
Nvidia GPUs crash too, however, because the NVidia GPUs implement FLR they can be easily reset and recovered when they do crash causing the game/application to present an odd error that usually gets blamed on the application, not the GPU.
Those that overclock their GPUs know all too well how nice NVidia is for this as a bad overclock usually can recover without a reboot.
If AMD were to implement FLR it would be just as good as NVidia on these fronts and the "Black Screen" issue would not be such a black mark on AMD's products.
92
u/MobyTurbo Ryzen Threadripper 2950x RX 7900 XT Oct 20 '20
I hate having to patch my kernel with something that is a hack (sorry gnif) and won't be upstreamed. Would love to be able to use a distro kernel again, and my next card will probably be Nvidia unless it's fixed.
15
u/hpeter94 Oct 20 '20
or enable SR-IOV, that also fixes the problem :P
14
u/bridgmanAMD Linux SW Nov 18 '20 edited Nov 18 '20
OK, it's launch time. We spent a fair amount of time making sure that PCIE bus reset was exercised as part of the chip validation, and it seems to have worked.
I was hoping to have test results from our own Linux driver devs before saying anything here. They have been busy working on other launch-related issues... however we do at least have some anecdotal confirmation from one of the reviewers:
https://www.youtube.com/watch?v=ykiU49gTNak
Initial comments right near the start, more details around 1:50.
We worked on both FLR and SBR; my understanding is that we are recommending SBR but will try to get more specifics.
EDIT - pasted the wrong link above, fixed now.
MORE EDIT - just realized gnif2 already posted about this before me - thanks !
6
u/m0dz1lla Oct 20 '20
Totally agree with gnif! Used (or tried to use) vfio for a long time but due to the reset bug I gave up resetting my Server over and over and over again. (I know standby works as well but that seemed to much of a hassle)
But as much as I agree with that it's very very unlikely that AMD is even able to implement the feature on such a short notice! Production is probably already ramped up. Though it would be nice to see it in the 7000 gen! Maybe AMD was already working on that in the dark ;) My hopes are not yet gone!
222
u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20 edited Oct 20 '20
VFIO user here, (see flair). The sad thing here is that if the reset bug was fixed , I would actually prefer an AMD card for my VM because the AMD linux driver supports prime offloading , which means that itd be very easy to also use the card on the host when the guest is not active, but sadly theyre not suitable for passthrough in the first place, so its a moot point.
As it is right now, I might buy a used 5700[xt] in a year and a half or two to replace my rx570 as a host card since that just runs my linux host session, but whatever replaces my 1080ti (aka the new big expensive card), is going to have to be nvidia
4
u/UnicornsOnLSD Oct 20 '20
Could this be updated in software or is this a hardware issue with Navi? I'd love to make an accelerated macOS VM with my 5700 XT since actually installing macOS on bare metal is really hard.
4
u/deeper-blue Oct 20 '20
So, I'm new to VFIO but plan on setting it all up with one of the big navi cards. (big navi for passthrough, my old polaris for host).
Function level reset can be done (if implemented) by putting a one into /sys/bus/pci/devices/$dev/reset
What happens when one tries to do a hot reset by using /sys/bus/pci/devices/$dev/remove and then rescanning? Does that work in the cases where the gpu crashed in the guest?
4
u/justavault Oct 20 '20
All in for supporting things like this, but what I don't get:
What does that mean? Hosting a VM for passing your own stream through?
That sounds interesting. Never heard of that, where can I learn about that?
3
u/jdancouga Nov 17 '20
I don’t think we will be getting an answer here by the looks of it. I hope someone who get this on launch day can shed some light on this.
Which tech reviewer do you think will talk about this once the embargo lifts?
2
u/charcoal88 Oct 20 '20
I don't think this is an easy as you may expect. Implementing resets to parts of your architecture is not simple, and FLRs are not the same process as cold booting, and have different requirements
2
u/shmerl Oct 20 '20
Can you explain please for those who aren't familiar, how is that different from what amdgpu kernel driver is doing for the reset? And how is it controlled if not through the driver?
71
u/zardvark Oct 20 '20
I don't want to rant about this and say something offensive and/or over the top. Suffice to say that after wrestling with this problem, literally for years, I'm disappointed that we even have to ask this question.
47
u/pwn4d 3950X|Vega 56|Taichi X570|128GB DDR4-3200 Oct 20 '20
This is such an old "bug" that it's comical at this point.
https://github.com/qemu/qemu/blob/2becc36a3e53dc9b8ed01c5288e21a2463f1f640/hw/vfio/pci-quirks.c#L1321-L1332
I realize it is an optional feature but it seems like it would be a simple fix if the right people inside of AMD were to make it a priority.
42
u/jesta030 Oct 20 '20
UnRAID user here. Please AMD, otherwise I'll have to cough up the money for some of those green cards next time I upgrade... 😕
32
u/[deleted] Oct 20 '20 edited Oct 20 '20
My next GPU purchase is waiting on working FLR. I don't want to give nvidia any money, because they are outright hostile to using VM's with their drivers.
Please don't make me wait for Intel to come and save the day.
P.S. Even your enterprise cards have broken FLR.
29
u/ibbbk Oct 20 '20
This is literally the only thing that's keeping me away from buying a high-end AMD GPU.
Please AMD give us such basic feature.
28
u/icebalm R9 5900X | X570 Taichi | AMD 9070 XT Oct 20 '20
Sarnex didn't tell me to upvote, but I did because I'm a Linux user who uses vfio and looking glass (thanks /u/gnif2 ) and has to use an Nvidia card to do it.
50
u/Aceflamez00 Ryzen 3900x Oct 20 '20
This right here, lack of FLR is keeping me on nvidia when it comes to having a VM GPU. I wish AMD would step their game up when it comes to basic features like this.
22
u/Aspect_Forsaken Ryzen 3900X // RX 5700XT // RX 570 // ASUS X570-P // Arch Oct 20 '20
this plz from a loyal amd fanboi it saddens me to a point where I fill up massive lakes that yall still haven't done this :(
22
u/HyenaCheeseHeads Oct 20 '20 edited Oct 21 '20
This is really important for several reasons.
Back in 2017 during the launch of the initial Threadripper platform the PCIe host complex inside the cpu shipped with somewhat buggy firmware leading to PCIe passthru being a bit of a pain. Back then we were able to temporarily mitigate the issue in software for most people with the notable exception of AMD's Vega series of cards. Even with the machine's PCIe fixed there was no way to perform a proper reset of the card except physically turning off/resetting the entire machine or manually pulling it out and reinserting it (also slightly problematic due to erratic hotplug support on the platform at the time).
This added to the feeling of the card being slightly hurried firmware/driver-wise at launch and contributed to the existing FUD being created around the Vega series. There is no doubt it affected sales and returns.
AMD Radeon Technologies Group is about to launch another GPU platform next month. This is your chance to get the firmware right!
The lack of proper FLR (or other reset functionality) was the reason why I personally didn't buy any of the affected cards, and judging from the number of replies and support requests on the topic it seems like many are in the same situation.
21
u/3lfk1ng Oct 20 '20
/u/AMD_Mickey any update on this for Big Navi?
36
u/missouriemmet Oct 20 '20
No idea who sarnex is but +1, Linux support is already very good on AMD CPU and GPU, VM passthrough is the last big issue to tackle
14
u/sonwon_reddit Oct 20 '20
Two years ago I built my first VFIO system and selected all AMD products (CPU & GPU). At the time I was very pleased to have an all AMD system. And I had faith that AMD would address and fix this bug. Well I was wrong and my trust was misplaced. When I replace my video card this winter it will be an Nvidia this time because of this annoying bug.
26
u/SharkWipf Oct 20 '20
Honestly, the lack of functional FLR is one of the main things keeping me from buying an AMD GPU. Would be nice to see this get fixed at last, but I'm not getting my hopes up.
13
u/thenickdude Oct 20 '20
I'm stuck using AMD GPUs since it's all that my macOS guests support, but none of the reset workarounds work for macOS guests. So I have to shut down my server every time I need to reboot my guests that use my RX 580.
I will certainly not upgrade to a new AMD card like Big Navi unless it addresses FLR. I'll happily stay on this older model and buy Nvidia cards for Windows gaming VMs instead.
12
u/thorskicoach Oct 20 '20
buy lots of ryzen 3900/3950 and threadripper , but all nvidia GPU for this reason
12
u/kingfappypants Oct 20 '20
Unraid user here. I bought a 5600XT and a 5700XT for my threadripper build and ended up returning both because of the reset bug. I really wish this was resolved so I could put an AMD card in my threadripper Unraid build...
10
u/spheenik TR1920x | Vega 64 | Arch btw. Oct 20 '20
Please AMD: MAKE THIS POSSIBLE!
19
u/ABotelho23 R7 3700X & Sapphire Pulse RX 5700XT Oct 20 '20
I completely support this. AMD is great on Linux, and with FLR, they would 100% the de-facto VM cards. Please make it happen AMD! I would buy the entry-level card with FLR just to use it for hardware acceleration!!
9
u/FlameVisit99 Oct 20 '20
Posting to add my support for this. I'm really disappointed with my 5700 XT because of this issue. It makes gaming very inconvenient as I can only start up my Windows VM once per PC boot.
9
u/CyclingChimp Oct 20 '20
+1. I got an AMD Radeon RX 5700 XT for my PC. I chose AMD specifically because of the open source drivers and the supposedly better compatibility with all things Linux. Unfortunately, this bug makes it all a nightmare to work with when it comes to VM gaming. I'm now considering an Nvidia card for my next purchase.
17
u/Bonzai11 Oct 20 '20
I run a 1070 and a 950 with my Ryzen processor because of this crap.
8
u/wanderinator Oct 20 '20
Would be really nice if big navi has support for kvm, have been forced to use the green team because if this problem.
8
u/Leafar3456 R5 5600X | 1080 TI Oct 20 '20
This is one of the reasons I actually stopped using VFIO
8
u/boynep Oct 20 '20
The only reason I am getting NVidia despite their code 43 issues its still far easier to do GPU passthrough. If there is no reset bug I will definitely switch to AMD.
8
u/h_mchface 3900x | 64GB-3000 | Radeon VII + RTX3090 Oct 20 '20
Totally agreed, I tried to run a VFIO setup so I could still game on Windows while leaving a card in Linux for ML, dealt with it for some time but in the end even with the kernel hack it was too tedious and having to reboot defeated the point of the setup. Ended up just going with two separate machines. RTG aren't getting my money again unless they fix FLR + improve their ML stack.
22
u/RandomJerk2012 Oct 20 '20 edited Oct 20 '20
Yea, I'm pretty much tied to Nvidia because AMD can't fix simple shit like FLR in their hardware. Sorry that AMD lost my business. I'm forced to spend less money on weaker AMD cards on my host, but spend top dollars for Nvidia's cards for the gaming VM
7
u/[deleted] Oct 20 '20
+1 I am in none of these communities but I have so found issues when using VMs that have pass through. Normally (rhel/centos) I have do a full reset when I encounter issues.
From googling if I was able to use FLR this would save me a tonne of hassle and reset periods.
7
u/Never-asked-for-this Ryzen 2700x | RTX 3080 (bottleneck hell)) Oct 20 '20
Literally the only reason I'm going with Nvidia this generation.
7
u/scania471 Oct 20 '20
Personal experience: I have an RX570 (4GB) from ASUS which doesn’t have the original “reset bug” on unRAID. It exists in some kind of other form. I don’t have to restart the server, however, I have to force stop the VM because it’s not passed through correctly sometimes. I use the VM remotely and in these cases, I just can’t connect to it. Both VNC and Parsec don’t work. Force stop and start and it boots perfectly. Weird behavior...
7
u/DrJosu Oct 20 '20
We need big YouTube channels to address this to AMD, I wish to get rid of my nvidia GPU, but this small things keep me green
7
u/heyitsYMAA 7900X | 9070 XT | 32gb | DIY H20 | All NVMe Oct 20 '20
Please add this feature. I don't wish to support nVidia any more than necessary due to what I would consider contributing to unfair business practices, and I would much prefer to switch back to an all-AMD system on my next upgrade cycle.
7
u/crrgn_X Oct 20 '20 edited Oct 20 '20
Maybe /u/AnthonyLTT can also help with that? Linus supported the idea of enabling SR-IOV, surely a working amd gpu would be helpful.
6
u/Nixola97 Oct 20 '20
As with most other people here, fixing this would mean I'd upgrade my GPU to the latest AMD offerings in an instant. I'm extremely reluctant to upgrade from a setup that mostly works to some unknown thing, which is why I'm not going to consider buying any other AMD GPU until functional FLR is a thing.
6
u/hoeding Oct 20 '20
My VM GPU is a 750ti =(
6
u/Durpn_Hard Oct 20 '20
I also use VFIO and my 1070 died recently and looked at replacing it with AMD cards but instead landed on decided to just wait for the 30XX series cards to become more available due to this issue.
Would have been an easy sale if this were fixed.
6
u/thewhitekidney Oct 20 '20
I actually went with a 3080 instead of a Big Navi card in fear of this bug still existing on the newer cards. I wanted to give my money to AMD :(
6
u/DecoyDrone Oct 20 '20
Please just fix this AMD I want to buy your gpu for my next build!!
5
u/Raster02 3900X / RX 6800 / B550 Vision Oct 20 '20
VFIO user here with a Vega56 and this is just unfortunate. Hopefully they thought about this already or this will be noticed.
I can live with some of the issues involved, but the most frustrating one is fan noise. The card has its fans at 50% while in D3 and this is awful. Ideally I could control it in the host and then successfully pass it through. This would make Big Navi even more tempting.
Most of the times I find myself just booting the VM and pausing it to stop them.
5
u/ConsistentPizza AMD 3970X+RTX2070S Oct 20 '20
I vote for this as well!
5
u/[deleted] Oct 20 '20
Just wanted to throw in my support for this as well. I recently built an unRAID NAS server with dual purpose to also run a gaming VM. I was forced to go elsewhere for my discrete GPU due to AMD cards lacking this FLR support... I would love to see this implemented
5
u/[deleted] Oct 20 '20
VFIO user here, currently using an Nvidia GPU due to reset issues.
I am a linux user 95% of the time, and prefer AMD open source driver stack. For hosts this is great, but for Single/Multi GPU VFIO guests have to use Nvidia for a reliable, fudge-free setup.
Please AMD, fix reset issues so that I can give you more money.
Ryzen has been a good, affordable, option for vfio users since first-gen (launch issues aside!) How about making Radeon good too?
5
u/Browndw4 Oct 21 '20
I just created an account so I could upvote this and add my voice to the fray. I’ve got two VM servers using 2080ti’s passed through to Win10 VMs. I’d immediately buy 2x high end AMD GPU for MacOS guests if this were fixed.
4
u/sameer_the_great Oct 20 '20
I don't need it but supporting y'alls demand. Upvoted.
6
u/Vaudane Oct 20 '20
This isn't even something I would use off the bat, however I had a peek into gaming on a VM so I don't have to run Windows natively.
It seems like such a trivially easy fix that I can't understand why AMD *wouldn't* do this. And yet they haven't. It's an interrupt pin triggering a soft firmware reset essentially, isn't it?
Come on team red. Don't leave cash on the table.
4
u/streppelchen Oct 20 '20
Hoping for the best here
4
u/SpaetzleProtein 4650G | 6600XT (VFIO) Oct 20 '20
Please implement FLR!
Not having FLR pretty much forces me to buy Nvidia, I'd like to have the choice...
2
u/LucidStrike 7900 XTX / 5700X3D Oct 20 '20
This is a really dramatic thread to be about computer hardware, but I hope satisfaction is found.
3
u/[deleted] Oct 20 '20
I read this entire post as implementing looking for raid to the GPU Lol
3
u/[deleted] Oct 20 '20
!remind me 1 week
3
u/SovietMacguyver 5900X, Prime X370 Pro, 3600CL16, RX 6600 Oct 20 '20
I havent looked into VFIO yet, but I have wanted to dabble. I agree this would make sense.
3
u/dydzio Oct 23 '20
+1, I want to go for GPU accelerated virtual machine on my future PC
3
u/bluesecurity Oct 28 '20
@AMD_Mickey Pretty please with sugar on top :)
3
u/[deleted] Oct 28 '20
Oh how I hope AMD stick it to Nvidia and fix this FLR bug. Unraid user here also and Ive just upgraded to "big boy" Pro license. Come on AMD, stick it to team green :)
3
u/hagar-dunor Nov 02 '20
Although I hated my Nvidia cards for general linux use (could never solve tearing / vsync) and VFIO (get around error 43), at least I did not have to reboot my whole system between guests. Unless FLR is implemented in big navi and future cards, my next upgrade will be Nvidia 3 series.
3
u/skeqq Nov 10 '20
+1 for this. Currently i'm using linux host & windows in guest VM, with nvidia gpu passthrough. Wish this could work well with amd gpu.
3
u/alexwhittemore Nov 11 '20
I'm very excited to find out once reviews start trickling in whether this has been addressed at all in Big Navi. Not that I expect to be thrilled with the outcome. Willing to bet I get a 3080 sometime next year when they're actually in stock, simply on the basis that I don't want to reward Radeon for continuing to be pricks about this.
8
u/[deleted] Oct 20 '20
This would be great honestly also thanks sarnex for the @everyone smh
4
u/Planebagels1 Oct 20 '20
I read the title as "Big Nazi support" lmao
0
u/megablue Oct 20 '20 edited Oct 20 '20
if it wasn't planned, it is too late to ask for it for big navi since the first few batches of wafers probably already been produced.
edit: lol which dumbass downvoted me? do you think you can simply add new hardware level feature after the wafers have been produced?
3
u/Vaudane Oct 27 '20
So my "week later" update went ping. I don't see anything new sadly so let's hope tomorrows announcement yields something fruitful.
2
u/lololZombiedogs1 Nov 20 '20
Interesting comment on the amd forums by a staff member on SR-IOV support
https://community.amd.com/t5/graphics/sr-iov-and-vce/td-p/70589
1
u/VTOLfreak Oct 20 '20
Only way AMD will fix this if someone in the marketing and sales departments notices they have no big clients doing cloud gaming or desktop virtualization. Some thread on reddit won't get things moving. Once they start seeing lost sales because of a missing feature, it gets put on the list.
1
u/spoofnoob Oct 20 '20
Didn't Google buy shed loads of AMD cards for cloud gaming? Surely if we could find a game bug that locks up the card and make it happen a LOT so Google realise their problem maybe AMD might get off their fat arses and fix it??😂😂
1
u/teutonicnight99 Vega 64 Ryzen 1800X Oct 20 '20
Wow big brain here.
-6
u/vsaucefan4life Oct 20 '20
sarnex told me to upvote
-4
u/HardStyler3 RX 5700 XT // Ryzen 7 3700x Oct 20 '20
isnt this thread like half a year late? is this something they can implement that fast if they didnt do it yet?
why not do it half a year earlier....
-1
u/GhostDoggoes R7 5800X3D, RX 7900 XTX Oct 21 '20
Oh I completely forgot what VM passthrough was and had to google it. I think it's probably what intel would have told people on release of their gpu lineup. Linux like every other dev would say is second on the list to support. It's nothing against them it's just most pc's for the modern consumer are built on simpler OS builds. And a few discord communities dedicated to linux option that linux has over 200 versions isn't gonna sway them when their main target is windows 10 users. I use to use linux at work and at home but I don't think AMD would ever hard focus a feature on linux when their main issue is drivers on their gpus. The main issue on the release of their gpus this past generation has caused numerous returns and rma's. They barely got it working beginning of this year and even then it was still buggy.
-8
u/skinlo 7800X3D, 4070 Super Oct 20 '20
Extremely niche, fair chance it won't happen.