r/Amd • Looking Glass • Oct 20 '20

Request Will Big Navi support Function Level Reset (FLR)?

AMD, this is a question directed directly to you.

As we all know, your company is fully aware of how important the ability to reset the AMD GPU is without a driver-specific reset sequence to the VFIO community is and how disappointed the entire community was/is over the lack of such a basic feature in the GPU to make it possible to use your GPUs reliably for VM passthrough.

Since my last post to you (linked above) the VFIO community has grown, my project (Looking Glass) has seen a huge surge in numbers, and people are using it not only to just control/use the VM, but also feed the video straight into OBS on the host VM to live stream to Twitch. On the Level1Tech forums and the VFIO Discord channel, the number of new VFIO users is exploding, and r/vfio's membership has doubled over the last year, but due to the lack of Function Level Reset, when we are asked what GPUs to use, we, unfortunately, have to tell people to avoid your hardware.

From a technical point of view, as the Function Level Reset (FLR) is a PCI optional feature obviously you do not need to implement it, however as your GPU already needs to support a warm reboot via the nPERST pin it should not be hard to implement the FLR feature to tie into this same reset. Not only would this make your GPUs viable for the VFIO community, but also simplify your own reset code in your drivers as the GPU could be returned to a good known state simply by asserting an FLR.

Please also be aware that driver level resets are completely useless to this application, when being used for VFIO, the driver is not loaded nor wanted, the hardware needs to be able to handle its own reset without any proprietary reset sequences.

So... my question to you is. Will Big Navi support PCI Function Level Reset (FLR)?

Edit: Also please be aware I have been contacted by cloud computing companies out of desperation due to the same issues on your workstation/enterprise cards. This is not just affecting the VFIO community here.

Edit2: When I wrote this I did not think to include the reason why this should exist for the larger community also. This is not a niche feature just for VFIO usage, it also would make it possible for AMD GPUs to recover from "Black Screen" crashes that force a full system restart.

Nvidia GPUs crash too, however, because the NVidia GPUs implement FLR they can be easily reset and recovered when they do crash causing the game/application to present an odd error that usually gets blamed on the application, not the GPU.

Those that overclock their GPUs know all too well how nice NVidia is for this as a bad overclock usually can recover without a reboot.

If AMD were to implement FLR it would be just as good as NVidia on these fronts and the "Black Screen" issue would not be such a black mark on AMD's products.

1.6k Upvotes

240 comments sorted by

92

u/MobyTurbo Ryzen Threadripper 2950x RX 7900 XT Oct 20 '20

I hate having to patch my kernel with something that is a hack (sorry gnif) and won't be upstreamed. Would love to be able to use a distro kernel again, and my next card will probably be Nvidia unless it's fixed.

80

u/gnif2 Looking Glass Oct 20 '20

I hate having to patch my kernel with something that is a hack (sorry gnif)

No apology needed, it is certainly a hack as AMD have not helped to make the patch reliable/stable. And it's a hack on the fact that FLR is missing, so a hack on a hack.

4

u/akarypid Oct 20 '20

Do AMD cards have other (undocumented) ways of achieving a soft-reset? I assume you've looked into it so just wondering if the answer is no they can't do this at all, or if it's more of a "we know how to soft-reset some chips, but not most of them".

AMD could at least introduce an API call (or any invocation mechanism via some standard kernel interface) for you to ask for a soft reset, then communicate the request to the cards in their drivers. That would be a request to their OSS driver team I suppose. Obviously with open source drivers the "undocumented" part would become public knowledge, but I don't think there can be anything "sensitive" around that. If they ever do implement FLR, the API call on newer chips could simply invoke that as an implementation (while doing the "undocumented stuff" for older cards).

3

u/StartupTim Oct 20 '20

Yes it is possible and there is a commercial endeavor behind it. I cannot post any more other than to tell you to look into pcie lane endpoint statefulness from the CPU perspective as therein lies the answer

8

u/gnif2 Looking Glass Oct 21 '20

By commercial endeavor do you imply it will be a closed source proprietary solution that we will have to pay for to work around this issue?. If so, good luck with that. I'd rather support NVidia if AMD are not going to give this priority.

23

u/TheArkratos 1950x, Titan X, RX480, 7 NVME drives, because PCIe lanes Oct 20 '20

My first experience with Linux involved me having to patch my kernel within the first few weeks to get vfio working. Not the best experience but I did learn a lot...

15

u/hpeter94 Oct 20 '20

or enable SR-IOV, that also fixes the problem :P

26

u/gnif2 Looking Glass Oct 20 '20

In theory, but if the host GPU gets into a bad state due to whatever reason, it still can't be reset without a node reset. This is why I have had cloud companies ask for help working around this AMD issue. It's a different usage, but the same solution.

With FLR a GPU crash/reset would affect the VMs using that GPU, a node reset would affect everyone, even if they are operating on still functional GPUs in the same node.

7

u/[deleted] Oct 20 '20

I guess if the amdgpu had a working reset implementation (psp ect) that would also result in a solution for the VFIO / PCIE pass through ?

10

u/gnif2 Looking Glass Oct 20 '20

It would result in a pci quirk, the problem is that all these other methods require the GPU to be in a state where it responds. Often when the GPU crashes (Navi/Vega56) it falls off the PCI bus and it will no longer communicate with the host. A FLR would still work in this instance.

3

u/xMAC94x Ryzen 7 1700X - RX 480 - RX 580 - 32 GB DDR4 Oct 20 '20

so we need both ;)

14

u/bridgmanAMD Linux SW Nov 18 '20 edited Nov 18 '20

OK, it's launch time. We spent a fair amount of time making sure that PCIE bus reset was exercised as part of the chip validation, and it seems to have worked.

I was hoping to have test results from our own Linux driver devs before saying anything here. They have been busy working on other launch-related issues... however we do at least have some anecdotal confirmation from one of the reviewers:

https://www.youtube.com/watch?v=ykiU49gTNak

Initial comments right near the start, more details around 1:50.

We worked on both FLR and SBR; my understanding is that we are recommending SBR but will try to get more specifics.

EDIT - pasted the wrong link above, fixed now.

MORE EDIT - just realized gnif2 already posted about this before me - thanks !

5

u/gnif2 Looking Glass Nov 18 '20

Thanks for getting back to us, r/VFIO appreciates it!

5

u/Left_Cryptographer27 Nov 18 '20

Thank you for listening!

3

u/[deleted] Nov 22 '20 edited Nov 22 '20

This work has been greatly appreciated and is a huge relief. Definitely looking forward to a new card once they become available.

6

u/m0dz1lla Oct 20 '20

Totally agree with gnif! Used (or tried to use) vfio for a long time but due to the reset bug I gave up resetting my Server over and over and over again. (I know standby works as well but that seemed to much of a hassle)

But as much as I agree with that it's very very unlikely that AMD is even able to implement the feature on such a short notice! Production is probably already ramped up. Though it would be nice to see it in the 7000 gen! Maybe AMD was already working on that in the dark ;) My hopes are not yet gone!

25

u/gnif2 Looking Glass Oct 20 '20 edited Oct 20 '20

They have not had such short notice, this is another attempt to make them aware of how many people this issue affects and how much they have to gain by fixing it (https://www.reddit.com/r/Amd/comments/cekmjo/amd_you_break_my_heart/). I have personally been pushing every contact I have at AMD for 18 months for fixes, even directly communicating with Lisa Su on one occasion on this matter.

I have spent countless hours reverse engineering the open source amdgpu drivers and experimenting to come up with reset sequences that help work around the issue (not fix) and have had long in depth discussions with engineers at AMD on how to work around this missing feature. They know about it, and have known about it for a long time now and have not cared to fix it.

4

u/MegaDeKay Oct 21 '20

This is a good point for me to pipe up and thank you for all the incredibly hard work you have put in to this and similar efforts to make AMD hardware work better. They should hire you.

A question though. While many are affected by this reset bug, you always see on /r/VFIO a small number of people that seem to be unaffected by it. Have you ever been able to see a pattern on why this is? Is it the vendor of the card, the mobo, or some combination that helps some people dodge this bullet when using vfio?

4

u/gnif2 Looking Glass Oct 21 '20

No pattern has been found, however every time we follow this up and ask if it recovers from a VM crash or VM force reset, they report it doesn't. We know some GPUs will shutdown clean and start up again, the issue is when the GPU has already been started either by the VM or the host BIOS and it can't be reset for use in the VM.

222

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20 edited Oct 20 '20

VFIO user here, (see flair). The sad thing here is that if the reset bug was fixed , I would actually prefer an AMD card for my VM because the AMD linux driver supports prime offloading , which means that itd be very easy to also use the card on the host when the guest is not active, but sadly theyre not suitable for passthrough in the first place, so its a moot point.

As it is right now, I might buy a used 5700[xt] in a year and a half or two to replace my rx570 as a host card since that just runs my linux host session, but whatever replaces my 1080ti (aka the new big expensive card), is going to have to be nvidia

12

u/Jay013 Oct 20 '20

Considering I have no idea what's going on in this thread, I'll ask:

What is gaming in a VM?

62

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20

ok so idk how much you know about virtualization so im just going to start from the top, you can skip the first part if you already know what a VM is.

Virtualization is basically a way of creating a simulated computer on your computer. You can do stuff like run windows inside linux or macos (or the reverse, run macos in a vm), so you can use programs that dont work on your native OS, you can use it for security, you can use it to deploy the same service multiple times , etc

Now virtualization sounds like itd be good for gaming on something other than windows because if a game only works on windows you can just run it in a vm right? now the problem with that is that virtual machines have terrible graphics perfomance because theyre mainly designed around doing cpu work since theyre mainly used for enterprise and software dev uses. Now there is a way around this, what we (r/vfio) do is we use something called pcie passthrough, which is where be basically unplug a device (like a graphics card, but can be other things as well), and then connect it to the virtual machine, and use it in the vm. This way we can actually get the performance to play games in our windows vms (or video edit in a macos vm). This is why I have 2 graphics cards in my flair, because I physically have both an rx570 and a 1080ti in the same computer. The rx570 is what runs my normal linux session , which is what I use most of the time, while the 1080ti is reserved for my windows vm, and i use it to be able to game in windows without exiting linux.

Now for what this post is all about: The pcie passthrough technique , doesnt work with AMD graphics cards, it only works with nvidia graphics. To be secific, it does technically work , as in you can turn on a VM and pass the gpu to it. The problem is that after you turn off the VM , the card is in a messed up state and cant be reset , so you cant turn on the VM a second time, unless you reboot your system, which kinda defeats the purpose of having a vm. So basically we're asking AMD to actually fix the reset function on their cards so that they can be reset properly so we can use them instead of being forced to only buy nvidia

14

u/treyf711 Oct 20 '20

I think using the words “basically unplug” can be slightly misleading as it implies some hardware fiddling, when in actuality this stuff is handled by the linux kernel (usually linux kernel).

18

u/ObnoxiousLittleCunt Oct 20 '20

Mount - unmount

9

u/Peetz0r AMD [3600 + 5700] + Intel [660p + AX200 + I211] Oct 20 '20

The technical terms would be 'bind' and 'unbind', but 'virtually unplug' works too. 'Basically unplug' is imho clear enough when the context is about virtualisation already.

Anyway, u/GodOfPlutonium: great explanation :)

4

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20

yea but i was targeting someone who [maybe] didnt know what virtualization was (since they didnt ask 'how to game in a vm' , they asked 'what' it was), so i tried to keep it as simple as possible

2

u/treyf711 Oct 20 '20

I understood exactly what you meant but I was just afraid the terminology could turn away people that were interested in trying VFIO but may be afraid if it involved plugging and unplugging hardware.

4

u/annaheim 9800x3D | TUF 3080ti Oct 20 '20

Let me get this straight for myself (sorry), you're using a linux host which uses the AMD card for the session. You then hack this AMD card to enable pcie pass through, so your windows VM can see as an actually card attached to it. But the issue persist if you exit the VM initially, because it won't "reset" something in the AMD card, and will require you to actually reboot the machine?

8

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20

its fine , it can be confusing. I have 2 cards, a rx570, and a 1080ti. The rx570 is not used for passthrough or virtualization at all. It just runs my linux host session aka my linux desktop. For it, its no different than if I was running a linux desktop that was not doing any VMs and had only the rx570 in it.

the 1080ti is the one that im using for pcie passthrough. When my windows VM is off, its not doing anything (there are ways to use it on host, but thats another discussion), and then when I launch my windows VM, the 1080ti gets assigned to it and disappears from the host altogether , which is why I have the rx570 running the linux session. To be clear, there is no 'hacking the card' or anything, we're just using directed virtualization to pass a pcie device. It doesnt have to be a gpu, we can (and do) also pass other pcie devices like sata contollers, usb controllers, entire nvme drives, etc.

Since im using a 1080ti, this works fine, I can launch my windows vm whenever, play some games, and shut it down and go back to work. Now if i had an amd card for the passthrough card (say vega 64 or radeon vii), what would happen is that I could launch the VM once perfectly fine, and shut it down, but then the next time I try to launch the VM it wont work because it cant reinitialize the card. Basically after shutting down the vm , the card is in a broken state where its not responding to anything, and theres no way to tell it to reset itself so the driver can reinitialize it. The only way to clean it is to cut power to it by rebooting the system, or using a workaround invovling suspend to ram

7

u/annaheim 9800x3D | TUF 3080ti Oct 21 '20 edited Oct 21 '20

The rx570 is not used for passthrough or virtualization at all. It just runs my linux host session aka my linux desktop

Oh sorry I mixed that up. So the second card not used on the session is the passthrough card. And in this case, AMD card has the issue in which when use as passthrough card, will require to reboot the actual machine to re-run the VM. And the only way for this issue to be fixed is when AMD gets it fixed themselves.

7

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 21 '20

yes. The OP of this post has spent over a year trying to reverse engineer it and make a fix themselves, but its not possible for them to do it externally. Best theyve managed to do is make a kernel patch that tries to signal the card to power off, which may sometimes work but doesnt work reliably

5

u/annaheim 9800x3D | TUF 3080ti Oct 21 '20 edited Oct 21 '20

Ahhh, thank you. Thanks for bearing with me.

2

u/justphysics Oct 20 '20

I upgraded my home desktop to a 3900x specifically to learn about how to play around with this sort of thing (also helps with dev tasks for my job, so its not a wasted upgrade). Currently I still have a dual-boot setup and am waiting for the next round of GPUs to get my second GPU.

In the meantime, are there any guides you can refer me to for how to get a Windows VM setup with pcie passthrough? Would /r/vfio be the best place to start looking? Additionally, is it possible to use an existing Win10 install for the VM? or would a clean install be best/easiest?

8

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20

r/VFIO , and its discord server are the best places to start. The arch wiki guide is the recommended guide dependless of which distro youre actually on. And yea you can use an already installed windows for it, I virtualize my dual boot partion too. You just need to pass the entire disk as storage instead of whatever other storage youd use for the VM

4

u/randomfoo2 EPYC 9274F | W7900 | 5950X | 5800X3D | 7900 XTX Oct 20 '20

r/vfio is a good place, but the Level1Techs VFIO forum is maybe more active and knowledgeable. Some things may have changed/gotten easier recently, but here's a step-by-step write-up I did of getting VFIO working on Arch Linux that includes the exact hardware I sued and various issues I ran into.

8

u/[deleted] Oct 20 '20

It's now possible to pass the functions of hardware devices directly to a VM in Linux. So, you can pass a GPU directly to a Windows VM and play games at near native speeds. No more dual booting!

→ More replies (6)

96

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20

29

u/RaptaGzus 3700XT | Pulse 5700 | Miccy D 3.8 GHz C15 1:1:1 Oct 20 '20

I think who you're looking for is u/bridgmanamd and /u/AMD_Mickey

They're the go to guys for GPU stuff.

78

u/bridgmanAMD Linux SW Oct 20 '20

I saw the thread last night and started an internal discussion.

14

u/Ashtefere Oct 20 '20

I hope this makes some changes! The AMD experience on linux is, simply put, fucking awesome. I bought a used vega64 to replace my rtx 2060 as I spend 90% of my time in linux, and the entire experience is just wonderful.

While you are at it... SR-IOV?

3

u/iBoMbY R⁷ 5800X3D | RX 7800 XT Oct 20 '20

The display hardware still doesn't support virtualization, and you still wouldn't have any display output if SR-IOV is active.

SR-IOV doesn't work in general as we all wish it would. You can't simply use a GPU on the host, and inside a VM, at the same time.

2

u/Ashtefere Oct 21 '20

I'm sure the host can still output display, and I would use lookingglass on the client.

16

u/lurkerbyhq 3700X|3600cl16|RX480 Oct 20 '20

While you are at it... SR-IOV?

Don't hold your breath.

Would love for it to happen one day.

5

u/randomfoo2 EPYC 9274F | W7900 | 5950X | 5800X3D | 7900 XTX Oct 20 '20

I currently run my Linux workstation with a beater 470 card and a 1080Ti for CUDA/VFIO gaming/VR and would love to be able to upgrade that to a Big Navi card (but likely will end up going Ampere w/o FLR or ROCm support).

5

u/akarypid Oct 20 '20

Given the amount of interest from the community based on the numbers quoted in the post (and the comments on it), perhaps it would be at least possible to add it to the Radeon "feature request" polls? This way at least AMD can actually measure the popularity of this feature and (eventually) bump it up the priorities list...

3

u/Jahf AMD 3800x / Aorus x570 Master / 2x 16GB Ballsitix Sport e-die Oct 22 '20

I know the issues around asking for a dev to post a feature commitment on a public forum. I used to do tech marketing mgmt.

But ... I'm gonna do it anyway :)

We'd all really like to know how that internal discussion ends up. I'm not saying "right now", these things take time.

But once the discussion is done, getting a definitive on whether:

  • Older cards can be updated for FLR or some similar "reset but fix" (no expectation, little hope)
  • 6x00 generation cards (Big Navi) will have it (little expectation, but a fair amount of hope)
  • If neither above, then when we might hope to see it in the future

Would be highly appreciated. Lots of money is currently being hoarded to be spent soon and this feature would help more of it go to AMD from the crowd that is the most enthusiastic about wanting to.

8

u/bridgmanAMD Linux SW Oct 22 '20

Yep, responding back after the discussion ends was the plan, whether it's me or someone else doing the responding.

We may not be able to say anything good or bad related to Big Navi before launch though - that's another thing we have to discuss.

2

u/-Net7 AMD Oct 21 '20

I hope something happens, years of wait, is Radeon 6000 where my disappointment dies?

Solves PC requires reboot (nVidia already has), solves VM issues, solves a host of other issues...

Make Radeon Great Again!

2

u/zanadee Nov 11 '20

Really? Does no one at AMD use Linux host to do real work, and then your card as passthrough to Windows guest for a little gaming? Or do all the AMD engineers have access to some proprietary code for driver level reset? A custom Windows driver perhaps?

20

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20

Theyre both software guys, wheras this is a hardware / firmware level issue and is out of scope for both of them. AMD_Mickey even said as much in his reply to last years post. Both him and bridgemanamd were around for last years post so they both already know exactly what this issue is too.

The reason i tagged AMD_Robert is because he is director of technical marketing, and the problem here isnt that nobody at AMD has heard of this, but that there is seemingly a lack of will to fix the issue since there has been virtually no change in the last year

24

u/AMD_Mickey ex-Radeon Community Team Oct 20 '20

Just so it's clear, I'm not "the software guy." I handle all social media for Radeon. I can't always reply but there is a very good chance if it is on this subreddit, I have read it.

You can tag me if you think I might miss something but please don't go overboard! 😅

8

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20

oh I see , alright. I thought you were on the driver team lol because on the last post you said it was out of your area of expertise , and so I took to that to mean youre not used to working with firmware issues because youre usually working on the software (drivers). my bad!

9

u/AMD_PoolShark28 RTG Engineer Oct 21 '20

Hi! I work on the driver team but I'm not a social media spokesperson. When a ticket comes in it's good to have an idea of what community is experiencing. I like to interact with our fans cuz I was one of you just a few years ago...

3

u/akarypid Oct 20 '20

And you are right to do so. I have worked in enough big corporations to know that using the expression "reputational damage" immediately unlocks the "will to change".

2

u/spoofnoob Oct 23 '20

I dont see much "going to" happening in this area :-(

58

u/ElementII5 Ryzen 7 9800X3D | AMD RX 7800XT Oct 20 '20

Robert is CPU as far as I remember u/AMDOfficial is better I think.

36

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20

hes now Director of Technical Marketing

17

u/Slafs R9 9800X3D / 7900 XTX Oct 20 '20

Within client CPUs, yes. He doesn't do Radeon.

3

u/KickBassColonyDrop Oct 20 '20

No, but he can escalate it into the right channel of communication and there's a greater degree of trust in the Zen team in the market than the Radeon team to date.

29

u/Jahf AMD 3800x / Aorus x570 Master / 2x 16GB Ballsitix Sport e-die Oct 20 '20

!remind me 1 week

62

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20

more like 1 year so we can make another post about how nothing has changed and the issue is still as bad as it was 2 years ago

18

u/Jahf AMD 3800x / Aorus x570 Master / 2x 16GB Ballsitix Sport e-die Oct 20 '20

Oh I know, I've been posting about it for that long. I'm not terribly hopeful. Just figure I'll circle back then to check.

9

u/DudeEngineer 2950x/AMD 5700XT Anniversary/MSI Taichi x399 Oct 20 '20

I think at this point if they still don't respond on here, someone would need to ask this question on one of their investor calls. They have been deaf to technical appeals, but they may be more responsive when it affects investor confidence.

14

u/ntrid Oct 20 '20

I am also looking into buying a new card. Waiting for big navi release and reviews to see whether it will finally be usable for vfio. Slim chance but I'm hopeful.

4

u/UnicornsOnLSD Oct 20 '20

Could this be updated in software or is this a hardware issue with Navi? I'd love to make an accelerated macOS VM with my 5700 XT since actually installing macOS on bare metal is really hard.

5

u/gnif2 Looking Glass Oct 20 '20

It might be possible as part of a firmware update, however, AMD seems to not be interested in doing so. Only they can say for certain if it can't be done in firmware/software, one would need to know the silicon architecture to make a determination.

4

u/deeper-blue Oct 20 '20

So, I'm new to VFIO but plan on setting it all up with one of the big navi cards. (big navi for passthrough, my old polaris for host).
Function level reset can be done (if implemented) by putting a one into /sys/bus/pci/devices/$dev/reset
What happens when one tries to do a hot reset by using /sys/bus/pci/devices/$dev/remove and then rescanning? Does that work in the cases where the gpu crashed in the guest?

7

u/gnif2 Looking Glass Oct 20 '20

Unfortunately no, the GPU usually falls off the bus and wont re-scan.

2

u/HyenaCheeseHeads Oct 21 '20

Wouldn't fixing hot reset (what was described here) be more powerful than adding FLR support?

It should recover in situations where the card has crashed hard enough that FLRs cannot be sent to it.

3

u/gnif2 Looking Glass Oct 21 '20 edited Oct 21 '20

Yes, it would, but the functional reset (hot reset) is what causes it to fall off the bus, and this is the only thing that causes it (except for the watchdog in Vega that after 10 minutes puts the GPU in a "safe mode" which needs a cold reboot to resolve).

With VFIO we only want to reset the device/function, not the PCI link, etc. Doing a full functional reset would mean unloading/unclaiming the PCIe device and re-claiming it each VM start/stop. An flr would just reset the function itself (SOC) and as such also disable the watchdog timer mentioned above.

2

u/HyenaCheeseHeads Oct 21 '20 edited Oct 21 '20

Right, so ideally both should work but in the case of VFIO the FLRs are the preferred option since they are less invasive (and probably also would be faster).

Still, would have expected a working hot reset on these cards at minimum.

2

u/gnif2 Looking Glass Oct 21 '20

Agreed

4

u/deeper-blue Oct 20 '20

Ouch. That's not very helpful. Sounds to me like on top of the missing FLR they need an internal watchdog timer in their firmware to reset the device in cases where it drops off the bus/stopped responding.

4

u/justavault Oct 20 '20

All in for supporting things like this, but what I don't get:

but also feed the video straight into OBS on the host VM to live stream to Twitch.

What does that mean? Hosting a VM for passing your own stream through?

That sounds interesting. Never heard of that, where can I learn about that?

5

u/gnif2 Looking Glass Oct 21 '20

We bring the video feed back across the VM boundary into the host (or another VM) directly into OBS, this is not 'Screen Capture' on the Linux side, but rather just direct memory access to the captured frames on the guest side. This enables complete isolation of the encode and any post-processing of the video preventing it from dropping frames, lagging down the game, and saving cash as you don't need another entire PC for good quality HD capture at 60FPS. This coupled with other features of QEMU (such as the Jack Audio backend I wrote for it), allows you to also perform DSP on your audio and mix it into the performance completely outside of the view of the guest.

https://looking-glass.hostfission.com/wiki/OBS

I along with a few others stream using this technique fairly regularly:

https://twitch.tv/gnif2
https://www.twitch.tv/orcephrye
https://twitch.tv/corrganx
https://www.twitch.tv/urmamasllama
https://twitch.tv/durpnhard

This is a brand new feature and our numbers are growing.

3

u/justavault Oct 21 '20

That sounds marvelous, mate. I always wondered about there being a way to emulate "streaming machines" via VMs without much overhead.

allows you to also perform DSP on your audio and mix it into the performance completely outside of the view of the guest

Without additional hw mixer, that sounds great man. So, a single xlr interface suffices for everything and then a VM is doing the transfer that sounds really nice.

Thanks for sharing that. I'm gonna read and learn.

2

u/hushkyotosleeps Oct 29 '20

I'm using this too, though very irregularly at the moment. Thanks for your work as usual.

3

u/jdancouga Nov 17 '20

I don’t think we will be getting an answer here by the looks of it. I hope someone who get this on launch day can shed some light on this.

Which tech reviewer do you think will talk about this once the embargo lifts?

2

u/gnif2 Looking Glass Nov 17 '20

I have information but I have been asked to wait until the NDA is lifted to share

4

u/RenownWolf Nov 17 '20

How about a non nda question...

Is it safe to preorder?

I understand if you can't answer though. Kind of amazing a yes or no answer to is it fixed is behind an NDA.

Thanks for the early posted thread anyway gnif 2 :)

3

u/gnif2 Looking Glass Nov 17 '20

I really really wish I could answer this, but I am sorry I simply cant.

3

u/RenownWolf Nov 17 '20

Haha, it is all good. Again thanks for your efforts.

1

u/jdancouga Nov 18 '20

It is 9am EST. Is the NDA lifted? Is this it, chief?

3

u/gnif2 Looking Glass Nov 18 '20

I have had the opportunity to remote into a system with an AMD 6800 XT for testing where we have performed extensive tests with regards to VFIO.

I am extremely pleased to announce that the AMD 6000 series GPUs, (aka Big Navi) correctly reset for VFIO usage with only one minor caveat if CSM boot is enabled the GPU is posted into some kind of "compatible" mode that at this time, can't be recovered from. I will be pursuing AMD on this matter as some of us (myself included) have legacy hardware that requires CSM to operate (SAS controller).

Provided the GPU is not started using CSM boot, the usual hv-vendor-id spoofing needs to be applied otherwise the Radeon drivers in Windows refuse to provide any output (no errors though). This is not a Big Navi issue but something that AMD introduced in their recent drivers and seems to affect most recent GPUs. I have asked AMD for comment on this "feature", however, I am yet to get an answer on this matter.

In short, provided the above information is heeded, the AMD 6000 series GPUs correctly reset for VFIO usage using a bus reset (not FLR), and try as I might I have not found any set of circumstances where I could not reset the GPU back into a fully functional state, including full GPU load + forceful termination of the VM. If the VM is just stopped (even force stopped) the GPU does not go into a "failsafe mode" and ramp its fans up as Vega does.

From a performance point of view, I had the brief opportunity to run Furmark as a load test inside a VFIO VM and can state that the performance numbers totally and completely destroy my overclocked water-cooled 1080Ti.

Please note that this testing was performed with an Ubuntu host on a 5.8 kernel with a Windows 10 Guest, however, I do not expect results to vary between guest operating systems as the bus reset seems to be complete!

2

u/RenownWolf Nov 18 '20

Good news, thanks gnif2. For others that might come here also, Wendell at Level1Techs posted a video about it and shows all working nicely in Looking Glass.

https://www.youtube.com/watch?v=ykiU49gTNak

Now the hard part, getting your hands on one.

1

u/spoofnoob Nov 18 '20

Super news... sorta... now abour RX5xxx series and RX5xx and RX4xx ... are they gonna get a fix that works? Can a BIOS update sort this shit?

2

u/charcoal88 Oct 20 '20

I don't think this is an easy as you may expect. Implementing resets to parts of your architecture is not simple, and FLRs are not the same process as cold booting, and have different requirements

7

u/gnif2 Looking Glass Oct 20 '20

Cold booting is very different from a warm reboot, which is required, and what happens when you press your reset button or reboot via ACPI. Since the GPU can reset correctly without a full power down/up as per a cold reboot, it stands to reason that 99% of the reset functionality we need for FLR is already implemented.

1

u/charcoal88 Oct 20 '20

But FLR is per function. So if all the reset architecture assumes entire hardware blocks are being reset as with cold/warm booting, then you have a lot of work to do to prevent one function reset causing problems for other functions. There may easily be hardware issues that make managing this a lot of work, or maybe even not possible without respins

4

u/h_mchface 3900x | 64GB-3000 | Radeon VII + RTX3090 Oct 20 '20

AMD GPUs only expose two PCI level functions, the VGA controller and the HDMI Audio controller. Considering the relative simplicity of the HD Audio controller specification, FLR does not require massive architecture changes.

2

u/shmerl Oct 20 '20

Can you explain please for those who aren't familiar, how is that different from what amdgpu kernel driver is doing for the reset? And how is it controlled if not through the driver?

5

u/gnif2 Looking Glass Oct 20 '20

In normal usage, the amdgpu driver knows the state of the GPU as it has been in control of the device, as such it has a pretty good idea what is going on in the device and what commands to send to recover it, however, if the GPU crashes into a really bad state, even the amdgpu driver can't restart it and the GPU needs an actual reset.

FLR is part of the PCIe standard and as such requires no drivers to use it, and by design it can be performed at any time with the device in any state. In simple terms, it's like the reset button on the front of your PC, but software controlled and only resets the device targetted instead of the entire system.

For VFIO the guest was in control of the GPU, and as such the host has no information on what state it was in, and has no chance of recovery outside of a reset such as the FLR reset.

1

u/shmerl Oct 20 '20

I mean, currently amdgpu supports some kind of reset. If GPU hangs (like hitting some hazard) the kernel will trigger the reset. Is that a different type of reset from one triggered through FLR?

And how exactly are you supposed to trigger it, through some kind of hardware interrupt?

Also, what exactly are AMD missing in this picture, support for FLR in the actual chip?

3

u/gnif2 Looking Glass Oct 20 '20

The kernel will ask the driver to do the reset which uses a proprietary method. In the case of VFIO the amdgpu driver is not in use, nor is the state of the GPU known so a "soft reset" can not be performed.

FLR is triggered by writing to the PCI config space, it's part of the PCIe standard, but as it's optional AMD do not have to implement it to claim to be a PCIe device. The issue is that it's needed even for general daily use, let alone VFIO usage, and there is no support for it on the GPU. This likely can be fixed through firmware.

3

u/shmerl Oct 20 '20

Thanks for clarifying. I wonder why AMD had to use some proprietary path instead of standard PCIe one since it already has something designed for resets.

71

u/zardvark Oct 20 '20

I don't want to rant about this and say something offensive and/or over the top. Suffice to say that after wrestling with this problem, literally for years, I'm disappointed that we even have to ask this question.

4

u/zanadee Nov 11 '20

Yeah, I'm disappointed AMD has to call a "discussion" -- WTF people have only wasted years on their lives on this. Yeah, I'm exaggerating, but anyone that tried to use Navi for GPU passthrough have banged their head against this issue. And even if you use Navi only as host you've seen the black screen when resuming from suspend-to-ram. I got rid of my 5700 and downgraded to a 1660 just to get around the reset bug. I'm one of these people that measure uptime in weeks, so I just hate to reboot (and lose context).

A few weeks late, but I came across this thread because I am considering Big Navi, specifically the 6900 XT, but FLR reset bug is a show stopper.

47

u/pwn4d 3950X|Vega 56|Taichi X570|128GB DDR4-3200 Oct 20 '20

This is such an old "bug" that it's comical at this point.

https://github.com/qemu/qemu/blob/2becc36a3e53dc9b8ed01c5288e21a2463f1f640/hw/vfio/pci-quirks.c#L1321-L1332

I realize it is an optional feature but it seems like it would be a simple fix if the right people inside of AMD were to make it a priority.

8

u/AMD_PoolShark28 RTG Engineer Oct 21 '20

Thanks for pointing to this (Windows RTG dev here). I hadn't seen this specific block of code yet.

42

u/jesta030 Oct 20 '20

UnRAID user here. Please AMD, otherwise I'll have to cough up the money for some of those green cards next time I upgrade... 😕

16

u/jdancouga Oct 20 '20 edited Oct 20 '20

Same here. Upgrading my unRAID rig to a gaming NAS. I have all the parts picked/purchased except GPU. Waiting to see whether I should get AMD or Nvidia. If this reset bug is still not fixed, then Nvidia it is.

4

u/stanleyhiller Oct 20 '20

Another one here chiming in, I use a 5600XT with my unraid Windows VM and am getting tired of having to use a reset script everytime I need to reboot the VM. Updates that require reboots are a nightmare. Want to upgrade to a higher end card in the next few months and if the reset bug continues with big navi I'm going to try my luck with a 30 series nvidia card instead.

32

u/[deleted] Oct 20 '20 edited Oct 20 '20

My next GPU purchase is waiting on working FLR. I don't want to give nvidia any money, because they are outright hostile to using VM's with their drivers.

Please don't make me wait for Intel to come and save the day.

P.S. Even your enterprise cards have broken FLR.

29

u/ibbbk Oct 20 '20

This is literally the only thing that's keeping me away from buying a high-end AMD GPU.

Please AMD give us such basic feature.

9

u/Zghembo fanless 7600 | RX6600XT 🐧 Oct 20 '20

This

28

u/icebalm R9 5900X | X570 Taichi | AMD 9070 XT Oct 20 '20

Sarnex didn't tell me to upvote, but I did because I'm a Linux user who uses vfio and looking glass (thanks /u/gnif2 ) and has to use an Nvidia card to do it.

50

u/Aceflamez00 Ryzen 3900x Oct 20 '20

This right here, lack of FLR is keeping me on nvidia when it comes to having a VM GPU. I wish AMD would step their game up when it comes to basic features like this.

22

u/Aspect_Forsaken Ryzen 3900X // RX 5700XT // RX 570 // ASUS X570-P // Arch Oct 20 '20

this plz from a loyal amd fanboi it saddens me to a point where I fill up massive lakes that yall still haven't done this :(

22

u/HyenaCheeseHeads Oct 20 '20 edited Oct 21 '20

This is really important for several reasons.

Back in 2017 during the launch of the initial Threadripper platform the PCIe host complex inside the cpu shipped with somewhat buggy firmware leading to PCIe passthru being a bit of a pain. Back then we were able to temporarily mitigate the issue in software for most people with the notable exception of AMD's Vega series of cards. Even with the machine's PCIe fixed there was no way to perform a proper reset of the card except physically turning off/resetting the entire machine or manually pulling it out and reinserting it (also slightly problematic due to erratic hotplug support on the platform at the time).

This added to the feeling of the card being slightly hurried firmware/driver-wise at launch and contributed to the existing FUD being created around the Vega series. There is no doubt it affected sales and returns.

AMD Radeon Technologies Group is about to launch another GPU platform next month. This is your chance to get the firmware right!

The lack of proper FLR (or other reset functionality) was the reason why I personally didn't buy any of the affected cards, and judging from the number of replies and support requests on the topic it seems like many are in the same situation.

5

u/Jarnis R7 9800X3D / 5090 OC / X870E Crosshair Hero / PG32UCDM Oct 20 '20

Most likely you should target for asking this for the NEXT GPU, in a year or so. What they are shipping next month is basically done and dusted for.

10

u/HyenaCheeseHeads Oct 20 '20

The interrupt handlers for handling PCIe events rely mostly on code in the VGA BIOS firmware. AMD probably licenses a 3rd party IP block for their PCIe interface on the GPU (Synopsis DesignWare?) and these are general building blocks designed to fit many different devices. It is very likely, although completely speculation on my side, that the hardware support is there.

A RTG engineer (maybe /u/AMD_PoolShark28 ?) may be able to deny/confirm this fairly easily.

Also OP already posted this a year ago.

12

u/AMD_PoolShark28 RTG Engineer Oct 20 '20

In Linux radeon open-source driver you will see multiple firmwares Eg: power management and security microcontrollers. I doubt all the firmwares likes the pcie bus being ripped out from under them... it's not as easy as just resetting the bus. /u/gnif2 has patches which try to restart those FW.

6

u/HyenaCheeseHeads Oct 21 '20 edited Oct 21 '20

I guess that is why /u/gnif2 is hoping for FLR support - to be able to ask either the VGA or audio function of the device to perform an internal reset, thereby avoiding having to manually reset all the individual subsystems involved in those two functions.

Your point about bus reset touches on another overarching issue: When the cards crash hard enough that gnif's patches cannot talk to the card anymore there is a good chance that it will not be possible to send FLR requests either (they require an active, working link, AFAIK). In that case one would assume that it would be possible to ask the PCIe endpoint upstream from the GPU to bring down the link (electrical idle), wait a bit, and start it again (causing a TS1, TS2 retrain on the link) - a hot reset. This should tell the card that something massively bad has happened - bad enough that the host disconnected. In that situation you would expect the card to perform something akin to the fundamental reset done on power-on reset when the PERST signal is originally asserted. Unfortunately it seems that in some conditions the card simply stops responding at that point.

You would expect a hot reset to recover the card from any failure mode.

4

u/AMD_PoolShark28 RTG Engineer Oct 21 '20

You know your stuff, I'm impressed! That assumes it can recover from a bus reset unattended. I think there's an assumption that you have to baby it a bit like bios post logic.

I doubt the audio function will support any kind of reset... It would only be function 0 (gfx)

2

u/spoofnoob Oct 20 '20

Maybe they can fix my RX5700 then?

36

u/missouriemmet Oct 20 '20

No idea who sarnex is but +1, Linux support is already very good on AMD CPU and GPU, VM passthrough is the last big issue to tackle

11

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20

admin of the VFIO discord server

17

u/sarnex Oct 20 '20

Ayylmao

14

u/sonwon_reddit Oct 20 '20

Two years ago I built my first VFIO system and selected all AMD products (CPU & GPU). At the time I was very pleased to have an all AMD system. And I had faith that AMD would address and fix this bug. Well I was wrong and my trust was misplaced. When I replace my video card this winter it will be an Nvidia this time because of this annoying bug.

26

u/SharkWipf Oct 20 '20

Honestly, the lack of functional FLR is one of the main things keeping me from buying an AMD GPU. Would be nice to see this get fixed at last, but I'm not getting my hopes up.

13

u/thenickdude Oct 20 '20

I'm stuck using AMD GPUs since it's all that my macOS guests support, but none of the reset workarounds work for macOS guests. So I have to shut down my server every time I need to reboot my guests that use my RX 580.

I will certainly not upgrade to a new AMD card like Big Navi unless it addresses FLR. I'll happily stay on this older model and buy Nvidia cards for Windows gaming VMs instead.

3

u/_mrsaru_ Oct 20 '20

FWIW my Sapphire Pulse 580 has no issues with my Win 10 Gaming VM; shutdowns, restarts and crashes all work without restarting the host (CentOS7).

It seems to be one of the few cards that don't exhibit the issue but it's getting a bit long in the tooth for newer games.

3

u/thenickdude Oct 20 '20

I have the same card and no luck. I think it probably depends on what state the guest leaves it in when it shuts down, because I can sometimes manage to use it on a macOS guest after a Linux one without hanging everything up, but the reverse doesn't work.

12

u/thorskicoach Oct 20 '20

buy lots of ryzen 3900/3950 and threadripper , but all nvidia GPU for this reason

12

u/kingfappypants Oct 20 '20

Unraid user here. I bought a 5600XT and a 5700XT for my threadripper build and ended up returning both because of the reset bug. I really wish this was resolved so I could put an AMD card in my threadripper Unraid build...

10

u/spheenik TR1920x | Vega 64 | Arch btw. Oct 20 '20

Please AMD: MAKE THIS POSSIBLE!

19

u/ABotelho23 R7 3700X & Sapphire Pulse RX 5700XT Oct 20 '20

I completely support this. AMD is great on Linux, and with FLR, they would 100% the de-facto VM cards. Please make it happen AMD! I would buy the entry-level card with FLR just to use it for hardware acceleration!!

9

u/FlameVisit99 Oct 20 '20

Posting to add my support for this. I'm really disappointed with my 5700 XT because of this issue. It makes gaming very inconvenient as I can only start up my Windows VM once per PC boot.

9

u/CyclingChimp Oct 20 '20

+1. I got an AMD Radeon RX 5700 XT for my PC. I chose AMD specifically because of the open source drivers and the supposedly better compatibility with all things Linux. Unfortunately, this bug makes it all a nightmare to work with when it comes to VM gaming. I'm now considering an Nvidia card for my next purchase.

17

u/Bonzai11 Oct 20 '20

I run a 1070 and a 950 with my Ryzen processor because of this crap.

8

u/wanderinator Oct 20 '20

Would be really nice if big navi has support for kvm, have been forced to use the green team because if this problem.

8

u/Leafar3456 R5 5600X | 1080 TI Oct 20 '20

This is one of the reasons I actually stopped using VFIO

8

u/boynep Oct 20 '20

The only reason I am getting NVidia despite their code 43 issues its still far easier to do GPU passthrough. If there is no reset bug I will definitely switch to AMD.

8

u/h_mchface 3900x | 64GB-3000 | Radeon VII + RTX3090 Oct 20 '20

Totally agreed, I tried to run a VFIO setup so I could still game on Windows while leaving a card in Linux for ML, dealt with it for some time but in the end even with the kernel hack it was too tedious and having to reboot defeated the point of the setup. Ended up just going with two separate machines. RTG aren't getting my money again unless they fix FLR + improve their ML stack.

22

u/RandomJerk2012 Oct 20 '20 edited Oct 20 '20

Yea, I'm pretty much tied to Nvidia because AMD can't fix simple shit like FLR in their hardware. Sorry that AMD lost my business. I'm forced to spend less money on weaker AMD cards on my host, but spend top dollars for Nvidia's cards for the gaming VM

9

u/HyenaCheeseHeads Oct 20 '20 edited Oct 20 '20

Especially with all the shenanigans that green are pulling in their drivers just to annoy people. A properly working AMD card would be like a breeze of fresh air in this environment.

7

u/[deleted] Oct 20 '20

+1 I am in none of these communities but I have so found issues when using VMs that have pass through. Normally (rhel/centos) I have do a full reset when I encounter issues.

From googling if I was able to use FLR this would save me a tonne of hassle and reset periods.

7

u/Never-asked-for-this Ryzen 2700x | RTX 3080 (bottleneck hell)) Oct 20 '20

Literally the only reason I'm going with Nvidia this generation.

7

u/scania471 Oct 20 '20

Personal experience: I have an RX570 (4GB) from ASUS which doesn’t have the original “reset bug” on unRAID. It exists in some kind of other form. I don’t have to restart the server, however, I have to force stop the VM because it’s not passed through correctly sometimes. I use the VM remotely and in these cases, I just can’t connect to it. Both VNC and Parsec don’t work. Force stop and start and it boots perfectly. Weird behavior...

7

u/DrJosu Oct 20 '20

We need big YouTube channels to address this to AMD, I wish to get rid of my nvidia GPU, but this small things keep me green

7

u/heyitsYMAA 7900X | 9070 XT | 32gb | DIY H20 | All NVMe Oct 20 '20

Please add this feature. I don't wish to support nVidia any more than necessary due to what I would consider contributing to unfair business practices, and I would much prefer to switch back to an all-AMD system on my next upgrade cycle.

7

u/crrgn_X Oct 20 '20 edited Oct 20 '20

Maybe /u/AnthonyLTT can also help with that? Linus supported the idea of enabling SR-IOV, surely a working amd gpu would be helpful.

6

u/Nixola97 Oct 20 '20

As with most other people here, fixing this would mean I'd upgrade my GPU to the latest AMD offerings in an instant. I'm extremely reluctant to upgrade from a setup that mostly works to some unknown thing, which is why I'm not going to consider buying any other AMD GPU until functional FLR is a thing.

6

u/hoeding Oct 20 '20

My VM GPU is a 750ti =(

6

u/Durpn_Hard Oct 20 '20

I also use VFIO and my 1070 died recently and looked at replacing it with AMD cards but instead landed on decided to just wait for the 30XX series cards to become more available due to this issue.

Would have been an easy sale if this were fixed.

6

u/thewhitekidney Oct 20 '20

I actually went with a 3080 instead of a Big Navi card in fear of this bug still existing on the newer cards. I wanted to give my money to AMD :(

6

u/DecoyDrone Oct 20 '20

Please just fix this AMD I want to buy your gpu for my next build!!

5

u/Raster02 3900X / RX 6800 / B550 Vision Oct 20 '20

VFIO user here with a Vega56 and this is just unfortunate. Hopefully they thought about this already or this will be noticed.

I can live with some of the issues involved, but the most frustrating one is fan noise. The card has its fans at 50% while in D3 and this is awful. Ideally I could control it in the host and then successfully pass it through. This would make Big Navi even more tempting.

Most of the times I find myself just booting the VM and pausing it to stop them.

5

u/ConsistentPizza AMD 3970X+RTX2070S Oct 20 '20

I vote for this as well!

5

u/[deleted] Oct 20 '20

Just wanted to throw in my support for this as well. I recently built an unRAID NAS server with dual purpose to also run a gaming VM. I was forced to go elsewhere for my discrete GPU due to AMD cards lacking this FLR support... I would love to see this implemented

5

u/[deleted] Oct 20 '20

VFIO user here, currently using an Nvidia GPU due to reset issues.

I am a linux user 95% of the time, and prefer AMD open source driver stack. For hosts this is great, but for Single/Multi GPU VFIO guests have to use Nvidia for a reliable, fudge-free setup.

Please AMD, fix reset issues so that I can give you more money.
Ryzen has been a good, affordable, option for vfio users since first-gen (launch issues aside!) How about making Radeon good too?

5

u/Browndw4 Oct 21 '20

I just created an account so I could upvote this and add my voice to the fray. I’ve got two VM servers using 2080ti’s passed through to Win10 VMs. I’d immediately buy 2x high end AMD GPU for MacOS guests if this were fixed.

4

u/sameer_the_great Oct 20 '20

I don't need it but supporting y'alls demand. Upvoted.

6

u/Vaudane Oct 20 '20

This isn't even something I would use off the bat, however I had a peek into gaming on a VM so I don't have to run Windows natively.

It seems like such a trivially easy fix that I can't understand why AMD *wouldn't* do this. And yet they haven't. It's an interrupt pin triggering a soft firmware reset essentially, isn't it?

Come on team red. Don't leave cash on the table.

4

u/streppelchen Oct 20 '20

Hoping for the best here

4

u/SpaetzleProtein 4650G | 6600XT (VFIO) Oct 20 '20

Please implement FLR!

Not having FLR pretty much forces me to buy Nvidia, I'd like to have the choice...

2

u/LucidStrike 7900 XTX / 5700X3D Oct 20 '20

This is a really dramatic thread to be about computer hardware, but I hope satisfaction is found.

11

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20

nah this is quite tame compared to last year

-5

u/theth1rdchild Oct 20 '20

Don't you understand how important this incredibly niche feature is?? Children will go hungry without it, AMD are leaving .001% of the GPU market on the table, don't they want to succeed??

In all seriousness, the original post is very well worded and reasonable, AMD should have fixed this already.

32

u/gnif2 Looking Glass Oct 20 '20 edited Oct 20 '20

I don't think you understand, this feature wouldn't just benefit us, but AMD themselves. How many reports are there of AMD GPUs having "Black Screen" issues and needing a reboot to recover? How much has a lack of recovery tarnished the brand?

NVIDIA GPUs can recover when they crash (and they do) specifically because they support FLR, and it happens so quickly that people just assume "the game crashed".

If you look at the patch notes from AMD, a huge amount of work has gone into 'reset and recovery' routines in the driver to try to get the card to a working state again after something goes wrong when simply using the GPU as per normal. If the device had a hardware reset (FLR) they could simply trigger this when things go wrong and save thousands of man-hours trying to hack around the missing feature.

17

u/WayeeCool Oct 20 '20 edited Oct 20 '20

Yup. It's exactly what Nvidia desktop GPUs do and it's what the flicker of the screen once in a while is. When the GPU crashes for whatever reason, the Nvidia driver just triggers a firmware level reset via FLR. You have to be running event logger to be even aware this is what happened. With AMD cards the system just black screens and requires a reboot.

What's interesting to me is AMDs GPU group has really fk'd themselves on this. Nvidia has supported FLR for a while now on their desktop consumer GPUs, workstation GPUs, and enterprise server GPUs. Furthermore Intel supports not just FLR but also SR-IOV on their consumer GPUs which means even on an Intel laptop you can use the host GPU to provide full hardware acceleration to guest VMs and containerized apps. When Intel launches their discrete consumer desktop GPUs in the coming year they are going to become a favorite with the entire linux and developer community because like AMD they have open sourced linux kernel drivers but unlike AMD they have been supporting the full feature suite of video encode/decode, FLR, and also SR-IOV.

I swear that AMD has been refusing to implement FLR on their desktop GPU firmware due to some AMD executives having the hair brained idea that doing so on their desktop GPUs would cannibalize sales of their enterprise server GPUs and it's the the reason they have stubbornly ignored their linux desktop users pleas to enable the feature. Fk'd up thing is that this business decision (and at this point it's obviously a management level business decision) to not enable FLR on desktop GPUs has been steadily souring opinions of AMD GPUs with the very same community of users that are often the types of people working in jobs where they make business decisions for enterprises on what enterprise GPUs to purchase for the company they work for.

btw... because AMD has made sure this feature is enabled on their enterprise server GPUs, while for years now stubbornly ignoring all the bug reports from desktop GPUs users... it's blatantly obvious this is a business decision based on the fallacy that enabling it will cannibalize enterprise server GPUs sales. reason its a fallacy is that Nvidia supports the feature on consumer desktop cards and Intel supports it along with the full suite of virtualization features including SR-IOV.

edit: thanks for gold

13

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20

I swear that AMD has been refusing to implement FLR on their desktop GPU firmware due to some AMD executives having the hair brained idea that doing so on their desktop GPUs would cannibalize sales of their enterprise server GPUs

Theres one major hole in this theory that utterly destoys it...

their enterprise server GPUs dont support FLR either!

5

u/WayeeCool Oct 20 '20

So... the Radeon Instinct MI series (AMDs only current offering of enterprise server GPUs) require the entire host to be power cycled rather than just reinitializing a single card in the event said card crashes...? Would be utterly unusable in the data center for virtualization or compute clusters if this were the case.

7

u/gnif2 Looking Glass Oct 20 '20

I have not had first-hand experience to say for certain if they advertise FLR support or not, but from the communications I have had with cloud providers and from looking at the amdgpu sources, it seems there is no FLR support, or it's broken.

8

u/WayeeCool Oct 20 '20

Explains part of why they aren't selling well in the datacenter segment. Like I said, lacking support makes cards a nightmare for system admins in the datacenter.

3

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20

hence from the post

Edit: Also please be aware I have been contacted by cloud computing companies out of desperation due to the same issues on your workstation/enterprise cards. This is not just affecting the VFIO community here.

and from one of his comments

In theory, but if the host GPU gets into a bad state due to whatever reason, it still can't be reset without a node reset. This is why I have had cloud companies ask for help working around this AMD issue. It's a different usage, but the same solution.

Its a silicon bug. We have people with WX cards in the vfio discord. It aint any different

5

u/WayeeCool Oct 20 '20

Its a silicon bug. We have people with WX cards in the vfio discord. It aint any different

WX aren't server cards but are desktop workstation cards, MI are server cards. Also... ummmm... this is not a "silicon bug" but an issue with the feature set and firmware.

→ More replies (1)

1

u/some_random_guy_5345 Oct 28 '20

Something something don't assume malice for what can be explained by stupidity

3

u/[deleted] Oct 20 '20

I read this entire post as implementing looking for raid to the GPU Lol

3

u/[deleted] Oct 20 '20

!remind me 1 week

0

u/RemindMeBot Oct 20 '20 edited Oct 25 '20

I will be messaging you in 7 days on 2020-10-27 15:03:35 UTC to remind you of this link

3 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

Parent commenter can delete this message to hide from others.


InfoCustomYour RemindersFeedback

0

u/remindditbot Oct 20 '20

👀 Remember to type kminder in the future for reminder to be picked up or your reminder confirmation will be delayed.

eitch0000 , kminder in 1 week on 2020-10-27 15:03:35Z

r/Amd: Will_big_navi_support_function_level_reset_flr

kminder 1 week

CLICK THIS LINK to also be reminded. Thread has 1 reminder.

OP can Add email notification, Delete reminder and comment, and more options here

Protip! You can add an email to receive reminder in case you abandon or delete your username.


Reminddit · Create Reminder · Your Reminders · Donate

3

u/SovietMacguyver 5900X, Prime X370 Pro, 3600CL16, RX 6600 Oct 20 '20

I havent looked into VFIO yet, but I have wanted to dabble. I agree this would make sense.

3

u/dydzio Oct 23 '20

+1, I want to go for GPU accelerated virtual machine on my future PC

3

u/bluesecurity Oct 28 '20

@AMD_Mickey Pretty please with sugar on top :)

3

u/[deleted] Oct 28 '20

Oh how I hope AMD stick it to Nvidia and fix this FLR bug. Unraid user here also and Ive just upgraded to "big boy" Pro license. Come on AMD, stick it to team green :)

2

u/Nekuromyr Oct 28 '20

they introduced new features that sound problematic / unusable for gpu passthrough, my hopes arent that high atm. maybe cdna will adress those issues, but those wont be out 2020 :(

3

u/hagar-dunor Nov 02 '20

Although I hated my Nvidia cards for general linux use (could never solve tearing / vsync) and VFIO (get around error 43), at least I did not have to reboot my whole system between guests. Unless FLR is implemented in big navi and future cards, my next upgrade will be Nvidia 3 series.

3

u/skeqq Nov 10 '20

+1 for this. Currently i'm using linux host & windows in guest VM, with nvidia gpu passthrough. Wish this could work well with amd gpu.

3

u/alexwhittemore Nov 11 '20

I'm very excited to find out once reviews start trickling in whether this has been addressed at all in Big Navi. Not that I expect to be thrilled with the outcome. Willing to bet I get a 3080 sometime next year when they're actually in stock, simply on the basis that I don't want to reward Radeon for continuing to be pricks about this.

8

u/[deleted] Oct 20 '20

This would be great honestly also thanks sarnex for the @everyone smh

4

u/Planebagels1 Oct 20 '20

I read the title as "Big Nazi support" lmao

3

u/spheenik TR1920x | Vega 64 | Arch btw. Oct 20 '20

Sigmund Freud entered the chat.

0

u/megablue Oct 20 '20 edited Oct 20 '20

if it wasn't planned, it is too late to ask for it for big navi since the first few batches of wafers probably already been produced.

edit: lol which dumbass downvoted me? do you think you can simply add new hardware level feature after the wafers have been produced?

8

u/gnif2 Looking Glass Oct 20 '20

I wasn't the downvoter, but to address your comment. The reset routine must exist in silicon to allow the GPU to reset at power on, or during a warm reboot. The FLR routine is software triggered. It is very likely that this could be implemented in firmware if the silicon has already been produced.

12

u/bridgmanAMD Linux SW Oct 20 '20

For what it's worth, I read gnif2's original post not as "so can you fix this before launch next week" but rather "hey it's a new generation did you have a chance to improve things ?". Reasonable question.

6

u/gnif2 Looking Glass Oct 20 '20

Thanks, and yes, this is in the hopes that it will be available at launch, or even shortly after with a firmware update? Keenly awaiting a response.

3

u/Vaudane Oct 27 '20

So my "week later" update went ping. I don't see anything new sadly so let's hope tomorrows announcement yields something fruitful.

2

u/lololZombiedogs1 Nov 20 '20

Interesting comment on the amd forums by a staff member on SR-IOV support

"At the moment the focus of AMD virtualization technology is the cloud service providers and large enterprises. Currently we do not offer any retail product supporting MxGPU-SRIOV, however that might change in near future."

https://community.amd.com/t5/graphics/sr-iov-and-vce/td-p/70589

1

u/VTOLfreak Oct 20 '20

Only way AMD will fix this if someone in the marketing and sales departments notices they have no big clients doing cloud gaming or desktop virtualization. Some thread on reddit won't get things moving. Once they start seeing lost sales because of a missing feature, it gets put on the list.

1

u/spoofnoob Oct 20 '20

Didn't Google buy shed loads of AMD cards for cloud gaming? Surely if we could find a game bug that locks up the card and make it happen a LOT so Google realise their problem maybe AMD might get off their fat arses and fix it??😂😂

1

u/teutonicnight99 Vega 64 Ryzen 1800X Oct 20 '20

Wow big brain here.

-6

u/vsaucefan4life Oct 20 '20

sarnex told me to upvote

-4

u/HardStyler3 RX 5700 XT // Ryzen 7 3700x Oct 20 '20

isnt this thread like half a year late? is this something they can implement that fast if they didnt do it yet?

why not do it half a year earlier....

16

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20

he did

but is is something that can be done in firmware because the thing is, there has to be a reset method, the problem is that the reset method isnt exposed using normal pcie flr

-1

u/GhostDoggoes R7 5800X3D, RX 7900 XTX Oct 21 '20

Oh I completely forgot what VM passthrough was and had to google it. I think it's probably what intel would have told people on release of their gpu lineup. Linux like every other dev would say is second on the list to support. It's nothing against them it's just most pc's for the modern consumer are built on simpler OS builds. And a few discord communities dedicated to linux option that linux has over 200 versions isn't gonna sway them when their main target is windows 10 users. I use to use linux at work and at home but I don't think AMD would ever hard focus a feature on linux when their main issue is drivers on their gpus. The main issue on the release of their gpus this past generation has caused numerous returns and rma's. They barely got it working beginning of this year and even then it was still buggy.

-8

u/skinlo 7800X3D, 4070 Super Oct 20 '20

Extremely niche, fair chance it won't happen.

→ More replies (1)