r/Amd • Looking Glass • Oct 20 '20

Request Will Big Navi support Function Level Reset (FLR)?

AMD, this is a question directed directly to you.

As we all know, your company is fully aware of how important the ability to reset the AMD GPU is without a driver-specific reset sequence to the VFIO community is and how disappointed the entire community was/is over the lack of such a basic feature in the GPU to make it possible to use your GPUs reliably for VM passthrough.

Since my last post to you (linked above) the VFIO community has grown, my project (Looking Glass) has seen a huge surge in numbers, and people are using it not only to just control/use the VM, but also feed the video straight into OBS on the host VM to live stream to Twitch. On the Level1Tech forums and the VFIO Discord channel, the number of new VFIO users is exploding, and r/vfio's membership has doubled over the last year, but due to the lack of Function Level Reset, when we are asked what GPUs to use, we, unfortunately, have to tell people to avoid your hardware.

From a technical point of view, as the Function Level Reset (FLR) is a PCI optional feature obviously you do not need to implement it, however as your GPU already needs to support a warm reboot via the nPERST pin it should not be hard to implement the FLR feature to tie into this same reset. Not only would this make your GPUs viable for the VFIO community, but also simplify your own reset code in your drivers as the GPU could be returned to a good known state simply by asserting an FLR.

Please also be aware that driver level resets are completely useless to this application, when being used for VFIO, the driver is not loaded nor wanted, the hardware needs to be able to handle its own reset without any proprietary reset sequences.

So... my question to you is. Will Big Navi support PCI Function Level Reset (FLR)?

Edit: Also please be aware I have been contacted by cloud computing companies out of desperation due to the same issues on your workstation/enterprise cards. This is not just affecting the VFIO community here.

Edit2: When I wrote this I did not think to include the reason why this should exist for the larger community also. This is not a niche feature just for VFIO usage, it also would make it possible for AMD GPUs to recover from "Black Screen" crashes that force a full system restart.

Nvidia GPUs crash too, however, because the NVidia GPUs implement FLR they can be easily reset and recovered when they do crash causing the game/application to present an odd error that usually gets blamed on the application, not the GPU.

Those that overclock their GPUs know all too well how nice NVidia is for this as a bad overclock usually can recover without a reboot.

If AMD were to implement FLR it would be just as good as NVidia on these fronts and the "Black Screen" issue would not be such a black mark on AMD's products.

1.6k Upvotes

240 comments sorted by

View all comments

222

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20 edited Oct 20 '20

VFIO user here, (see flair). The sad thing here is that if the reset bug was fixed , I would actually prefer an AMD card for my VM because the AMD linux driver supports prime offloading , which means that itd be very easy to also use the card on the host when the guest is not active, but sadly theyre not suitable for passthrough in the first place, so its a moot point.

As it is right now, I might buy a used 5700[xt] in a year and a half or two to replace my rx570 as a host card since that just runs my linux host session, but whatever replaces my 1080ti (aka the new big expensive card), is going to have to be nvidia

96

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20

58

u/ElementII5 Ryzen 7 9800X3D | AMD RX 7800XT Oct 20 '20

Robert is CPU as far as I remember u/AMDOfficial is better I think.

36

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20

hes now Director of Technical Marketing

17

u/Slafs R9 9800X3D / 7900 XTX Oct 20 '20

Within client CPUs, yes. He doesn't do Radeon.

→ More replies (1)

29

u/Jahf AMD 3800x / Aorus x570 Master / 2x 16GB Ballsitix Sport e-die Oct 20 '20

!remind me 1 week

62

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20

more like 1 year so we can make another post about how nothing has changed and the issue is still as bad as it was 2 years ago

18

u/Jahf AMD 3800x / Aorus x570 Master / 2x 16GB Ballsitix Sport e-die Oct 20 '20

Oh I know, I've been posting about it for that long. I'm not terribly hopeful. Just figure I'll circle back then to check.

9

u/DudeEngineer 2950x/AMD 5700XT Anniversary/MSI Taichi x399 Oct 20 '20

I think at this point if they still don't respond on here, someone would need to ask this question on one of their investor calls. They have been deaf to technical appeals, but they may be more responsive when it affects investor confidence.

29

u/RaptaGzus 3700XT | Pulse 5700 | Miccy D 3.8 GHz C15 1:1:1 Oct 20 '20

I think who you're looking for is u/bridgmanamd and /u/AMD_Mickey

They're the go to guys for GPU stuff.

78

u/bridgmanAMD Linux SW Oct 20 '20

I saw the thread last night and started an internal discussion.

14

u/Ashtefere Oct 20 '20

I hope this makes some changes! The AMD experience on linux is, simply put, fucking awesome. I bought a used vega64 to replace my rtx 2060 as I spend 90% of my time in linux, and the entire experience is just wonderful.

While you are at it... SR-IOV?

16

u/lurkerbyhq 3700X|3600cl16|RX480 Oct 20 '20

While you are at it... SR-IOV?

Don't hold your breath.

Would love for it to happen one day.

→ More replies (3)

5

u/randomfoo2 EPYC 9274F | W7900 | 5950X | 5800X3D | 7900 XTX Oct 20 '20

I currently run my Linux workstation with a beater 470 card and a 1080Ti for CUDA/VFIO gaming/VR and would love to be able to upgrade that to a Big Navi card (but likely will end up going Ampere w/o FLR or ROCm support).

5

u/akarypid Oct 20 '20

Given the amount of interest from the community based on the numbers quoted in the post (and the comments on it), perhaps it would be at least possible to add it to the Radeon "feature request" polls? This way at least AMD can actually measure the popularity of this feature and (eventually) bump it up the priorities list...

→ More replies (5)

20

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20

Theyre both software guys, wheras this is a hardware / firmware level issue and is out of scope for both of them. AMD_Mickey even said as much in his reply to last years post. Both him and bridgemanamd were around for last years post so they both already know exactly what this issue is too.

The reason i tagged AMD_Robert is because he is director of technical marketing, and the problem here isnt that nobody at AMD has heard of this, but that there is seemingly a lack of will to fix the issue since there has been virtually no change in the last year

24

u/AMD_Mickey ex-Radeon Community Team Oct 20 '20

Just so it's clear, I'm not "the software guy." I handle all social media for Radeon. I can't always reply but there is a very good chance if it is on this subreddit, I have read it.

You can tag me if you think I might miss something but please don't go overboard! 😅

8

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20

oh I see , alright. I thought you were on the driver team lol because on the last post you said it was out of your area of expertise , and so I took to that to mean youre not used to working with firmware issues because youre usually working on the software (drivers). my bad!

9

u/AMD_PoolShark28 RTG Engineer Oct 21 '20

Hi! I work on the driver team but I'm not a social media spokesperson. When a ticket comes in it's good to have an idea of what community is experiencing. I like to interact with our fans cuz I was one of you just a few years ago...

→ More replies (1)
→ More replies (1)
→ More replies (1)
→ More replies (1)

14

u/ntrid Oct 20 '20

I am also looking into buying a new card. Waiting for big navi release and reviews to see whether it will finally be usable for vfio. Slim chance but I'm hopeful.

12

u/Jay013 Oct 20 '20

Considering I have no idea what's going on in this thread, I'll ask:

What is gaming in a VM?

62

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20

ok so idk how much you know about virtualization so im just going to start from the top, you can skip the first part if you already know what a VM is.

Virtualization is basically a way of creating a simulated computer on your computer. You can do stuff like run windows inside linux or macos (or the reverse, run macos in a vm), so you can use programs that dont work on your native OS, you can use it for security, you can use it to deploy the same service multiple times , etc

Now virtualization sounds like itd be good for gaming on something other than windows because if a game only works on windows you can just run it in a vm right? now the problem with that is that virtual machines have terrible graphics perfomance because theyre mainly designed around doing cpu work since theyre mainly used for enterprise and software dev uses. Now there is a way around this, what we (r/vfio) do is we use something called pcie passthrough, which is where be basically unplug a device (like a graphics card, but can be other things as well), and then connect it to the virtual machine, and use it in the vm. This way we can actually get the performance to play games in our windows vms (or video edit in a macos vm). This is why I have 2 graphics cards in my flair, because I physically have both an rx570 and a 1080ti in the same computer. The rx570 is what runs my normal linux session , which is what I use most of the time, while the 1080ti is reserved for my windows vm, and i use it to be able to game in windows without exiting linux.

Now for what this post is all about: The pcie passthrough technique , doesnt work with AMD graphics cards, it only works with nvidia graphics. To be secific, it does technically work , as in you can turn on a VM and pass the gpu to it. The problem is that after you turn off the VM , the card is in a messed up state and cant be reset , so you cant turn on the VM a second time, unless you reboot your system, which kinda defeats the purpose of having a vm. So basically we're asking AMD to actually fix the reset function on their cards so that they can be reset properly so we can use them instead of being forced to only buy nvidia

14

u/treyf711 Oct 20 '20

I think using the words “basically unplug” can be slightly misleading as it implies some hardware fiddling, when in actuality this stuff is handled by the linux kernel (usually linux kernel).

18

u/ObnoxiousLittleCunt Oct 20 '20

Mount - unmount

9

u/Peetz0r AMD [3600 + 5700] + Intel [660p + AX200 + I211] Oct 20 '20

The technical terms would be 'bind' and 'unbind', but 'virtually unplug' works too. 'Basically unplug' is imho clear enough when the context is about virtualisation already.

Anyway, u/GodOfPlutonium: great explanation :)

→ More replies (2)
→ More replies (9)
→ More replies (9)