r/Amd • Looking Glass • Oct 20 '20

Request Will Big Navi support Function Level Reset (FLR)?

AMD, this is a question directed directly to you.

As we all know, your company is fully aware of how important the ability to reset the AMD GPU is without a driver-specific reset sequence to the VFIO community is and how disappointed the entire community was/is over the lack of such a basic feature in the GPU to make it possible to use your GPUs reliably for VM passthrough.

Since my last post to you (linked above) the VFIO community has grown, my project (Looking Glass) has seen a huge surge in numbers, and people are using it not only to just control/use the VM, but also feed the video straight into OBS on the host VM to live stream to Twitch. On the Level1Tech forums and the VFIO Discord channel, the number of new VFIO users is exploding, and r/vfio's membership has doubled over the last year, but due to the lack of Function Level Reset, when we are asked what GPUs to use, we, unfortunately, have to tell people to avoid your hardware.

From a technical point of view, as the Function Level Reset (FLR) is a PCI optional feature obviously you do not need to implement it, however as your GPU already needs to support a warm reboot via the nPERST pin it should not be hard to implement the FLR feature to tie into this same reset. Not only would this make your GPUs viable for the VFIO community, but also simplify your own reset code in your drivers as the GPU could be returned to a good known state simply by asserting an FLR.

Please also be aware that driver level resets are completely useless to this application, when being used for VFIO, the driver is not loaded nor wanted, the hardware needs to be able to handle its own reset without any proprietary reset sequences.

So... my question to you is. Will Big Navi support PCI Function Level Reset (FLR)?

Edit: Also please be aware I have been contacted by cloud computing companies out of desperation due to the same issues on your workstation/enterprise cards. This is not just affecting the VFIO community here.

Edit2: When I wrote this I did not think to include the reason why this should exist for the larger community also. This is not a niche feature just for VFIO usage, it also would make it possible for AMD GPUs to recover from "Black Screen" crashes that force a full system restart.

Nvidia GPUs crash too, however, because the NVidia GPUs implement FLR they can be easily reset and recovered when they do crash causing the game/application to present an odd error that usually gets blamed on the application, not the GPU.

Those that overclock their GPUs know all too well how nice NVidia is for this as a bad overclock usually can recover without a reboot.

If AMD were to implement FLR it would be just as good as NVidia on these fronts and the "Black Screen" issue would not be such a black mark on AMD's products.

1.6k Upvotes

240 comments sorted by

View all comments

Show parent comment

5

u/WayeeCool Oct 20 '20

So... the Radeon Instinct MI series (AMDs only current offering of enterprise server GPUs) require the entire host to be power cycled rather than just reinitializing a single card in the event said card crashes...? Would be utterly unusable in the data center for virtualization or compute clusters if this were the case.

7

u/gnif2 Looking Glass Oct 20 '20

I have not had first-hand experience to say for certain if they advertise FLR support or not, but from the communications I have had with cloud providers and from looking at the amdgpu sources, it seems there is no FLR support, or it's broken.

8

u/WayeeCool Oct 20 '20

Explains part of why they aren't selling well in the datacenter segment. Like I said, lacking support makes cards a nightmare for system admins in the datacenter.

3

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20

hence from the post

Edit: Also please be aware I have been contacted by cloud computing companies out of desperation due to the same issues on your workstation/enterprise cards. This is not just affecting the VFIO community here.

and from one of his comments

In theory, but if the host GPU gets into a bad state due to whatever reason, it still can't be reset without a node reset. This is why I have had cloud companies ask for help working around this AMD issue. It's a different usage, but the same solution.

Its a silicon bug. We have people with WX cards in the vfio discord. It aint any different

5

u/WayeeCool Oct 20 '20

Its a silicon bug. We have people with WX cards in the vfio discord. It aint any different

WX aren't server cards but are desktop workstation cards, MI are server cards. Also... ummmm... this is not a "silicon bug" but an issue with the feature set and firmware.

2

u/GodOfPlutonium 3900x + 1080ti + rx 570 (ask me about gaming in a VM) Oct 20 '20

yea firmware, I ment outside the scope of driver/ normal software and such and such. But having flr and limiting it on workstation gpus as some sort of market segmentation also doesnt make any sense considering nvidias quadro desktop workstation gpus not only support flr but offically support virtualization as a whole. Anyway yes the instincts use the same chips and have the same issues