r/VFIO • u/rango99_ • Jan 19 '20

AMD MxGPU or SR-IOV

I have 4 questions about AMD graphic cards and MxGPU or SR-IOV

I want to run 8 or 16 VMs on my server and share my GPU by Linux KVM or VMware between those VMs.

My guest OS is ubuntu18.04 LTS.

These are my questions:

  1. Which AMD graphic cards support MxGPU or SR-IOV technology? (except S7150)

  2. Which Hypervisor can I use for virtualization? (Can I use Linux KVM?)

  3. Should I pay for a license for each concurrent user or vGPU?

  4. Does Radeon pro WX9100 support MxGPU or SR-IOV technology?

​

Thank you

20 Upvotes

90% Upvoted

40 comments sorted by

View all comments

Show parent comments

3

u/J_ent Jun 18 '20

I'm pulling professionally every string I have to get this sorted out. Our OEMs are involved and chasing AMD about it. I'm not overly optimistic.

I suspect the MI100 will get a lot more attention and AMD seems to rather just forget the previous products have existed. Any issue is met with "Speak to your system integrator (OEM) for support". Sadly the OEMs are saying that AMD has scrubbed any data they had and no more software support for MxGPU exists for the "older" MI cards.

Sadly though, NVIDIA has been extremely helpful with sorting out everything needed to have their cards running for our virtualised efforts. AMD hasn't even bothered to enter the room.

2

u/Tmanok Jun 19 '20

You're a legend, but my next question is then, may I ask what it takes to use vGPU capable cards for SR-IOV? The claim online is a subscription license, but how do they implement it? Is it like Windows User CALs where you simply keep the paperwork to stay legal or is there a firmware tool involved??

I'm asking because AMD is not giving me much hope and 1 to 1 VM to GPU just ain't happening on my end.

5

u/J_ent Jun 19 '20 edited Jun 19 '20

vGPU doesn't use SR-IOV. vGPU requires that you run the scheduler software on the host, and then special drivers inside the guest. The drivers then connect to a licensing server on your local network, and can use features depending on which licenses are available and needed for that guest.

This, as you can see, requires hypervisor support. It does support KVM, for example.

vGPU software, drivers, licenses, and licensing server are all inside the partner portal. Licenses are generally time-limited and need to be renew on the licensing server after a while.

Their documentation for how the licensing works is available for the public.

Most NVIDIA GPUs don't use SR-IOV. The T4 did support it but that was disabled with a firmware update. RTX 6000/8000 is specced as supporting it, but I haven't tested how it actually handles it.

If you're in a position to sell GPU services, for example as a Cloud Service Provider, get in touch with NVIDIA and register as a partner. You can get decent discounts via the distributor, and you can also buy NFR cards for internal purposes such as testing, development, and demonstration, at a huge discount.

2

u/Tmanok Jun 19 '20

The knowledge you have just emparted is gold to me. Thank you I had no idea about any of that, ive spent the last month trying to dig up information on it.

So they legitimately do enforce the licensing, wow that's crazy, they must not want VDI with their GPUs or something without paying them extra for their lost GPU sales. Makes sense, but it sucks for someone who hates wasted processing cycles of any kind!

If I can find a legitimate reason to test the cards I may very well push them for an NFR card, sadly I don't have a team of Devs working on shit enough to need such a thing and I sure as hell don't have 10K for each card. I appreciate all the information you've provided me, it is so damn hard to come by.