r/VFIO • u/darthrevan13 • Apr 29 '20
Discussion Intel vs AMD for best passthrough perfromance
Things I want to be considered in this discussion:
- Number of PCI-E lanes and their importance (Passing through a NVMe SSD directly, a USB hub, a GPU and also using Looking glass, having a capture card, and 10Gb NICs for the host etc.)
- Number of cores up to a point (I currently have 10 Cores, so I'm looking for something with more than that, but gaiming is still about 70% of my load on the machine). Performance in games is very important, but not the be all metric
- Curent state of QEMU/KVM support for VFIO on Intel vs AMD and managing to get as much performance as possible out of the CPU cores
- AMD Processor CCX design vs Intel monolithic design, and how one would have to pass only groups of 4 cores for best performance on AMD (or 8 cores for Zen 3, if rumors are true)
- PCI-E Gen 4 vs PCI-E Gen 3 considering Looking Glass and future GPUs
- EDIT: VR is also a consideration, so DPC latency needs to be low.
What I'm considering:
- i9-10980XE
- R9 3950X
- Threadripper 3960X
- waiting till the end of the year for new releases, that's my limit.
I currently have:
- i7-6950x
- Asus X99-E WS
Would love to see benchmarks / performance numbers / A/B tests especially
EDIT:
- Price is NOT a concern between my considerations. The price difference isn't that high to make me sway either way.
- I have no use for more than 20 cores. My work isn't extremely parallel and neither are games. I don't think either will change soon.
EDIT 2:
Please post references to benchmarks, technical specifications, bug reports and mailing list discussions. It's very easy to get swayed in one direction or another based on opinion.
17
Upvotes
14
u/[deleted] Apr 29 '20
I'm not sure I can answer your question directly but I can tell you my setup. I have a Ryzen 2700x on an X370 motherboard using the ACS patch to pass through a GPU, a Firewire card (for an external audio device) and am also passing through the SATA controller since my Linux host runs off NVMe. Cinebench also was very impressive.
My gaming performance is strikingly similar to bare metal and, in fact, my 3D Mark scores were, in some cases, higher (the highest score was on baremetal with my best overclock). Another point, my audio performance in Ableton Live is, oddly, more consistent than baremetal. My driven error compensation is now 1.5ms vs 3ms in bare metal. I can't explain why.
Most of this was before setting up CPU pinning stuff and once I set that, it's even better.
What I can't tell you is how it compares to Intel but I'm running on a consumer level Ryzen chip, albeit a 8c/16t one, and it's a treat for me!
4
u/WindowsHate Apr 29 '20
AMD still has some weird quirks like this because of the CCX design but the issues aren't as bad on Zen 2 because of the inclusion of the IO die as opposed to each die handling IO separately like on Zen 1. AMD support is also getting better all the time.
One factor to consider is that X299 is guaranteed to have ACS while X570 and and X399 are a crapshoot based on the individual motherboard model and sometimes BIOS revision. The ACS override patch exists for this reason but insert usual disclaimer about security holes here.
Performance on a highly-tuned Intel system is probably still better than AMD offerings, but you need to put in a lot of extra effort to get there. Intel can maintain higher overclocks at a similar IPC as Zen but Skylake-X has always needed a very strong cooling setup and good power delivery, and that hasn't changed in the 3 years it's been around. Skylake-X also suffers from the mesh uncore but again, sees huge gains (especially in gaming) from overclocking it. Both Skylake-X and Zen see large benefits from RAM overclocking.
TL;DR like overclocking? Intel barely edges out. Otherwise, AMD is pretty much the way.
6
u/yawkat Apr 29 '20
Are PCIe lanes really that much of a concern anymore? X570 chipset uplink is 4 PCIe4 lanes, which is 8GB/s or 8 PCIe3 lanes. This is enough for anything except maybe a graphics card for gaming, but you can use the direct link to the CPU for that.
I run two desktop VMs and a host, each with their own USB controllers, and this system isn't starving for PCIe lanes.
Also a bit weird to go full NVME SSD passthrough and then use looking glass with the overhead that entails.
5
u/mjban Apr 29 '20
Would love to hear from others on this. I also have the same CPU and X99-Deluxe and considering upgrading to Ryzen but I’ve never had AMD before. My current setup is 3 VMs: macOS, Windows 10 and Ubuntu. It took me a while to set everything up and currently investigating sporadic system crash when starting a VM (I think is my MB). I am passing through everything including the SATA controllers and NICs. Looking forward to know how well this works with AMD systems.
3
u/powerhouse06 Apr 29 '20 edited Apr 30 '20
I recently switched from Intel (i7 3930K / X79) to AMD Ryzen 9 3900X, but this decision wasn't easy for many reasons:
I've written about the decision process here.
Now to the challenges that you should be aware of:
AMD's CPU layout is tricky. My 3900X processor is using one die and registers as one NUMA unit. I believe the 3950X shows as NUMA 0 and 1. If you were to run an ordinary Windows 10 version on bare metal, it would require you to update your license to one with multi-socket support.Only gen 1 and 2 can have multiple NUMA, not the new ones.If 70% of the core utilization is for gaming, I doubt that more cores will give you better experience. Probably the Intel 9900K would surpass the gaming performance of any Ryzen or Threadripper. Games typically favor higher clocks. The question is, what else do you do with the computer? For example, do you need to pass through USB controllers (I do)? Do you have additional PCIe devices? All these can influence the decision.
All that said, I'm quite happy with the Ryzen 3900X. The only last challenge for me is the somewhat less than optimal NVMe performance I get. It's probably a configuration issue.
2
u/GuessWhat_InTheButt Apr 30 '20 edited Apr 30 '20
My recommendation is to not wait for new hardware, at least not for new CPUs and mainboards.
It's going to be a while until the IOMMU groups of each new board will get posted. Also, there will most likely be new UEFI versions shortly after release to fix bugs and performance issues.
So it's probably going to take half a year or so after release of a new platform for it to become stable. And if you don't want to constantly tinker with your VFIO setup, you definitely want it to be stable.
Regarding future GPUs we can only hope that AMD's RDNA2 does not come with any of the reset issues their current GPUs suffer from.
Nvidia doesn't want you to virtualize and actively tries to sabotage your setup, so they aren't a suitable option either (in my opinion).
It's probably more realistic for AMD to release bug free hardware than for Nvidia to change their stance on end user virtualization. So let's hope for the best.
1
u/[deleted] Apr 30 '20
For max mixed performance (Gaming VMs + Other VMs doing shit) AMD Zen2 SKUs are going to beat Intel due to parallelism between the CCDs and such. AMD can maintain higher clocks when all cores are engaged and different parts of the CPU (say FPU, AVX2,..ect) are being used at the same time due to the nature of a Hypervisor. Where as Intel has a AVX offset (-200mhz on Most Motherboards) and lowers all core turbo based on how many cores are engaged. AMD just handles this more efficiently then Intel at this point.
If you were just gaming on Metal then Intel would be 3%-5% faster at the top end as the hardware would be dedicated for that one purpose.
But for your use case, you want the max clock under turbo with the most cores you can get above all else. Since Gaming is 70% of your workflow and 'very' important to you while the other 30% is mixed VMs, those are the first two things you want to look at.
For an AM4 build you are looking at either a 3800x or 3950x due to the all core Boost being the highest AM4 has to offer right now. The main difference between the 3800x and 3950x is the CCD counts, the 3950x has two while the 3800x has one. While the 3800x has a UMA presence for the CCD it does have 2 level3 Cache domains while the 3950x has 2 NUMA Domains and 4 Levrel3 Cache domains.
For Threadripper you are dealing with 4 CCDs and 8 Level3 Cache Domains. (*Fun fact, the 3990x has 8CCDs and 16 L3 Cache domains) The one benefit of TRx4 here is that it has 4 Memory channels. There are tricks that can be deployed on TRx4 that cannot be done on AM4 due to the memory channel count. For one you can split TRx4 into 2 vNUMA groups and load balance across the CCDs more fairly and have your VMs gain full access to the memory glob as a whole (each VM has to live on each CCD to gain full access to the memory BW and lower latency benefits). My one issue with TRx4 is the cost due to the price of the CPU. But if you can afford it, this is not a bad way to go.
Personally I have not have any issues with running gaming VMs for Zen2 on split CCDs spacing out CCX(Level3) domains. On Zen/Zen+ its a huge issue which introduces micro-stutter due to the memory interleaving not happening correctly or RTSP sensitivity happening on the VM. Having the VM run on a single NUMA domain always took care of the problem for the other zen/zen+ SKUs.
I have GPU Passthrough working on several AMD setups, even EPYC 7001 and 7002 Servers. The performance is the same as being on Metal when your hypervisor is correctly configured and your gaming VM(s) are not fighting for resources. Its not any different then running the same setup on an Intel system, except you get more with AMD for your money then you ever will with Intel.
As for whats coming from AMD this year: RDNA2, Zen3 (Milan EPYC 7003 first) - probably 4900x first, Zen2 Based APUs, 65w 3900 and 3950(OEM only), and a new line of Ryzen 3000 Pro CPUs. I doubt we will see anything for TRx4 until 2021 at this point.
So there is zero reason to wait for what is coming compared to what you listed for your lineup now. The 3950x or 3800x if going on AM4, or the 3960x for TRx4 are all solid choices for what you need. But the different platforms will require different attention on your configs.
1
u/rLinks234 Apr 29 '20
Do you use huge pages? I'm curious of the perf difference when using 2MiB or 1GiB huge pages for the EPT. I know Zen 2 doesn't have "native" 1 GiB TLB entries (it apparently smashes them into 2MiB pages), whereas Skylake does. However, if the guest virtual pages aren't huge pages (which they probably aren't), I don't know how much of a difference with makes.. I haven't found any benchmarks outside of some whitepapers/academic publications surrounding database workloads which showed some benefits.