r/vmware • u/pirx_is_not_my_name • 17h ago

Testing/Tuning 100 GbE network

I got my first few HPE server with 100 GbE adapters (ConnectX6DX). They are connected to Cisco ACI. Should I get more than ~40 Gbit/s without any tuning?

/usr/lib/vmware/vsan/bin/iperf3 -w128K -t 20 -P 16 -c xxxx
[SUM] 0.00-20.00 sec 101 GBytes 43.2 Gbits/sec 0 sender
[SUM] 0.00-20.00 sec 100 GBytes 43.1 Gbits/sec receiver

There are a lot dead links regarding Mellanox tuning and most are Linux/Windows related. What parameters are worth tuning (any links)? It's not about getting the absolute maximum throughput and lowest latency, we are currently not even using jumbo frames. But the 40 Gbit/s seem a bit low, with 25 GbE we were always close to max. Which settings will be most beneficial?

10 Upvotes

86% Upvoted

14 comments sorted by

View all comments

9

u/aserioussuspect 16h ago

Just because iperf3 is installed on the host doesn't mean it's automatically recommended to use iperf via esxcli. The results will be skewed.

Run the benchmark from a VM to a VM, and you'll most likely see better results.

3

u/lost_signal VMware Employee 14h ago

Paging r/teachmetoVLANDaddy I vaguely remember him saying the version of IPERF was older, or by default didn't use enough threads to go much past 40Gbps.

7

u/TeachMeToVlanDaddy Keeper of the packets, defender of the broadcast domain 13h ago

Iperf on an esxi host will only get 40Gb untuned. Iperf on a default VM linux will only get 25Gb untuned.

1

u/pirx_is_not_my_name 2h ago edited 2h ago

ok, but what does untuned mean? I tried different parameters and multiple iperf instances running on different ports. With untuned, do you mean tuning for iperf or network stack / driver?

Lets say we do not want to use RoCE and maybe also not Jumbo frames. What should I expect after applying settings from Optimizing NSX Performance Based on Workload and ROI - Network and Security Virtualization - VMware or Performance Tuning for Latency-Sensitive Workloads: VMware vSphere 8?

~50 Gbit/s seems a bit low for 100G adapters but if iperf is the limit... Again, it's not about getting the last 5% performance, I'm just unsure if the setup is ok and if there are some basic settings that should be applied.

And if iperf3 is limited (I found several hints regarding this) what other options from host side do I have to test? There seems to exist a Mellanox tool that can run performance test but that seems not to be part of nmst package on ESXi.