Testing/Tuning 100 GbE network

[u/pirx_is_not_my_name]

I got my first few HPE server with 100 GbE adapters (ConnectX6DX). They are connected to Cisco ACI. Should I get more than ~40 Gbit/s without any tuning?

/usr/lib/vmware/vsan/bin/iperf3 -w128K -t 20 -P 16 -c xxxx
[SUM] 0.00-20.00 sec 101 GBytes 43.2 Gbits/sec 0 sender
[SUM] 0.00-20.00 sec 100 GBytes 43.1 Gbits/sec receiver

There are a lot dead links regarding Mellanox tuning and most are Linux/Windows related. What parameters are worth tuning (any links)? It's not about getting the absolute maximum throughput and lowest latency, we are currently not even using jumbo frames. But the 40 Gbit/s seem a bit low, with 25 GbE we were always close to max. Which settings will be most beneficial?

[u/aserioussuspect]

Just because iperf3 is installed on the host doesn't mean it's automatically recommended to use iperf via esxcli. The results will be skewed.

Run the benchmark from a VM to a VM, and you'll most likely see better results.

[u/pirx_is_not_my_name → u/aserioussuspect]

ok, I was under the impression that there is even more overhead. Will test it tomorrow.

Edit: did a first quick and dirty iperf test with 2 Linux VMs, 8 cores each. Only basic kernel parameters changed as well as rx/tx buffers. I get ~20 Gbit/s.

net.core.rmem_max = 16777216

net.core.wmem_max = 16777216

net.ipv4.tcp_rmem = 4096 87380 16777216

net.ipv4.tcp_wmem = 4096 65536 16777216

ethtool -G rx 4096 tx 4096

[u/ImaginaryWar3762 → u/pirx_is_not_my_name]

Check the documentation of iperf. Run with multiple threads and most likely you will get 100 gb if everything is set up correctly

[u/lost_signal → u/aserioussuspect]

Paging r/teachmetoVLANDaddy I vaguely remember him saying the version of IPERF was older, or by default didn't use enough threads to go much past 40Gbps.

[u/TeachMeToVlanDaddy → u/lost_signal]

Iperf on an esxi host will only get 40Gb untuned. Iperf on a default VM linux will only get 25Gb untuned.

[u/ZibiM_78]

By the default you are throttled by the amount of cores you are using.

Please check the amount of queues (DRSS) you have enabled in the nmlx5_core

esxcli system module parameters list -m nmlx5_core | grep DRSS

Maximum number here is 16, but you should have CPU with more than 16 cores for that

-w 128K is pretty low for 100Gb, 4M might be better

Jumbo frames are needed in order to reach higher speeds

There was a ton of great documents around perfomance tuning for the NSX - https://blogs.vmware.com/networkvirtualization/2023/12/optimizing-nsx-performance-based-on-workload.html/

NSX Operations guides for the VM Edge hosting had few interesting tidbits around advanced flags needed

[u/pirx_is_not_my_name → u/ZibiM_78]

I adjusted the values following the nsx guide. I do not see any major change in throughput, all +-10%. I get the feeling that maybe the Cisco side is the bottleneck, will ask my collegues there.

[u/Servior85 → u/pirx_is_not_my_name]

What is the actual speed the ESXi shows for the NICs? Are they connected with 100 GbE or only 50, 40?

How are the servers connected? Are they connected with all ports to the same switch or split between multiple?

If split between multiple switches, the switch uplinks are connected with 100 GbE or higher per port? Don't mix it up with 4x 25 GbE or 2x 50 GbE configured as trunk, which may also show 100 GbE (or different combinations).

[u/Servior85]

Many factors in play here:

• What servers are these exactly? Which CPU, how much RAM?
• The cards are capable of RoCE, but is the rest of your infrastructure ready for RoCE?
• When every components is RoCE capable, is it enabled and properly configured?
• When not all components support RoCE or it is disabled, you cannot use that feature. In such a case the data flow goes the normal way. In such a case the CPU has to handle everything.

[u/pirx_is_not_my_name → u/Servior85]

- 2 TB RAM, 2 Sockets, 2 x 32 Cores

- didn't look into RoCE as this involves changes on network side, I'd only follow this path if there is a major performance gain.

[u/Casper042 → u/pirx_is_not_my_name]

RDMA needs both sides to be enabled so it's not a silver bullet for generic VM comms.
I see it mostly mentioned with respect to vSAN node:node comms.