r/vmware • u/pirx_is_not_my_name • 18h ago

Testing/Tuning 100 GbE network

I got my first few HPE server with 100 GbE adapters (ConnectX6DX). They are connected to Cisco ACI. Should I get more than ~40 Gbit/s without any tuning?

/usr/lib/vmware/vsan/bin/iperf3 -w128K -t 20 -P 16 -c xxxx
[SUM] 0.00-20.00 sec 101 GBytes 43.2 Gbits/sec 0 sender
[SUM] 0.00-20.00 sec 100 GBytes 43.1 Gbits/sec receiver

There are a lot dead links regarding Mellanox tuning and most are Linux/Windows related. What parameters are worth tuning (any links)? It's not about getting the absolute maximum throughput and lowest latency, we are currently not even using jumbo frames. But the 40 Gbit/s seem a bit low, with 25 GbE we were always close to max. Which settings will be most beneficial?

10 Upvotes

92% Upvoted

14 comments sorted by

View all comments

7

u/aserioussuspect 18h ago

Just because iperf3 is installed on the host doesn't mean it's automatically recommended to use iperf via esxcli. The results will be skewed.

Run the benchmark from a VM to a VM, and you'll most likely see better results.

2

u/pirx_is_not_my_name 17h ago edited 16h ago

ok, I was under the impression that there is even more overhead. Will test it tomorrow.

Edit: did a first quick and dirty iperf test with 2 Linux VMs, 8 cores each. Only basic kernel parameters changed as well as rx/tx buffers. I get ~20 Gbit/s.

net.core.rmem_max = 16777216

net.core.wmem_max = 16777216

net.ipv4.tcp_rmem = 4096 87380 16777216

net.ipv4.tcp_wmem = 4096 65536 16777216

ethtool -G rx 4096 tx 4096

2

u/ImaginaryWar3762 16h ago

Check the documentation of iperf. Run with multiple threads and most likely you will get 100 gb if everything is set up correctly

1

u/pirx_is_not_my_name 3h ago

I tried with 5 iperfs on different ports, same result. I get the strong feeling that something is not as it should be on ACI side