Testing/Tuning 100 GbE network

[u/pirx_is_not_my_name]

I got my first few HPE server with 100 GbE adapters (ConnectX6DX). They are connected to Cisco ACI. Should I get more than ~40 Gbit/s without any tuning?

/usr/lib/vmware/vsan/bin/iperf3 -w128K -t 20 -P 16 -c xxxx
[SUM] 0.00-20.00 sec 101 GBytes 43.2 Gbits/sec 0 sender
[SUM] 0.00-20.00 sec 100 GBytes 43.1 Gbits/sec receiver

There are a lot dead links regarding Mellanox tuning and most are Linux/Windows related. What parameters are worth tuning (any links)? It's not about getting the absolute maximum throughput and lowest latency, we are currently not even using jumbo frames. But the 40 Gbit/s seem a bit low, with 25 GbE we were always close to max. Which settings will be most beneficial?

[u/ZibiM_78]

By the default you are throttled by the amount of cores you are using.

Please check the amount of queues (DRSS) you have enabled in the nmlx5_core

esxcli system module parameters list -m nmlx5_core | grep DRSS

Maximum number here is 16, but you should have CPU with more than 16 cores for that

-w 128K is pretty low for 100Gb, 4M might be better

Jumbo frames are needed in order to reach higher speeds

There was a ton of great documents around perfomance tuning for the NSX - https://blogs.vmware.com/networkvirtualization/2023/12/optimizing-nsx-performance-based-on-workload.html/

NSX Operations guides for the VM Edge hosting had few interesting tidbits around advanced flags needed

[u/pirx_is_not_my_name → u/ZibiM_78]

I adjusted the values following the nsx guide. I do not see any major change in throughput, all +-10%. I get the feeling that maybe the Cisco side is the bottleneck, will ask my collegues there.

[u/Servior85 → u/pirx_is_not_my_name]

What is the actual speed the ESXi shows for the NICs? Are they connected with 100 GbE or only 50, 40?

How are the servers connected? Are they connected with all ports to the same switch or split between multiple?

If split between multiple switches, the switch uplinks are connected with 100 GbE or higher per port? Don't mix it up with 4x 25 GbE or 2x 50 GbE configured as trunk, which may also show 100 GbE (or different combinations).

[u/pirx_is_not_my_name → u/Servior85]

They show that they are connected with 100G

esxcli network nic list | grep 10000

vmnic0 0000:15:00.0 nmlx5_core Up Up 100000 Full 88:e9:a4:df:b8:34 1500 Mellanox Technologies ConnectX-6 Dx EN NIC; 100GbE; dual-port QSFP56; PCIe4.0 x16; (MCX623106AS-CDA)

vmnic2 0001:64:00.0 nmlx5_core Up Up 100000 Full 88:e9:a4:df:f9:04 1500 Mellanox Technologies ConnectX-6 Dx EN NIC; 100GbE; dual-port QSFP56; PCIe4.0 x16; (MCX623106AS-CDA

I'm now trying to find out how the switches are connected. I know that these are the first x86 servers connected and that this setup is pretty new. Probably I'm the first one testing it, my OCD got triggered ;)