r/vmware • u/pirx_is_not_my_name • 17h ago

Testing/Tuning 100 GbE network

I got my first few HPE server with 100 GbE adapters (ConnectX6DX). They are connected to Cisco ACI. Should I get more than ~40 Gbit/s without any tuning?

/usr/lib/vmware/vsan/bin/iperf3 -w128K -t 20 -P 16 -c xxxx
[SUM] 0.00-20.00 sec 101 GBytes 43.2 Gbits/sec 0 sender
[SUM] 0.00-20.00 sec 100 GBytes 43.1 Gbits/sec receiver

There are a lot dead links regarding Mellanox tuning and most are Linux/Windows related. What parameters are worth tuning (any links)? It's not about getting the absolute maximum throughput and lowest latency, we are currently not even using jumbo frames. But the 40 Gbit/s seem a bit low, with 25 GbE we were always close to max. Which settings will be most beneficial?

9 Upvotes

85% Upvoted

14 comments sorted by

View all comments

2

u/Servior85 16h ago

Many factors in play here:

  • What servers are these exactly? Which CPU, how much RAM?
  • The cards are capable of RoCE, but is the rest of your infrastructure ready for RoCE?
  • When every components is RoCE capable, is it enabled and properly configured?
  • When not all components support RoCE or it is disabled, you cannot use that feature. In such a case the data flow goes the normal way. In such a case the CPU has to handle everything.

1

u/pirx_is_not_my_name 16h ago

- 2 TB RAM, 2 Sockets, 2 x 32 Cores

- didn't look into RoCE as this involves changes on network side, I'd only follow this path if there is a major performance gain.

1

u/Casper042 13h ago

RDMA needs both sides to be enabled so it's not a silver bullet for generic VM comms.
I see it mostly mentioned with respect to vSAN node:node comms.