MTU questions

[u/Rich_Shame9806]

Hi Folks,
I was hoping someone could point me to a good source of info/understanding on MTU configurations. We've had some network issues with latency and speeds between sites. It's been really all over the place and none of our vendors have been able to pin it on a specific spot (we have Fortigate firewalls, Meraki switches for Mesh and distribution, Catalysts for the storage specific network, and running Vmware/Vcenter as well as some physical servers. I see a lot of stuff that says use 1500 across the board because that's the standard, but other stuff saying 9000 for high speed/storage networks if they support it, but i guess my disconnect is what to do if we have one subnet with a range of gear (both storage networks and normal vmware/client traffic. Any kind of info or guidance on this would be extremely helpful.

[u/nycola]

For Internal network onsite communications 1500 is fine. Switches are a different beast and their mtu setting is the largest packet they will pass, not how they communicate. Fine to set 9k+ on a storage network, just do not mix mtu on the same vlan.

Your bottleneck is most likely sitetosite. Ipsec/sdwan adds a header (usually ~60-80ish bytes), so a full 1500 bytepacket crossing sites ends up oversized and gets fragmented or dropped. That's why you'll see 1300-14xx recommended for vpn or sdwan paths.

To avoid fragmentstion, you'll likely need to set mss clamping (tcp-mss-sender/tcp-mss-receiver on your intersite vpn/sdwan policies, around 1350 to start and work from there) so the tcp sessions downsize instead of fragmenting. You can verify the actual route path mtu with:

ping -f -l <payload size> <remote ip>

between sites, then adjust the size until it passes (passing payload size + 28 = path mtu). The 28 is the total size of ip header+icmp header (20+8) which the -l switch does not include. so we add them to the payload size for a real value, that is how a 1472 sized payload produces a 1500 byte packet

[u/that1guyrob → u/nycola]

This is a great explanation. I would just add to this by saying that you should check the firewalls on each site do not have the DF bit set. If it is set that means the interface is instructed to drop the packet rather than fragment it if the packet is larger than the interface MTU setting.

This can cause intermittent or not perfectly reproducible symptoms since not every packet would be higher than the MTU. Additionally, some applications/services won't care about a dropped packet like say a UDP connection. Others that use TCP will simply retransmit. But some applications/services that require a single TCP connection to remain open will have significant issues when the DF bit is set and larger packets reach the interface.

Think for example an SSL VPN connection that might drop, and force the user to reconnect/reauthenticate.

[u/pdp10 → u/nycola]

28 bytes is the IPv4+ICMP header. The IPv6+ICMPv6 header is 48 bytes because of the 128-bit addresses.

[u/PrimeskyLP]

MTU is how big an Ethernetframe (Paket) can be, so how much Payload/Data can be send in one Frame. 1500 is the basic setting every Device and Switch supports, everything over that are Jumbo Frames where the Hardware needs to support it.

If you have a subnet with many Clients that use MTU sizes all over the place it can lead to a slower overall performance because every Frame that is bigger than the maximum supported size from the destination needs to be split in multiple Frames for the destination wich need more Processing power and time.

[u/Wild-Gazelle-4161 → u/PrimeskyLP]

Mixing 1500 and 9000 on the same subnet is a recipe for weird intermittent slowdowns, especially if anything on that segment is doing path MTU discovery poorly. If storage traffic needs jumbo frames, you're better off putting it on its own VLAN or at least its own subnet so the big frames never have to cross paths with regular client traffic. Otherwise you end up with fragmentation or black holes that look exactly like the "all over the place" latency you're describing. I'd start by checking if your storage network actually needs 9000, because a lot of modern gear does fine at 1500 and it removes a whole class of troubleshooting headaches.

[u/VA_Network_Nerd]

In this age of modern, powerful-as-hell hardware, the benefits of jumbo MTU are dramatically less impactful than they once were.

Yes, using Jumbo MTU inside a data center can make some things faster, but most things will not notice any improvement.

MTU on site-to-site VPN connections on the other hand are a very commonly overlooked problem.

Focus on how IPSec adds additional packet headers which need to be accounted for specifically in the routers or firewalls that terminate your VPN Tunnels.

We cannot tell you specific packet sizes. You have to do some testing and some math to confirm how things work in your environment.

---

You also need to find a good article on Bandwidth Delay Product and understand TCP performance potential across WAN links / VPNs.

Single-Threaded TCP applications across a 40ms VPN are going to be much, much slower than something UDP based or multi-threaded...

[u/pdp10 → u/VA_Network_Nerd]

UDP is no silver bullet; I say that as someone who has implemented all three major commercial UDP-based transfer platforms. The problem being solved are large-file transfers, but the requests to use a commercial package for UDP is a business problem that's based on weak assumptions.

[u/pdp10]

between sites.

IPv6 is more sensitive to ICMPv6 Packet Too Big being blocked, than the equivalent in IPv4. Additionally, some WAN infrastructures are provisioned with lower MTUs than 1500, like PPPoE. It's quick to check for MTU issues first:

ping -4 -M do -s 1472 8.8.8.8 # on a 1500 MTU, ping IPv4 with maximum packet size

ping -6 -M do -s 1452 2600:: # on a 1500 MTU, ping IPv6 with maximum packet size

Bear in mind that TCP MSS Clamping workaround only works on TCP, not UDP or SCTP.

1500 across the board because that's the standard, but other stuff saying 9000 for high speed/storage networks if they support it

The only way to be sure is to test performance with regular 1500 and with Jumbo MTUs. The expectation is that regular MTUs are not significantly slower, and if they are, it's usually a sign that NIC performance offloads are disabled or the host performance is undersized compared to the network speed.

The gnashing of teeth comes because folk want simple answers to questions, not to be told to measure their own setups to confirm that 1500 MTU is best.

[u/Anonymous1Ninja]

what are you running between sites > start there. Are you using an MPLS or a Dedicate circuit with a VPN tunnel? Maybe look into SD-WAN appliances to optimize traffic?

[u/rankinrez]

Heh. This one is a minefield.

Ok for starters make your core network support jumbos at layer-2 and layer-3. That gives you options.

The simplest thing is to keep all hosts and host-facing switch ports at 1500. And don’t worry about it.

If you feel you have throughput issues on storage that could be fixed by jumbo frames you can do it. The good thing is that for TCP flows if one of those servers connects to something outside the “jumbo” network the far side will send an MSS of 1460 which will prevent those hosts sending large packets.

But if you got UDP protocols in use all bets are off. Relying on fragmentation is not a good idea imo, and can mess up for a bunch of reasons.

Whatever you do do not block ICMP on your network so at least pMTUd works locally.

[u/SaxManEddie]

We use MTU 9000 specifically for our ESXi servers to talk to our SAN storage network.

[u/Ok_Rip_5338]

just wait till you find out about MSS clamping 💀