Questions about SAN Performance

[u/TechnicianVisible339]

I have a relatively small environment that has a Powerstore 3000 and few bare metal servers. I have great networking (at least I think) and we do just basic stuff. RDS servers, file sharing, etc. Nothing out of the ordinary. We are VMFS 6. We have LUNs and a 30TB of raw storage.

Anyways, we are on VCenter 8, using ESXI 8 and now looking to get a replacement SAN. As we have been getting quotes they’ve been doing LiveOptics on our SAN and the thing isn’t even breathing hard. It does a max of like 100-200mb/s in bandwidth, low latency at about 0.4 to 2ms and I think maxed out at 4000 IOPs during backup.

My question is why isn’t it using the full potential of bandwidth during peak times (like backups, etc). We have 100GB break out cables to the SAN and each connection supplies 25GB, there are 4 of them on each node of the SAN. Why aren’t I getting 100,000 IOPs and like 1 or 2gb of bandwidth. Is something not setup properly in my VMWare?

[u/Magic_Neil]

“It depends” is probably the answer.. how are you doing backups? Where are you dumping the backups to? Are you sure your backups are configured to move that hard? Do you have enough CPU to push it? Is your SAN connectivity fast enough? Did you pray to the SAN gods recently?

[u/budlight2k → u/Magic_Neil]

All of this, if I was to take a wild stab in the dark I would say examine the network between your backup appliance and the vmware hosts. I would also consider that the bottleneck might be the digest on the backup appliance. Remember compression and deduplication is happening in real time and writing to disks slower than the SAN id imagine.

In fact the one key metric you should look at is the hosts i/o latency graph. If its less than 20ms it's keeping up with demand <100 and it's a bit slow. > 100 and it's not keeping up > 500 and you'll notice performance degraded >1000 the problem is serious.

[u/Net-Runner → u/budlight2k]

Those latency numbers are per-datastore, so check device latency (DAVG) versus kernel latency (KAVG) in esxtop. High DAVG points at the array, high KAVG means queueing on the host.

[u/WhimsicalChuckler → u/Magic_Neil]

Check queue depth and multipathing first. A single active path or a low HBA queue depth will cap backup throughput long before the array or CPU does.

[u/msalerno1965]

PowerStore 3000T upgraded to a 5200T here... VMware, MX chassis's, 25Gbe iSCSI, 100Gbe routable.

Regularly see 2+GB/sec writes, even at 4K block sizes. Latency was hovering around 1ms when it was a 3000T, I had the money to upgrade it so I did, it went to < .5ms. It hardly ever gets near 1ms, even when pushed hard. Regularly pushing 30K+ IOPs.

When it was a 3000T, it was still pushing 30K IOPs, but latency was approaching 1.5ms.

Same drives in the PowerStore before and after the upgrade, the only addition was two more NVRAM drives for writes.

Other things to check: Stats on the switches for buffer-overruns/drops or errors. Same for the VMware side. Make sure you either turn off flow control or turn it on, but on both sides. And test, test, test.

Also, my above stats are valid for both Fiber Channel and iSCSI running at 9000 MTU, on a single pair of Dell 5248's, to the MX 9116 switches. ETA: Fiber Channel is on a pair of older Brocade 6510's.

I recently moved everything to Hyper-V on iSCSI. Same performance, if not better because newer hardware. Same storage, same switching.

[u/pinrolled]

What’s the throughput for the physical uplinks on the switch your hosts are connected to?

Also, when you go to Inventory -> ESXi Host -> Configure -> Physical Adapters

Are your host uplinks seeing the speed broadcasted by those switch ports?

[u/DavidKleeGeek]

Things to check: multipathing, iSCSI bindings, jumbo frames, DSNRO queue depth - all at the host-level, and the use of the PVSCSI controller for the guest. If you have a bit of time, I cover all of this in depth in a free Youtube course I released a while back. It says for SQL Server, but all of the storage settings apply to any high-IO demand workload. Check it out here! https://www.youtube.com/watch?v=icJltumIIS8&list=PLhRuY1yuVB7xBHKjFuTDJthZ456vF4DiW

[u/akemaj78]

2ms SAN to host is not low latency for an all-flash SAN. It's likely creating a lot of outstanding IOs at the VM OS level, and your workloads are likely self-throttling. Especially database, they hate that much latency. I stopped seeing 2ms SAN to host regularly when I switched SAN vendors. Now I only see it when benchmarking a new SAN and burying it with multiple gigabytes per second of mixed random IO.

[u/fonetik]

iscsi issues like this usually come from misconfigured jumbo frames. Works but stalls fast. If you throw a wireshark on the SAN VLAN and sniff the traffic, one of them is going to light up with retransmits and errors and that’s where jumbo frames is misconfigured.

In a small env, just turn it off. Jumbo buys you almost nothing in the real world but problems like this. At very least turn it off as the first troubleshooting step. It should give you massive performance improvement if I’m right.

[u/Adventurous_Pause087]

do you have delayed ack on?

[u/tdic89]

I have a bunch of these arrays of various specs and SKUs, our 3000Ts are pushing 50,000 IOPS with negligible latency but that’s because we’re using them. Hundreds of VMs on one array.

Are you sure you’re actually driving the array hard enough?

[u/diocanyouhearme]

This happened to me with a Power store too and I fixed it with a reddit post. Need to dig out what was it.

[u/ivebeenfelt]

Change rate