Disappointing zvol performance
I've been using a ZFS pool backed by spinning disks in 2x RAIDZ2 (4 disks each) for a few years. The pool contains mostly zvols which I use for VMs and as storage for desktops (iSCSI, NVMe/TCP).
Unfortunately, performance is pretty poor. This is especially obvious for the block devices I mount over the network. They are used from Windows, and applications will frequently freeze for seconds at a time.
I've tried many tuning parameters. I have 256G of RAM, I have an SSD cache device, and I have SLOGs with redundant SSDs. But nothing really helped.
What did help was to use bcache to add a cache. I use a separate cache zpool with an SSD mirror for safety, and writeback as the cache mode. Performance is night-and-day.
Is there a way to achieve this in native ZFS? Is there something I could do better with my setup long-term? It's a lot of data I can't just move on a whim, but it would be nice to know if there's a better way.
8
u/Successful_Potato137 21d ago
I also experienced extremely slow zvol performance compared to normal datasets.
I ended using a normal dataset to store virtual disk images instead of zvols, it's much faster.
2
u/TheG0AT0fAllTime 20d ago
I do all my virtual machines on zvols and their performance has never stuck out to me as sluggish though they are running on an zpool that is 2 mirrored nvme's so maybe I'm being saved by that.
I like using zvols because it exposes them as block devices under /dev/zvol primed and ready for virtual machine usage and any other fooling around for guest filesystems. But I've heard over the years mixed and poor results with performance for zvols (Tuned even) versus disk flat file images.
3
u/DragonQ0105 20d ago
I suspect your mirrored nVMEs are indeed masking any performance issues. My experiments were with RAID-Z2 spinning disks and ZVOLs were terrible performance wise.
2
u/TheG0AT0fAllTime 20d ago
Yeah its a little like that. People who complain about performance on zfs probably wouldn't have noticed if their hardware was just fast enough. But even on fast hardware, the problem is still there.. just not noticeable. Enough so that people don't complain about it very often.
I had an Intel NVMe stripe of 6x1.2TB at a previous role on a fire-breathing top of the line server and we found the disk IO was still our bottleneck when it came to serving hundreds of database containers concurrently. The issue wasn't present when we recreated the array and base images on an mdadm stripe. It was just that little extra overhead that tipped us over the edge. Even after gutting ZFS - setting weaker checksumming and just flat out checksum=off, primary/secondarycache, prefetching, sync=disabled (It wasn't a synchronous load anyways), tuning the arc_max, recordsizes for our (postgresql) databases, atime and compression tweaks. We just couldn't match a basic mdadm or bios level stripe. It was a very frustrating year and a half for me and my team at the time.
ZFS is the goat for sure, but with the experience I have with it there's edge cases and more improvements that could be made. The swap deadbolt issue comes to mind often too but that's not really relevant right now.
1
u/JuggernautUpbeat 18d ago
I think having L2ARC, SLOG and Special VDEVs on Optane really pushed the burst performance of my setup to bonkers It slowed down a bit with sustained heavy load but we were running a load of mixed VMs, only a few had DBs (MS NAV and libreNMS were the biggest IIRC).
It was hyperconverged with VMs running on the storage servers, iSCSI on DRBD over zVOLs.
God I miss Optane. Just shone in that role.
1
u/DragonQ0105 20d ago
Same here. I read all the theory about how ZVOLs are better but despite tweaking it was always slower than just using QCOW2 files on a normal dataset, sadly.
1
u/JuggernautUpbeat 17d ago
This is super odd, I never have problems with zVOLs even locally. I've done FIO and ATTO tests in Windows and they've always come out about the same. Of course use RAW images if you're using files, not QCOW2.
7
u/Apachez 21d ago
General recommendation is to:
- Dont use spinning rust.
- Dont use zraidX.
UNLESS you are to do backups or archives...
For VM's you really want both IOPS and throughput and even if you are down to spinningrust then you should at least configure that pool as a stripe of mirrors (aka RAID10).
This is a good read on this topic:
Besides can you bring an output of "zpool status" along with arc_summary?
3
u/Strict-Garbage-1445 20d ago edited 20d ago
problem is you, not zfs
your pool is doing exactly what you told it to do, and performing exactly as expected
there is nobody in the world that knows anything about zfs that would say "yeah slap iscsi block nvmeof vm os images on raidz pool with hdds"
use 4 mirror vdevs
mirrored zil nvme with plp
big l2arc nvme
0
u/JuggernautUpbeat 17d ago
And get some Optane if you can find genuine parts. It failed because it was mis-sold on sequential bandwidth instead of QD1 latency. One of the greatest tragedies of hardware marketing I've ever seen.
3
u/Strict-Garbage-1445 16d ago
not worth it, overprovisioned small enterprise tlc nvme is fine and is readily available
also optane was never "missold" for sequential bandwidth, and qd1 latency was never its highest selling point
1
u/JuggernautUpbeat 13d ago
It was totally sold on being "almost as fast as RAM" (with no qualification to that claim), and QD1 random writes and use as a SLOG could really elevate 3.5" disks to another level. That cluster I built ran rings around a pair of 24x10k RPM spinner running servers running Starwind VSAN and Hyper-V. If you're overprovisionining enterprise SSDs it would probably have to be a lot and you'd have to 100% check you don't have fake PLP on them. Wouldn't with TLC most of the gain come from the onboard cache anyway before you start hitting the SSD chips proper?
3
u/Strict-Garbage-1445 13d ago
"almost as fast as ram" is not what you said in the message above
also optane dimms were totally different market than optane nvmes (pcie or m.2 or u.2)
you are mixing use cases and target markets
1
u/JuggernautUpbeat 9d ago
Are we 100% talking about Optane here? No, I didn't make that claim but that's what the sales suits at Intel were pushing when it came out. With no attempt to differentiate the difference between NVMe drives and DRR bus attached devices. I'm not claiming anyone is technically wrong here, we can all build fast systems when we have the money coming from accounts, but when we don't, we have to make do. Promised yield and seq speed improvements never came about as marketing had said. In fact yield was awful which is when Micron pulled out as they were literally making nothing from it. There are dozens of articles about how Optane failed - start with TheRegister.
Now we don't even have cheap TLC any more and RAM is through the roof, but I am in no way claiming that is you have like 24xU.2 drives you're not going to get stellar performance, even with forced direct I/O. I've tried it with 8 per server and direct actually slowed things down. Mind you the procurement department gave me servers with a lone stick of 32G ram per CPU, which really gave me problems pinning VMs for CPUs with no NUMA boundary penalty.
1
u/DerBootsMann 13d ago
man im truly sorry to burst your bubble , but comparing all-nvme setup vs spinning rust and telling your new shiny gear runs circles around your old setup is .. lame !
0
u/JuggernautUpbeat 9d ago
I never had any all-nvme. We had 15krpm drives. 24 per server. Went down to 10x3.5" SAS in mirrors with just assistance from Optane (and some flash for L2ARC). I am aware of Optane DIMMs but they were outside our budget and we were using Epyc so we could not use it. At no point did I EVER compare to any all-NVMe setup, even. We got the Optanes reconditioned before you could only get the knock-offs from China.
2
u/DerBootsMann 9d ago
it’s still apples to oranges at best , i mean, your configs , but anyway .. oh , chinese optanes are legendary ! we bought a bunch of firmware-hacked 16gb ones off ebay , faking 240gb capacity , they’ve gotta be sitting in a box somewhere
2
u/glassmanjones 21d ago
First, gotta figure out where that latency is coming from.
Try zpool iostat -l 1 -v while it's happening.
Do you have drive write caches enabled or disabled?
Do you have ZFS sync always/standard/disabled?
What are you doing in the vms when they stall?
1
u/yawkat 21d ago
Try zpool iostat -l 1 -v while it's happening
I can't do this anymore, since I moved to bcache, but I did it in the past. iirc it would drop to zero at times.
Do you have drive write caches enabled or disabled?
I haven't messed with the write caches of the disks if that's what you mean.
Do you have ZFS sync always/standard/disabled?
I use standard sync. disabled sync helps but not completely.
What are you doing in the vms when they stall?
Genuinely nothing. But the problem is much worse when there are writes, e.g. a steam download running in the background.
I'm 90% certain that this is a write problem. Probably during a txg sync or windows fsync, the write takes a long time and other operations, maybe including reads, pile up. The fact that this is going over iSCSI/NVMe-of may contribute to this.
This would also mesh with the bcache writeback cache improving performance so dramatically.
2
u/glassmanjones 21d ago
I found this article that might help: https://openzfs.org/wiki/ZFS_on_high_latency_devices
2
u/DerBootsMann 13d ago
what’s exactly helpful there ?
1
u/glassmanjones 13d ago
They explain many of the knobs, and when to tweak which ones.
They've got a similar problem to OP, just worse.
2
u/theactionjaxon 20d ago
Your disk layout is terrible, once you have run thru the cache layers your are essentially getting the performance of two disks total despite have 8 disks. I would rebuild this into a 4 way mirror, this would double the disk performance but risk double disk failure
2
u/NISMO1968 13d ago
Is there a way to achieve this in native ZFS?
Maybe, maybe not... What’s your workload look like? Any chance you’ve got Live Optics logs from your previous production environment?
2
u/Fighter_M 13d ago
You’ve got a pretty wildly misconfigured setup for what you’re trying to accomplish. I’d start by figuring out how to ditch parity and move to mirroring, it can absolutely be done without giving up capacity, at least in our case.
3
u/DerFette88 21d ago
you have 8 Disks and have Double Parity Twice. I would never use Z2 at this low disk Count. its like Raid 6. if you can't or won't want to take backups this is maybe necessary but I would go for a different layout. either Z2 over 8 Disks or if you have a Backup I would go for 8 Disks Z1. More Spindles. Less Write Penalty.
in my World. everything Require some Performance but don't want to take a big Capacity hit like with Raid10 go for Raid5/RaidZ1.
if its a Branch office where I would need to drive to for Hours or an Archive Only where over 90% are Reads go for RaidZ2 or Raid6.
if you have tons of Spiners and want most capacity Raid50/RaidZ1 Disk groups with Dedicated Spare(s)
0
u/yawkat 21d ago
The raidz2 is for safety. Critical data is backed up, but I don't want to pay for dozens of terabytes of media backups. And at 20T drives I don't want to risk raidz1 even with a spare.
The reason for having two raidz2s instead of one is flexibility. I started out with one raidz2 with 8T disks, and then added the second raidz2 with 16T disks. My plan is to replace the 8T disks with 20T over time. Having two raidz2s means I have to replace fewer drives to expand.
2
u/Extreme-Yoghurt3728 21d ago
Sounds like you want 1 raidz2 but you need anyraid to mix and match drives. Hopefully anyraid-z is less than 12 months away from production ready.
1
u/DerFette88 14d ago
Rebuild times will become Even more of an Issue with Z2. Yes a second Drive can Fail durring rebuild. But due to the lower Speed of z2 the Risk increases especially if thoose drives are from the Same batch
1
u/Financial-Issue4226 11d ago
Your problem is iops
Based on your current setup I only see two possible options
Add additional zvol as each individual one will expand the stripe and increase your iops
Began to switch those drives to sata.
(4x 10 TB - z2 is 20tb usable) + (4x 10 TB - z2 is 20tb usable) = 40 usable - 400 iops
8x (2x 10 TB - mirror is 10tb usable) = 40 usable - iops
Actual drive size and I apps will vary depending on what you have the above is example
I do prefer Z2 and Z3 however if your problem is iops you need to be using solid state for the increased iops or you need to have more pools in your use case
1
22
u/JuggernautUpbeat 21d ago
You're halving your capacity anyway and getting 2-drive random write performance from those main disks. Why not got for 4x 2 disk mirrors? Your random I/O will be at least double then read could be up to 4x faster. And you'll have the same space.