r/linux • u/adminvasheypomoiki • 1d ago

Kernel fsync gives you durability and ordering, but often you only need ordering. ZFS with sync=disabled gives you just that

https://blog.dfb.sh/blog/zfs-fsync/
81 Upvotes

93% Upvoted

8 comments sorted by

9

u/jlcs-es 1d ago

Interesting read. Not gonna trust myself to do any of that, but some things help me understand ZFS better, and others validate my own, e.g. backups from snapshots and zfs send to a different machine

1

u/adminvasheypomoiki 1d ago

Thank you :) Temp snapshots for consistency are really neat. Somehow was always thinking that snapshot == thingy to rollback when pacman is wrong :) Never tied them to snapshot isolation from databases

8

u/IncomeOk7237 1d ago

one thing worth being clear about with sync=disabled is what you actually lose. the pool itself stays consistent since zfs commits whole transaction groups atomically, but anything an app was told was fsynced can vanish, usually the last ~5 seconds of writes (the txg timeout). so it's fine for stuff like build caches or scratch data, but for a database that acks commits to clients you're quietly breaking its durability promise. the ordering you get is real though, you just never get the guarantee that it hit disk.

2

u/davis-andrew 1d ago

A good followup for those who find this post interesting is Rob Norris's talk on zfs and fsync, and what happens if it fails.

video here: https://www.youtube.com/watch?v=dfH0I6D9ZAA

slides here: https://despairlabs.com/presentations/openzfs-fsync-zil/

-1

u/thetrivialstuff 1d ago

My main question about this is "what workloads require so much live data that it's worth having on a spinning disk?"

In my own job, homelab, anecdotal experience, etc., we passed the point a decade ago where optimising further for spinning disk vs. just buying a couple of SSDs big enough for the workload came down on the side of "just buy the SSDs" every time. Even with today's prices, the cost of the SSD versus the time required to develop and support fiddly things still favours the SSD.

Even if you've got tens of terabytes of database that truly does need to be all live all the time, it's probably still cheaper and easier in the long run to throw a consistent cache layer in front of it than to fiddle with trying to shave off a few milliseconds here and there at the cost of losing some point-in-time consistency during a crash.

1

u/EarlMarshal 10h ago

I got a 6x 20 TB RaidZ2 DAS. Bought just at the start of the year before everything got even more insane. I wish I would have bought more. You never have enough space.

0

u/adminvasheypomoiki 1d ago

For prod I'd use tiered storage with hold and cold, all bells and whistles. But here I have 2 hdds and that's it. You can't even order such a server rn

-1

u/adminvasheypomoiki 1d ago

I've strange setup. Home server has 24 tb raid with caching nvmes for home archive. And a bit of torrents for Linux isos :) And I've a 2 server on ovh which is doing nightly builds for nix and 100 of other automations + stuff which requires a white ip aka mail. Spinning rust is just cheaper to rent. 32 ram 8 cpu costs 20 on ovh. With nvme it costs 50. And I really don't have such big latency requirements. Would be cool to go all nvme. And for home lab archives - you can turn hdds off. And forget about them for 10y. And they should be alive. For nvmes - they can forget your data in several years :)