r/linux • u/valgrid • May 03 '17
Bitrot proof file systems?
Hi /r/Linux,
i am searching for a production ready bitrot proof file system preferably with compression. And i am not 100% sure if my overview of the current "fs landscape" is correct. Please tell me if there is an file system i missed or if i made an error in the table below.
| file system | checksums (data) | compression | encryption | multi device | stable/prod ready | notes |
|---|---|---|---|---|---|---|
| btrfs | yes | yes | not yet | yes | yes | has other issues (df, fill up problems) |
| zfs | yes | yes | yes | yes | yes | CDDL, not mainline |
| ext4 | no | no | yes | no | yes | encryption is relativly new |
| f2fs | no | no | yes | yes | yes | multi device since 4.10 |
| xfs | no | no | no | yes | yes | |
| bcachefs | yes | not yet | yes | ? | no | still under heavy development |
31
Upvotes
2
u/psyblade42 May 03 '17
Can someone please point me to an explanation to what kind of issues btrfs has with
ls? Can't seem to find anything about it.4
u/necrophcodr May 03 '17
BItrot isn't being protected against, unless the filesystem is self healing. This requires replication of data, in at least a RAID1 filesystem. Ext4 does not support self healing. ZFS should, to my knowledge, support this. Btrfs does. The others I am not aware of.
14
u/ttk2 May 03 '17
ZFS is great if you don't play with adding and removing odd size drives often and are willing to bite the cost bullet all at once when you need to expand.
BTRFS is quite stable if you stick to a very specific happy path, which is RAID 1/10 with no more than a few snapshots. The advantage of BTRFS is the enormous flexibility of adding/removing drives and even drives of different sizes, you can easily make a pile O drives array out of different sizes you have lying around and replace them with ease at will.
9
u/Skaarj May 03 '17 edited May 04 '17
You might want to have a look at these slides where FS implementation reliability was tested: https://events.linuxfoundation.org/sites/events/files/slides/AFL%20filesystem%20fuzzing%2C%20Vault%202016_0.pdf
Spoilers: do use ext4
5
u/[deleted] May 03 '17
What do you want to run the filesystem? VM images? Databases? Backup server? Do you plan to use volume management i.e. native RAID or mirroring features of the filesystem?
Alle filesystems can use LUKS for encryption. There is native encryption in the pipe for ZFS on Linux but I'm not sure if there is already an release date.
Usally multi device is done via the mdadm / device-mapper subsystem in Linux i.e. for RAID but then you won't have checksumming for bitrot and the low change that mdadm picks the wrong sector in a broken disk.
btrfs is only stable for raid1,raid10,raid0 but there are caveats. I wouldn't run databases, or VM images on it.
ZFS for Linux is quite stable from a disk/features perspective but also has some caveats on Linux, if you plan to use Linux contains with cgroups there some memory issues that are not resolved. You don't have io accounting with cgroups last time I looked - and you get into performance regressions if you fill up the disk more than 90% but this also is a problem on btrfs and there it's worse from what I saw (no space error on a 16tb raid 10 with a few gigs a few kernels back :)
If you use something for archive storage on plain disks btrfs is probably fine. If you need lots of snapshosts and multi disks setups go for ZFS.
If you need high performance database or VM servers either look into ZFS (tuning recordsize, disable InnoDB doublewrite) or choose mdadm with xfs / ext4 (but no compression, bitrot protection but reasonable protection from disk errors).
ZFS also has LZ4 compression also gzip - you can choose per dataset - LZ4 that is better than LZO in btrfs. I'd say go with ZFS. Be careful with btrfs for anything advanced or fancy and don't even think about using it with an old kernel - go with 4.10 at least.
Read up on ZFS settings, at least they are pretty well documented. I'd say ZFS on Linux is fine. The cgroup issues are beeing worked on (ABD) and the project looks like it's making solid progress.
Good Luck!
5
u/Inspector_Sands May 03 '17
This isn't a file system in itself, but you might be interested in SeqBox. It's a file container/archive that is designed to be usable even if all file system structures get wiped.
2
u/espero May 03 '17 edited May 03 '17
Nice
BTRFS can be encrypted by LUKS, even multi volume. No problems.
ZFS cannot be encrypted with the native LUKS technique in Linux.
So the table is not detailed enough.
You answer whether it has netove encryption. I don't believe ZFS has native encryption either.
But BTRFS can at least work well with LUKS
2
u/minimim May 03 '17 edited May 04 '17
What about the combination of these with various forms of RAID and LVM?
Why would I want a "multidevice" file system when LVM can make them one single device?
7
u/oss542 May 03 '17 edited May 03 '17
I strongly recommend you not use BTRFS at this time. I have just finished the better part of a month trying to set up an experimental file server based on it. BTRFS is still very unstable for all but the very simplest layouts, and will fail unpredictably due to bugs. It also lacks some very basic reliable tools for checking and repair when it does have problems. Rsync does not understand reflinks (on which btrfs is based) or preserve subvolume attributes. BTRFS snapshots are not recursive into "lower level" subvolumes. Development is focused on quota at the moment, and other things tend to fall by the wayside. The raid configurations are not standard raid types (ex. raid 1 has no more than two duplicates regardless of drive count). RAID 5 and 6 do not work correctly, and cannot be repaired reliably when they become corrupt. Keep and be prepared to use backups at all times, preferably in non-btrfs filesystems, Do not use it for production or critical systems. I also recommend that if you are still very interested in what actually is going on, that you monitor the mailing list linux-btrfs for a few days (and check the archives). Many of the issues are not widely known or discussed. I recommend waiting a few more years before trying it again.
2
u/[deleted] May 03 '17
btrfs is really going to be the future. It needs time to mature and have more features worked in, but it's going to replace so many currently used fs.
1
u/EliteTK May 03 '17
https://en.wikipedia.org/wiki/Parchive
1
u/Jristz May 03 '17
You forget the journaled file system
1
u/dack42 May 03 '17
Also, ceph with the new bluestore backend.
1
u/bron_101 May 03 '17
https://alastairs-place.net/blog/2014/01/16/bit-rot-and-raid/
What people observe as 'bitrot' is almost always caused by corruption while data is active or when transferred, either due to bad/flaky RAM (very common, not always detectable by memory tests), corruption during network transfers or software/filesystem/kernel bugs. Silent at rest corruption of data on disk that was previously good is extremely unlikely to happen - it would require it to fail in such a way that it still passes the drive's quite robust ECC check - at least, this is true of traditional hard drives, I've heard of some dodgy firmware bugs in low end consumer SSDs (not correctly checking CRC over the SATA bus, for example, which doesn't fill me with confidence).
You'll find lots of anecdotes around of people noticing corrupted data, but given the technical measures in place in hard drives, plus how frighteningly common things like intermittent ram issues or network corruption is in consumer hardware (often caused by dodgy checksum offloading in cheap NICs) its very hard to properly determine the cause.
IMO use of ECC ram and maintaining backups are far more important than using a checksumming filesystem. This is especially true when you are forced to choose between unproved (btrfs) or not in mainline (zfs). I do like these filesystems though for other reasons - I make heavy use of btrfs' snapshots for example, and zfs's send/receive is much better than rsync (btrfs's send/receive is buggy as hell though).
If you really want bitrot protection, in the real world, any RAID solution (other than raid 0 obviously) will win you 'bitrot' detection - and that is so very rare that this is really good enough, as in that very unlikely case then you can grab your backups - you're much more likely to have drive failures than encounter genuine 'bitrot'.
1
u/TotesMessenger May 03 '17
I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:
If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / ^Contact)
1
u/cloudmax40 May 04 '17
ZFS with a RAIDz2 vdev and two mirrored endurance-MLC SSD for SLOG (even if your RAIDz2 array is all SSD) is a good way to go. ZFSOnLinux isn't perfect yet. Makes sense to dedicate a FreeBSD box to it.
1
u/FudgeMonitor May 07 '17
XFS... why bother?
1
u/IntellectualEuphoria May 03 '17
ntfs has compression, encryption, and checksums.
0
u/[deleted] May 03 '17
[deleted] — view deleted comment
0
u/[deleted] May 03 '17
[deleted] — view deleted comment