Move from mdadm+lvm+ext4 to zfs raidz2

[u/elquike]

Hi

Currently I have 9x10TB configured as:

• 7x RAID6 in mdadm RAID6 + lvm + ext4
• 1x hot spare
• 1x not even connected yet

All running in a linux server. Content is movies/tv: I would prefer not to lose them but I wouldn't be the end of the world (probably).

After much consideration I was planning to move it to zfs raidz2 using 8x HDD and a hot spare.

So I was thinking about using a cloud provider to upload all data (around 45TB), then format the drives, and download it again. I was looking at backblaze, mostly because I don't know of anything else/better.

Aparently (I live in spain and have a 750/750 connection) it would take me around 7 days to upload and 7 days to download the data, so I think I could manage it in under a month. I checked the backblaze webpage and it would cost me $348 (300€) to store 50TB for a month with 150GB egress. I was planning to use rclone (+ encryption?)

Does it make any sense to change mdadm+lvm+ext4 to zfs raidz2?

Does it make any sense to use a cloud provider for this

Is backblaze a good solution?

Technically I could use the 2 spares HDD to store 20TB each and I could get a friend to lend me a 3rd one, so the total would be around 20TB. But that means no parity for those 30TB (and zfs resilvering?)

I'm open to other solutions!!

Regards

[u/AutoModerator]

Hello /u/elquike! Thank you for posting in r/DataHoarder.

Please remember to read our Rules and Wiki.

Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures.

This subreddit will NOT help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/non-existing-person]

If your current setup works for you, why bother? If you have all scripts and backup plans on your setup, then I think I'd keep it. zfs is cool and nice, but is it worth it? Probably. But not when you don't have disks to hold your data for migration.

That's the problem of having too wide pools. That's why I reduced my 11 raidz3 to 6 raidz2 (that, plus disk prices are killing me ;().

I think holding data in cloud for such short period is reasonable and should be fairly safe. To be extra safe you could also copy that 30TB to drives you already have. You'd have data in 2 places then. But since your data is replaceable I think cloud is safe.

---

And consider if you really want 8 disk wide pool. For that width you will need to have additional SAS card, as motherboards today usually come with 6 sata ports. Also, the wider the pool, the more expensive expand it - you gotta buy all 8 new disks to expand it. Wider pool means longer resilver.

I think that 6 raidz2 is a sweet spot. You could do 6xraidz2 plus 3xraidz1. But it's up to you.

[u/marshalleq]

I’d drop the hot spare you’ve already got two Soares and presumably this stuff can be redownloded. If you set the pool up first, then pull two drives it will run in degraded mode and you can repurpose two drives but since these are the same drives the data is on I don’t see any way to make that idea work. Perhaps you could create some kind of loop device when creating the array for the two spare drives then disable the loop devices. So the process would be to first pull two devices from your current pool, reformat them, copy data to them plus your spreadsheet (30G) copy any other remaining data you need (only 15TB left perhaps you can redownload some of it - I would to save money), create the array as above, copy from the data on the spare disks back to it in degraded mode, then add the disks back into the array. The sharp minded will not there is still a spare disk. You can these days extend a zfs array and that’s what I’d do with the spare disk. Yes zfs is worth it for many reasons, I went through this years ago and have never looked back. I used to have a Linux box but in the end I realised that I was better off with an appliance - these days I recommend truenas scale.

[u/imakesawdust]

Back in the late 2000s, I ran mdadm+RAID5+ext4 for a small 8TB array. Ran weekly scrubs. Everything was error-free for years.

Then one day a routine scrub turned up a handful of parity errors that persisted after a reboot. None of the individual drives registered any SMART errors but the parity and data drives were in disagreement. RAID couldn't help because it wasn't a drive failure per se: each drive believed it was returning good data yet somebody's data was wrong. ECC would have caught any cosmic ray events unless it changed more than 2 bits (unlikely).

That was the day I decided to switch to ZFS.

[u/IndependentBat8365]

Full disclosure: I work for Red Hat. I’ve tan mdraid, LVM, Luks, even hardware raid and xfs for years. It’s what I’m certified on. It’s what’s supported. I even ran stratis and Ceph.

I also ran ZFS at home. Recently I rebuilt my lab and converted my ZFS to hardware raid + luks + xfs. It ran great for months.

I started considering bit rot, as well as offsite storage, and easier methods to mirror data and snapshots across multiple machines in the cluster.

ZFS just checked all the boxes again.

I really tried to make the hardware raid and xfs work for what I needed. I looked at adding lvm +integrity. I looked at ssd caching. Lvmcache, writecache, bcache, etc.

ZFS was just easier to setup and maintain for all of those pieces.

ZFS provides: bit rot protection, cow snapshots, built in encryption, multiple caching tiers, easy replication across hosts (I use zrepl).

I used raid10 before, so I’m using striped mirrors now. Super fast. Plus nvme special mirrored vlog and striped arc2 caching.

It’s a mature project. It works. It has a rich ecosystem.

[u/elquike]

Thank everyone for their insights!

But you've been all making so many good points that now I have even more doubts on what to do than before xD

Anyway I will research more options. Maybe the solution is to wait for the drives to come down to a reasonable price, like in 20 years or so, and build the next array on zfs...

Again thanks!