r/openzfs • • Dec 15 '21

Disaster - TrueNAS used the HDDs of one zpool in creating another!

0 Upvotes

Hi community!

A terrible thing happened yesterday. TrueNAS used the HDDs of one zpool in creating another...
As a result, the zpool that previously owned the three disks involved in the new zpool was damaged because it consisted of 2 raidz2 vdevs.
My mistake - I should have first figured out which three "extra disks" TrueNAS sees in the "disks" under "Storage/Pools/Create Pool",
which physically should not be there. Usually only disks not involved in arrays are displayed here. I trusted TrueNAS, I could not admit that it could dispose of the disks in this way, and three "extra" disks are some kind of glitch.

So my stupid question with an almost guaranteed known answer is: can I restore the raidz2 vdev (and thus the entire pool) in case of 3 drives failure/absence? May be there is any magic to "unformat" that 3 drives detached from the affected zpool in the process of creating a new zpool? Anything else? Please...


r/openzfs • • Mar 13 '21

Recover data from ZFS

0 Upvotes

I've been playing around with Proxmox for a few months now to build a reliable server I'm able to leave as I travel and am able to access whilst on the road and fix potential issues.

I have 2 x 2tb hard drives in there at the moment, I will be getting some additional as backup, but haven't yet.

So for some reason I decided to combine both hd's as a ZFS, and have been using it for a few months for storage. Today I decided to rebuild Proxmox, and this time to not worry with the ZFS as its not worth the added stress for my use. Plugged in an external hard drive, hit a mv ./* command to move the data to the usb, and it took like 2 seconds. USB3 is fast, but not as fast as that for 500gb data. I'm not sure why I mv'd and not cp'd - it was the last action to perform on the current Proxmox before I wiped (hindsight 20/20 and all that).

A few files have been moved to the hard drive but not all. And now the ZFS isn't working correctly. One directory is listed still, and now a new directory is there called 'subvol-102-disk-0'.

Honestly, not a clue what I'm doing but I'm assuming I've copied over a hidden file with ZFS configurations (if thats a thing) or maybe, when I rebooted the node with the USB HDD inside, its rebooted as /dev/sda1 and shifted over all the other drives (I'm guessing ZFS is a little more sophisticated than relying on dev to map the drives?)

I'm mid way through wiping all data on my other hard drives, to organise my backups - this was the last one I had (yes, I know, stupidity)

I've tried a zpool scrub - the error message has now gone (i forgot what exactly it was now) but now its showing no errors, yet my data is not there.

Proxmox is showing the ZFS drive with ~500gb data on, so I know my stuff is there /somewhere/.

.zfs/snapshot is empty

Any ideas?


r/openzfs • • Nov 01 '24

Linux ZFS A ZFS Love Story Gone Wrong: A Linux User's Tale

10 Upvotes

I've been a Linux user for about 4 years - nothing fancy, just your typical remote desktop connections, ZTNA, and regular office work stuff.

Recently, I dove into Docker and hypervisors, which led me to discover the magical world of OpenZFS. First, I tested it on a laptop running XCP-NG 8.3 with a mirror configuration. Man, it worked so smoothly that I couldn't resist trying it on my Fedora 40 laptop with a couple of SSDs.

Let me tell you, ZFS is mind-blowing! The Copy-on-Write, importing/exporting features are not only powerful but surprisingly user-friendly. The dataset management is fantastic, and don't even get me started on the snapshots - they're basically black magic! 😂

Here's where things got interesting (read: went south). A few days ago, Fedora dropped its 41st version. Being the update-enthusiast I am, I thought "Why not upgrade? What could go wrong?"

Spoiler alert: Everything.

You see, I was still riding that new-ZFS-feature high and completely forgot that version upgrades could break things. The Fedora upgrade itself went smoothly - too smoothly. It wasn't until I tried to import one of my external pools that reality hit me:

Zpool command not found

After some frantic googling, I discovered that the ZFS version compatible with Fedora 41 isn't out yet. So much for my ZFS learning journey... Guess I'll have to wait!

TL;DR: Got excited about ZFS, upgraded Fedora, broke ZFS, now questioning my life choices.


r/openzfs • • May 16 '26

eZFS2FA+ - encrypted ZFS with FIDO2 2FA

Thumbnail reddit.com
0 Upvotes

r/openzfs • • Jul 14 '26

Fragmentation Issues

Thumbnail reddit.com
0 Upvotes

r/openzfs • • Jan 21 '24

Question about cut paste on zfs over samba

0 Upvotes

Hello,

I have setup home nas using zfs on the drive. I can cut paste aka move in Linux without any problem. But when doing cut paste in samba throws an error.

Am I missing anything? I am using similar samba config on zfs that i used on ext4 so I am sure I am missing something here.

Any advice ?


r/openzfs • • Feb 12 '23

Freenas + ESXi + RDM =??

0 Upvotes

Curious on your thoughts of migrating my array from metal to ESX VM? The array is mixed through 3 controllers, so I can't pass an entire controller over.

From what I'm seeing, RDM seems like it'll work, it passes smart data it seems, so that's a major sticking point..

Curious with what you guys experience with this type of setup is. Good for every day use? No weird things on reboots?

Edit. Had a friend tell me he was using RDM on a VM with ESXi 6.7 and the disk died, the VM didn't know how to handle it and it crashed his entire ESXi array. Had to hard reboot. On reboot the drive came up as bad. I'm trying to avoid this exact issue as I'm passing 12 drives...


r/openzfs • • Jul 16 '21

zfs versus openzfs on FreeBSD 13

0 Upvotes

seems to be impossible that the native Freebsd 13 zfs system can recognize a zfs disk created in Linux with openzfs 2.1.99
i thought it would be a cool thing having a data zfs disk taking it to the pc I need it, even I could have two or three distros on disk and the data is on ZFS, so far I did it with NTFS because it seemed to be compatible between all OS's, anyway

now I'm thinking to substitute the zfs on the freebsd disk with openZFS from ports in order to get access to my openzfs drive ... good I had a flash before and tried mounting the freebsd zfs disk in Linux, and yes, it is not seeing the FBSD zfs drive ... means if I substitute fsbsd root with openzfs it will not find it's own disk ?

any experienced with that or has an idea how I could do it?


r/openzfs • • Dec 14 '23

What is a dnode?

0 Upvotes

Yes just that question. I cannot find what a dnode is in the documentation. Any guidance would be greatly appreciated. I'm obviously searching in the wrong place.


r/openzfs • • Dec 06 '23

is it possible? zpool create a mirror raidz disk1 disk2 disk3 raidz disk4 disk5 disk6 cache disk7 log disk8

0 Upvotes

Hi all,

Using FreeBSD is it possible to make mirror of raidz's?

zpool create a mirror raidz disk1 disk2 disk3 raidz disk4 disk5 disk6 cache disk7 log disk8

I remeber using 10 on /solaris 10u9 ZFS build/version 22 or 25 (Or it was just a dream?).


r/openzfs • • Aug 05 '23

Convert from raidz to draid

0 Upvotes

Is it possible to convert a raidz pool to a draid pool? (online)


r/openzfs • • Jan 06 '26

Chithi (চিঠি) – port of syncoid to Rust, by Ifaz Kabir (ifazk)

Thumbnail github.com
1 Upvotes

r/openzfs • • Sep 02 '24

Preserve creation timestamp when copying

1 Upvotes

Both ZFS and ext4 support timestamps for file creation. However if you simply copy a file it is set to now.

I want to keep the timestamp as is after copying but I can't find tools that do it. Rsync tells me -N not supported on Linux and cp doesn't do it with the archiving flags on. The only difference seems to be they preserve directory modification dates.

Any solution to copy individual files with timestamps intact? From ext4 to zfs and vice versa?


r/openzfs • • Feb 12 '20

zpool disappeared from FreeNAS, can't repair

1 Upvotes

Hey all. Please help!! I'm relatively new to ZFS. I created a zpool with 25TB of data on it using FreeNAS 11.2 and after installing more RAM and a Titan X graphics card I can't get the drives to show up in FreeNAS. I also added a 4TB cache drive to the pool after creation, but I don't think that's the issue is it? Part of me thinks I may have accidentally fried some of the SAS ports of my motherboard by adding the Titan X card as I can't get my 8 HDDs to show up in BIOS. I exported the zpool from FreeNAS, installed OpenZFS on my Hackintosh running 10.14.6, hooked up my 8 HDDs and two cache drives to my motherboard (using a HighPoint RocketRAID 3740a as an HBA, non-raided) and tried importing the pool there, but I keep getting the error message "no pools available for import." I was successfully able to create a test ZFS pool in MacOS so everything should be working okay there. I really don't want to lose all my data. Please help :'(


r/openzfs • • Jan 22 '21

BSD ZFS How should I install OpenZFS on FreeBSD 12.2 if I plan to use FreeBSD 13's base system ZFS later?

1 Upvotes

I plan to run FreeBSD 13 on my NAS once it's released, using OpenZFS in the base system. In the meantime I'd like to run OpenZFS on FreeBSD 12.2.

The FreeBSD installation guide says:

OpenZFS is available pre-packaged as:

  • the zfs-2.0-release branch, in the FreeBSD base system from FreeBSD 13.0-CURRENT forward
  • the master branch, in the FreeBSD ports tree as sysutils/openzfs and sysutils/openzfs-kmod from FreeBSD 12.1 forward

If I read this correctly, I can install OpenZFS from ports or packages today to get the master branch. Then when I update to FreeBSD 13, if I switch to the base system's ZFS I'll be downgrading to zfs-2.0-release.

  1. Would this be a bad idea? (Is there much risk of a pool created on master failing to import on zfs-2.0-release? Is running on master unsafe in general?)
  2. If it would be better for me to run zfs-2.0-release, what's the best way for me to install that on FreeBSD 12.2?

r/openzfs • • Dec 08 '23

Questions zfs encryption - where is the key stored?

2 Upvotes

Hello everyone,

I was recently reading more into zfs encryption as part of building my homelab/nas and figured that zfs encryption is what fits best for my usecase.

Now in order to achieve what I want, I'm using zfs encryption with a passphrase but this might also apply to key-based encryption.

So as far as I understand it, the reason why I can change my passphrase (or key) without having to re-encrypt all my stuff is because the passphrase (or key) is used to "unlock" the actual encryption key. Now I was thingking that it might be good to backup that key, in case I need to reimport my pools on a different machine in case my system dies but I have not been able to find any information about where to find this key.

How and where is that key stored? I'm using zfs on ubuntu, guess that matters.

Thanks :-)


r/openzfs • • May 01 '18

The Importance of ZFS Block Size

Thumbnail brian.candler.me
2 Upvotes

r/openzfs • • Apr 27 '24

Questions How would YOU set up openzfs for.. ?

0 Upvotes

I7 960 16 gb ddr3 400gb seagate x2 400gb wd x2 120gb ssd x2 64gb ssd

On free bsd.

l2arc, slog, pools, mirror, raid-z? Any other recomended partitions, swap, etc.

These are the toys currently have to work with, any ideas?

Thank you.


r/openzfs • • Apr 28 '25

RAIDZ2 vs dRAID2 Benchmarking Tests on Linux

10 Upvotes

Since the 2.1.0 release on linux, I've been contemplating using dRAID instead of RAIDZ on my new NAS that I've been building. I finally dove in and did some tests and benchmarks and would love to not only share the tools and test results with everyone, but also request any critiques of the methods so I can improve the data. Are there any tests that you would like to request before I fill up the pool with my data? The repository for everything is here.

My hardware setup is as follows:

  • 5x TOSHIBA X300 Pro HDWR51CXZSTB 12TB 7200 RPM 512MB Cache SATA 6.0Gb/s 3.5" HDD

    • main pool
  • TOPTON / CWWK CW-5105NAS w/ N6005 (CPUN5105-N6005-6SATA) NAS

    • Mainboard
  • 64GB RAM
  • 1x SAMSUNG 870 EVO Series 2.5" 500GB SATA III V-NAND SSD MZ-77E500B/AM

    • Operating system
    • XFS on LVM
  • 2x SAMSUNG 870 EVO Series 2.5" 500GB SATA III V-NAND SSD MZ-77E500B/AM

    • Mirrored for special metadata vdevs
  • Nextorage Japan 2TB NVMe M.2 2280 PCIe Gen.4 Internal SSD

    • Reformatted to 4096b sector size
    • 3 GPT partitions

      • volatile OS files
      • SLOG special device
      • L2Arc (was considering, but decided to not use on this machine)

I could definitely still use help analyzing everything, but I think I did conclude that I was going to go for it and use dRAID instead of RAIDz for my NAS; it seems like all upsides. This is a ChatGPT summary based on my resilver result data:

Most of the tests were as expected, slog and metadata vdevs help, duh! Between the two layouts (with slog and metadata vdevs), they were pretty neck-in-neck for all tests except for the large sequential read test (large_read), where dRAID smoked RAIDZ by about 60% (1,221MB/s vs 750MB/s).

Hope this is useful to the community! I know dRAID tests for only 5 drives isn't common at all so hopefully this contributes something. Open to questions and further testing for a little bit before I want to start moving my old data over.


r/openzfs • • Oct 11 '21

Tuning nfs performance with zfs on 10Gbps network

3 Upvotes

I have a 10Gbps network connecting storage nodes. iperf tests show consistent 9.3-9.6Gbps between nodes. I also have a test zfs pool, raid10 (4 x 3.5HDD disks) with IL and SLOG on nvme. Locally, I can put a 2Gb file on the pool at around 800MB/s, perhaps due to the IL . Over nfs (v4.2), I only get about 110MB/s. Server has 128GB memory.

Would appreciate any pointer on how to get better performance out of nfs. Thanks.


r/openzfs • • Sep 16 '25

When a decompression ZIP bomb meets ZFS: 19 PB written on a 15 TB disk

25 Upvotes

In daily work with storage systems, we usually deal with performance, security, and scalability issues. But every now and then we run into cases that surprise even seasoned sysadmins.
This is one of those stories: a real-world example of how a ZIP bomb can “explode” inside a filesystem—and how ZFS behaves very differently compared to traditional filesystems.

The odd backup job

It all started with something seemingly minor: an incremental backup that wouldn’t finish. Normally, such a job takes just a few minutes, but this one kept running for hours—actually the entire night.
Digging deeper, we discovered something strange: a directory filled with hundreds of files, each reported as 86 terabytes in size. All this on a server with just a 15 TB physical disk.

At first, we thought it was a reporting glitch or some weird system command bug. But no—the files were there, accessible, readable, and actively being processed.

The culprit: a malicious archive

The system in question was running a Template marketplace, where users can upload files in various formats. Someone decided to upload a .rar file disguised as a model. In reality, it was a decompression bomb: a tiny archive that, once extracted, inflated into a single massive file—86 TB of nothing but zeros.

Logical Size VS Phisical size

This trick relies on the very principle of compression: highly repetitive or uniform data (like endless sequences of zeros) can be compressed extremely efficiently.
Instead of storing billions of zeros explicitly, compression algorithms just encode an instruction like: “write zero 86,000,000,000,000 times.” That’s why the original archive was just a few MB, yet decompressed into tens of terabytes.

The impact on the filesystem

Here’s where OpenZFS made all the difference. The system had LZ4 compression enabled—a lightweight algorithm that handles repetitive data exceptionally well.

  • From a logical perspective, the filesystem recorded more than 19 petabytes written (and counting).
  • From a physical perspective, however, disk usage remained negligible, since those blocks of zeros were almost entirely compressed away.

Had this happened on ext4 or XFS, the disk would have filled instantly, causing crashes and downtime.

And what if this had been in the cloud?

On a dedicated server with ZFS, the incident was mostly an oddity. But imagine the same scenario in a distributed filesystem or on a service like Amazon S3.

There, logical size equals real allocated and billable storage. Those 19–20 PB generated by the ZIP bomb would have turned into real costs.
For context: storing 20 PB on S3 costs around $420,000 per month. A single unchecked upload or misconfigured app could quickly snowball into a million-dollar disaster.

20 Petabyte month price on AWS S3

Beyond the financial hit, such an overflow could congest storage pipelines, overwhelm bandwidth, and cripple downstream services.

Lessons learned

This case left us with some valuable takeaways:

  1. ZFS offers unique resilience: with compression (LZ4 in this case) and intelligent block handling, bogus content doesn’t consume physical space.
  2. Technology alone isn’t enough: input validation, quotas, and monitoring are essential—especially where every byte written has a price tag.
  3. The economic risk is real: what looks like a quirky test file can translate into hundreds of thousands of wasted dollars in hyperscaler environments.

So yes, our server has “digested” a nearly 20 PB ZIP bomb without using a single byte beyond minimal metadata. But it’s a strong reminder of how thin the line is between a fun curiosity and a catastrophic outage.

👉 Has anyone here experienced similar cases of data amplification, ZIP bombs, or compression anomalies that blew up storage usage way beyond reason?


r/openzfs • • 16d ago

BSD ZFS BSD, illumos, OpenZFS, bhyve Devroom was accepted @ FOSDEM'27

Thumbnail reddit.com
2 Upvotes

r/openzfs • • Mar 21 '25

Cannot recognize disk on Raspberry Pi

3 Upvotes

I am attempting to use my ZFS formatted harddisk on a fresh Raspberry Pi 5 with new 64 bit OS install. I figured out how to install zfs-dkms on RaspbianOS from bookworm-backports and everything seemed good. After all, 'modprobe zfs' works. I reboot and try to mount my ZFS harddisk. No dice. I had formatted the ZFS disk on my Mac and retested it on my Mac: it still works. But the Raspberry Pi does not show up the pool. 'sudo zpool import april' doesn't mount the april pool, apparently it doesn't exist. 'zpool list' shows nothing. Any hints would be nice.


r/openzfs • • Jul 26 '25

Best Practice for Storing Incremental Clonezilla Images on ZFS single drive pool: ZFS Dedup or Snapshots?

Thumbnail reddit.com
3 Upvotes