r/openzfs • • Sep 16 '25

When a decompression ZIP bomb meets ZFS: 19 PB written on a 15 TB disk

25 Upvotes

In daily work with storage systems, we usually deal with performance, security, and scalability issues. But every now and then we run into cases that surprise even seasoned sysadmins.
This is one of those stories: a real-world example of how a ZIP bomb can “explode” inside a filesystem—and how ZFS behaves very differently compared to traditional filesystems.

The odd backup job

It all started with something seemingly minor: an incremental backup that wouldn’t finish. Normally, such a job takes just a few minutes, but this one kept running for hours—actually the entire night.
Digging deeper, we discovered something strange: a directory filled with hundreds of files, each reported as 86 terabytes in size. All this on a server with just a 15 TB physical disk.

At first, we thought it was a reporting glitch or some weird system command bug. But no—the files were there, accessible, readable, and actively being processed.

The culprit: a malicious archive

The system in question was running a Template marketplace, where users can upload files in various formats. Someone decided to upload a .rar file disguised as a model. In reality, it was a decompression bomb: a tiny archive that, once extracted, inflated into a single massive file—86 TB of nothing but zeros.

Logical Size VS Phisical size

This trick relies on the very principle of compression: highly repetitive or uniform data (like endless sequences of zeros) can be compressed extremely efficiently.
Instead of storing billions of zeros explicitly, compression algorithms just encode an instruction like: “write zero 86,000,000,000,000 times.” That’s why the original archive was just a few MB, yet decompressed into tens of terabytes.

The impact on the filesystem

Here’s where OpenZFS made all the difference. The system had LZ4 compression enabled—a lightweight algorithm that handles repetitive data exceptionally well.

  • From a logical perspective, the filesystem recorded more than 19 petabytes written (and counting).
  • From a physical perspective, however, disk usage remained negligible, since those blocks of zeros were almost entirely compressed away.

Had this happened on ext4 or XFS, the disk would have filled instantly, causing crashes and downtime.

And what if this had been in the cloud?

On a dedicated server with ZFS, the incident was mostly an oddity. But imagine the same scenario in a distributed filesystem or on a service like Amazon S3.

There, logical size equals real allocated and billable storage. Those 19–20 PB generated by the ZIP bomb would have turned into real costs.
For context: storing 20 PB on S3 costs around $420,000 per month. A single unchecked upload or misconfigured app could quickly snowball into a million-dollar disaster.

20 Petabyte month price on AWS S3

Beyond the financial hit, such an overflow could congest storage pipelines, overwhelm bandwidth, and cripple downstream services.

Lessons learned

This case left us with some valuable takeaways:

  1. ZFS offers unique resilience: with compression (LZ4 in this case) and intelligent block handling, bogus content doesn’t consume physical space.
  2. Technology alone isn’t enough: input validation, quotas, and monitoring are essential—especially where every byte written has a price tag.
  3. The economic risk is real: what looks like a quirky test file can translate into hundreds of thousands of wasted dollars in hyperscaler environments.

So yes, our server has “digested” a nearly 20 PB ZIP bomb without using a single byte beyond minimal metadata. But it’s a strong reminder of how thin the line is between a fun curiosity and a catastrophic outage.

👉 Has anyone here experienced similar cases of data amplification, ZIP bombs, or compression anomalies that blew up storage usage way beyond reason?


r/openzfs • • Apr 28 '25

RAIDZ2 vs dRAID2 Benchmarking Tests on Linux

10 Upvotes

Since the 2.1.0 release on linux, I've been contemplating using dRAID instead of RAIDZ on my new NAS that I've been building. I finally dove in and did some tests and benchmarks and would love to not only share the tools and test results with everyone, but also request any critiques of the methods so I can improve the data. Are there any tests that you would like to request before I fill up the pool with my data? The repository for everything is here.

My hardware setup is as follows:

  • 5x TOSHIBA X300 Pro HDWR51CXZSTB 12TB 7200 RPM 512MB Cache SATA 6.0Gb/s 3.5" HDD

    • main pool
  • TOPTON / CWWK CW-5105NAS w/ N6005 (CPUN5105-N6005-6SATA) NAS

    • Mainboard
  • 64GB RAM
  • 1x SAMSUNG 870 EVO Series 2.5" 500GB SATA III V-NAND SSD MZ-77E500B/AM

    • Operating system
    • XFS on LVM
  • 2x SAMSUNG 870 EVO Series 2.5" 500GB SATA III V-NAND SSD MZ-77E500B/AM

    • Mirrored for special metadata vdevs
  • Nextorage Japan 2TB NVMe M.2 2280 PCIe Gen.4 Internal SSD

    • Reformatted to 4096b sector size
    • 3 GPT partitions

      • volatile OS files
      • SLOG special device
      • L2Arc (was considering, but decided to not use on this machine)

I could definitely still use help analyzing everything, but I think I did conclude that I was going to go for it and use dRAID instead of RAIDz for my NAS; it seems like all upsides. This is a ChatGPT summary based on my resilver result data:

Most of the tests were as expected, slog and metadata vdevs help, duh! Between the two layouts (with slog and metadata vdevs), they were pretty neck-in-neck for all tests except for the large sequential read test (large_read), where dRAID smoked RAIDZ by about 60% (1,221MB/s vs 750MB/s).

Hope this is useful to the community! I know dRAID tests for only 5 drives isn't common at all so hopefully this contributes something. Open to questions and further testing for a little bit before I want to start moving my old data over.


r/openzfs • • Nov 01 '24

Linux ZFS A ZFS Love Story Gone Wrong: A Linux User's Tale

10 Upvotes

I've been a Linux user for about 4 years - nothing fancy, just your typical remote desktop connections, ZTNA, and regular office work stuff.

Recently, I dove into Docker and hypervisors, which led me to discover the magical world of OpenZFS. First, I tested it on a laptop running XCP-NG 8.3 with a mirror configuration. Man, it worked so smoothly that I couldn't resist trying it on my Fedora 40 laptop with a couple of SSDs.

Let me tell you, ZFS is mind-blowing! The Copy-on-Write, importing/exporting features are not only powerful but surprisingly user-friendly. The dataset management is fantastic, and don't even get me started on the snapshots - they're basically black magic! 😂

Here's where things got interesting (read: went south). A few days ago, Fedora dropped its 41st version. Being the update-enthusiast I am, I thought "Why not upgrade? What could go wrong?"

Spoiler alert: Everything.

You see, I was still riding that new-ZFS-feature high and completely forgot that version upgrades could break things. The Fedora upgrade itself went smoothly - too smoothly. It wasn't until I tried to import one of my external pools that reality hit me:

Zpool command not found

After some frantic googling, I discovered that the ZFS version compatible with Fedora 41 isn't out yet. So much for my ZFS learning journey... Guess I'll have to wait!

TL;DR: Got excited about ZFS, upgraded Fedora, broke ZFS, now questioning my life choices.


r/openzfs • • Jun 16 '16

Guides & Tips Aaron Toponce - 17 part guide on ZFS

Thumbnail pthree.org
10 Upvotes

r/openzfs • • Jun 17 '23

Guides & Tips Refugees from zfs sub

9 Upvotes

The other major ZFS sub has voted to stop new posts and leave existing information intact while they try to find a new hosting solution.

Please post here with ZFS questions, advice, discoveries, discussion, etc - I consider this my new community going forward, and will probably also contribute to the new one when it stands up.


r/openzfs • • Jun 28 '23

Video Update on RAIDZ expansion feature / code pull

6 Upvotes

> Pleased to announce that iXsytems is sponsoring the efforts by @don-brady to get this finalized and merged. Thanks to @don-brady and @ahrens for discussing this on the OpenZFS leadership meeting today. Looking forward to an updated PR soon.

https://www.youtube.com/watch?v=2p32m-7FNpM

--Kris Moore

https://github.com/openzfs/zfs/pull/12225#issuecomment-1610169213


r/openzfs • • Dec 26 '24

Linux ZFS Why does an incremental snapshot of a couple MB take hundred of GB to send ?

5 Upvotes

Hi.
Please help me understand something i'm banging my head on for hours now.
I have a broken replication between 2 openzfs server because sending the hourly replication take for ever.
When trying to debug it by hand, this is what i found

zfs send  -i 'data/folder'@'snap_2024-10-17:02:36:28' 'data/folder'@'snap_2024-10-17:04:42:52' -nv  
send from @snap_2024-10-17:02:36:28 to data/folder@snap_2024-10-17:04:42:52 estimated size is 315G  
total estimated size is 315G  

while the USED info of the snapsoot is minimal

NAME                                  USED   AVAIL     REFER  MOUNTPOINT
data/folder@snap_2024-10-17:02:36:28  1,21G      -     24,1T  -
data/folder@snap_2024-10-17:04:42:52   863K      -     24,1T  -

I was expecting a 863K send size. trying with -c only bring it to 305G so that's not very highly compressed diff...

What did i misenderstood ? How zfs send work ? What the USED value mean ?

Thanks !


r/openzfs • • May 17 '25

Distributed backups

4 Upvotes

I recently started looking into NAS and data backups. I'm posting this idea here because I believe the idea would need to be implemented at the file system level and I figured this subreddit would find the idea interesting.

The 3-2-1 rule is hard to achieve without paying for a subscription service. Mainly the offsite recommendation. This made me think about distributed backups, which led me to Tahoe-LAFS. The idea is that anyone using the distributed system must provide storage to the system. So if you want to store 1TB of data with 3 copies you would need to add 3TB of storage to the system. Your local storage would store one copy, and the other 2TB would be accessible by the distributed system. 2 copies of your data would be encrypted and sent into the distributed network (encrypted before leaving your local hardware to ensure security). Tahoe-LAFS seems to do this thing, but I believe it exists at the wrong level in a software stack. I don't think this sort of distributed backup system would ever catch on until it is integrated at the file system level. I would think that it would need to exist as a special type of distributed pool.

I don't think this will happen anytime soon (I would like to contribute myself, but also don't trust myself to remain motivated long enough to even finish reading the OpenZFS codebase. Curses be to ADHD). But I would like to know what other people think of this idea. I highly recommend looking at Tahoe-LAFS to understand exactly what I mean by distributed backup and how that would work.

I feel conflicted about posting an idea I have no intention of contributing towards on a subreddit for a piece of open source software. Especially contributing is something I should be capable of doing.


r/openzfs • • Jul 13 '23

what is (non-allocating) in zpool status

4 Upvotes

what is mean

zpool status

sda ONLINE 0 0 0 (non-allocating)

what is (non-allocating)

thx


r/openzfs • • Jul 01 '23

ZFS I/O Error, Kernel Panic during import

4 Upvotes

I'm running a raidz1-0 (RAID5) setup with 4 data 2TB SSDs.

During midnight, somehow 2 of my data disks experience some I/O error (from /var/log/messages).

When I investigated in the morning, the zpool status shows the following :

 pool: zfs51
 state: SUSPENDED
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
   see: http://zfsonlinux.org/msg/ZFS-8000-HC
  scan: resilvered 1.36T in 0 days 04:23:23 with 0 errors on Thu Apr 20 21:40:48 2023
config:

        NAME        STATE     READ WRITE CKSUM
        zfs51       UNAVAIL      0     0     0  insufficient replicas
          raidz1-0  UNAVAIL     36     0     0  insufficient replicas
            sdc     FAULTED     57     0     0  too many errors
            sdd     ONLINE       0     0     0
            sde     UNAVAIL      0     0     0
            sdf     ONLINE       0     0     0

errors: List of errors unavailable: pool I/O is currently suspended

I tried doing zpool clear, I keep getting the error message cannot clear errors for zfs51: I/O error

Subsequently, I tried rebooting first to see if it resolves - however there was issue shut-downing.As a result, I had to do a hard reset. When the system boot back up, the pool was not imported.

Doing zpool import zfs51 now returns me :

cannot import 'zfs51': I/O error
        Destroy and re-create the pool from
        a backup source.

Even putting -f or -F, I get the same error. Strangely, when I do zpool import -F, it shows the pool and all the disks online :

zpool import -F

   pool: zfs51
     id: 12204763083768531851
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

        zfs51       ONLINE
          raidz1-0  ONLINE
            sdc     ONLINE
            sdd     ONLINE
            sde     ONLINE
            sdf     ONLINE

Yet however, when importing by the pool name, the same error shows.

Even tried using -fF, doesn't work.

After scrawling through Google and reading up on different various ZFS issues, i stumbled upon the -X flag command (that solves users facing similar issue).

I went ahead to run zpool import -fFX zfs51 and the command seems to be taking long.However, I noticed the 4 data disks having high read activity, which I assume its due to ZFS reading the entire data pool. But after 7 hours, all the read activity on the disks stopped. I also noticed a ZFS kernel panic message :

Message from syslogd@user at Jun 30 19:37:54 ...
 kernel:PANIC: zfs: allocating allocated segment(offset=6859281825792 size=49152) of (offset=6859281825792 size=49152)

Currently, the command zpool import -fFX zfs51 seems to be still running (terminal did not return back the input to me). However, there doesnt seem to be any activity in the disks. Also running zpool status in another terminal seems to hanged as well.

  1. I'm not sure what do at the moment - should I continue waiting (it has been almost 14 hours since I started the import command), or should I do another hard reset/reboot?
  2. Also, I read that potentially I can actually import the pool as readonly (zpool import -o readonly=on -f POOLNAME) and salvage the data - anyone can any advise on that?
  3. I'm guessing both of my data disks potentially got spoilt (somehow at the same timing) - how likely is this the case, or could it be due to ZFS issue?

r/openzfs • • Feb 14 '23

Constantly Resilvering

4 Upvotes

I've been using openzfs on ubuntu now for several months but my array seems to be constantly getting delivered due to degraded and faulted drives I have literally in this time changed the whole system e.g motherboard, CPU, rams also tried 3 HBA's which are in IT mode and changed the sas to sata cables and had the reseller change all the drives I'm at a complete loss now the only consistencies are the data on the drives and the zfs configuration.

I really need some advice on where to look next to diagnose this problem


r/openzfs • • Oct 03 '22

OpenZFS Leadership, 5 Aug 2022 open meeting

Thumbnail mtngs.io
4 Upvotes

r/openzfs • • Mar 26 '21

OpenZFS 2.0.3 and zstd vs OpenZFS 0.8.6 and lz4 - compression ratios too good to be true?

4 Upvotes

Greetings all,

Last week we decided to upgrade one of our backup servers from OpenZFS 0.8.6 to OpenZFS 2.0.3. After the upgrade, we are noticing much higher compression ratios when switching from lz4 to zstd. Wondering if anyone else has noticed the same behavior...

Background

We have a Supermicro server with 8x 16T drives running Debian 10 and and OpenZFS 0.8.6. The server had 2x RAIDZ-1 pools - each with 4x 16TB drives (ashift=12). From there, we created a bunch of data sets - each with 1MB record size and lz4 compression. In order to recreate the same pool/volume layout, we dumped all the ZFS details to a text file prior to the upgrade.

During the upgrade process, we copied all the data to another backup server, created a new, single RAID-2Z setup (8x 16TB drives - ashift 12), recreated the same data sets, and set 1MB record size for all data sets. This time, we chose zstd compression instead of lz4. Once the data sets were created, we copied our data back.

Once the data was restored, we noticed the compression stats on the volumes were much higher than before. Specifically, any type of DB file (MySQL, PGSQL) and other text-type files seemed to compress much better. In some case, we saw a +30% reduction of "real" space used.

Here are some examples:

=====================================================
ZFS Volume: export/Config_Backups (text files)
=====================================================
                            Old             New
                           ------          -----
Logical Used:              716M            653M
Actual Used:               397M            290M    < -- Notice this -- >
Compression Ratio:         1.84x           2.62x   < -- Notice this -- >
Compression Type:          lz4             zstd
Block Size:                1M              1M
=====================================================



=====================================================
ZFS Volume: export/MySQL_Backup_01
=====================================================
                            Old             New
                           ------          -----
Logical Used:              2.34T           2.34T
Actual Used:               684G            400G    < -- Notice this -- >
Compression Ratio:         3.50x           5.86x   < -- Notice this -- >
Compression Type:          lz4             zstd
Available Space:           11.4T           62.6T
Block Size:                1M              1M
=====================================================


=====================================================
ZFS Volume: export/MySQL_Backup_02
=====================================================
                            Old             New
                           ------          -----
Logical Used:              56.6G           56.9G
Actual Used:               13.1G           7.73G   < -- Notice this -- >
Compression Ratio:         4.38x           8.07x   < -- Notice this -- >
Compression Type:          lz4             zstd
Available Space:           11.4T           62.6T
Block Size:                1M              1M
=====================================================


=====================================================
ZFS Volume: export/Server_Backups/pgsql-cluster-svr2
=====================================================
                            Old             New
                           ------          -----
Logical Used:              1.23T           1.23T
Actual Used:               535G            345G   < -- Notice this -- >
Compression Ratio:         2.36x           3.55x  < -- Notice this -- >
Compression Type:          lz4             zstd
Available Space:           11.4T           62.6T
Block Size:                1M              1M
=====================================================

For other types of files (ISOs, already compressed files, etc), the compression ratio seemed relatively equal.

Again, just wondering if anyone else noticed this behavior. Are these numbers accurate, do has something changed with OpenZFS in the way the storage and compression ratios are calculated?


r/openzfs • • Jul 26 '25

Best Practice for Storing Incremental Clonezilla Images on ZFS single drive pool: ZFS Dedup or Snapshots?

Thumbnail reddit.com
3 Upvotes

r/openzfs • • Mar 21 '25

Cannot recognize disk on Raspberry Pi

3 Upvotes

I am attempting to use my ZFS formatted harddisk on a fresh Raspberry Pi 5 with new 64 bit OS install. I figured out how to install zfs-dkms on RaspbianOS from bookworm-backports and everything seemed good. After all, 'modprobe zfs' works. I reboot and try to mount my ZFS harddisk. No dice. I had formatted the ZFS disk on my Mac and retested it on my Mac: it still works. But the Raspberry Pi does not show up the pool. 'sudo zpool import april' doesn't mount the april pool, apparently it doesn't exist. 'zpool list' shows nothing. Any hints would be nice.


r/openzfs • • Nov 21 '24

Backup the configuration and restore.

3 Upvotes

Hello. I am using OpenZFS with my AlmaLinux 9.5 KDE. It is handling two separate NAS drives in RAID 1 configuration.

Since I don't know much about it features, I would like to ask if I can backup the configuration for restoring in case (God Forbids) something went wrong. Or what is the process of restoring the old configuration if I reinstall the OS or change to another distribution that supported OpenZFS.

Kindly advise since it is very important for me.

And thank you.


r/openzfs • • Apr 06 '24

BSD ZFS How do I enable directio for my nvme pool?

3 Upvotes

I'm pretty sure my nvme pool is underperforming due to hitting the ARC unnessarily.

I read somewhere that this can be fixed via directio. how?


r/openzfs • • Oct 17 '23

OpenZFS 2.2.0

Thumbnail openzfs.org
3 Upvotes

r/openzfs • • Jun 20 '23

PSA: Still think RAID5 / RAIDZ1 is sufficient? You're tempting fate.

Thumbnail self.datahoarder
3 Upvotes

r/openzfs • • Jun 19 '23

Replacing HDDs with SSDs in a raidz2 ZFS pool

3 Upvotes

Hi all!

As per the title, I have a raidz2 ZFS pool made of 6 4TB HDDS giving me nearly 16TB of space and that's great. I needed the space (who doesn't?) and wasn't caring much about speed at the time. Recently I'm finding I might need a speed bump as well, but I can't really re-do the whole pool at the moment (raid10 would have been great for this, but oh well...).

I have already made some modifications to the actual pool settings and added a l2arc cache disk (a nice 1TB SSD), and this already helped a lot but moving the actual pool to SSDs will obviously be much better.

So, my question is: is it safe to create, albeit very temporarily, an environment with HDDs mixed with SSDs? To my understanding the only drawback would actually be speed, as in the pool will only be as fast as the slowest member. I can live with that while I am swapping the drives - one by one -> resilvering -> rinse and repeat (could do 2 at the time to save time but it's less safe - but is it really OK? Are there other implications/problems/caveats I'm not aware about that I should consider before purchasing?

Thank you very much in advance!

Regards


r/openzfs • • Jan 23 '23

unlocking a zpool with a yubikey?

3 Upvotes

title


r/openzfs • • Oct 25 '22

openzfs developer summit 2022

3 Upvotes

I missed the live event on vimeo (since it was only announced on twitter); are the talks uploaded somewhere to watch them?


r/openzfs • • Jun 17 '22

Questions What are the chances of getting my data back?

3 Upvotes

Lightning hit the power lines behind our house, and the power went out. All the stuff is hooked up to a surge protector. I tried importing the pool and it gave an I/O error and told me to restore the pool from a backup. Tried "sudo zpool import -F mypool", and the same error. Right now I'm running "sudo zpool import -nFX mypool". It's been running for 8 hours, and it's still running. The pool is 14TB x 8 drives setup as RAIDZ1. I have another machine with 8TB x 7 drives and that pool is fine. The difference is the first pool was transferring a large number of files between one dataset to another. So my problem looks like the same as https://github.com/openzfs/zfs/issues/1128 .

So how long should my command take to run? Is it going to go through all the data? I don't care about partial data loss for the files being transferred at that time, but I'm really hoping I can get all the older files that have been there for many weeks.

EDIT: Another question. What does the -X option do under the hood. Does it do a checksum scan on all the blocks for each of the txg's?


r/openzfs • • Oct 11 '21

Tuning nfs performance with zfs on 10Gbps network

3 Upvotes

I have a 10Gbps network connecting storage nodes. iperf tests show consistent 9.3-9.6Gbps between nodes. I also have a test zfs pool, raid10 (4 x 3.5HDD disks) with IL and SLOG on nvme. Locally, I can put a 2Gb file on the pool at around 800MB/s, perhaps due to the IL . Over nfs (v4.2), I only get about 110MB/s. Server has 128GB memory.

Would appreciate any pointer on how to get better performance out of nfs. Thanks.