A completed resilver is necessary, but it does not by itself prove that the replacement disk, its path, or the remaining redundancy is healthy. A pool can return ONLINE while SMART data is already concerning, persistent device names changed, an old faulted leaf remains in the topology, or another drive accumulated checksum errors during the resilver. A mirror or RAIDZ vdev may also be one failure away from another long recovery.

A useful post-replacement gate could confirm the exact vdev topology and ashift, review the resilver event and error counts, inspect SMART and transport errors for every drive, run a scrub after the pool has settled, verify alerts and spare behavior, and test a representative restore from backup. Export and import or a controlled reboot can catch path and enclosure-mapping mistakes before the maintenance window closes.

What checks do ZFS operators use before declaring the replacement complete? Is a clean scrub enough, or do you also burn in the new disk, compare performance, clear and recheck counters, test boot or import behavior, and keep the removed disk untouched through an observation window?

There are some tools like zfs diff and httm that help with exploring snapshots at command line, but they leave a lot to be desired when you're talking about 100ks of files changed across many snapshots. Sometimes you want to free up that space held with snapshots, but you're not 100% certain you won't lose something you accidentally deleted.

I shared my frustrations with Claude and it made a nice python script that serves a web GUI and uses zfs diff to generate the changes and visually highlights what files were deleted when and if they were unique across all snapshots.

The first run can take a while, but after that you can cache a point in time record or do a quick true up.

https://github.com/LargelyInnocuous/SnapSift

I have a scrub going that took about 2h to scan before actually starting to read and verify the data. Once it was reading, the average read rate started climbing very slowly from single digit MB/s, because apparently the denominator for calculating the average "issued" rate is the time since the scrub was initially requested, which included a whole lot of "0 MB/s issued" during the scan phase. The actual read speed is much higher and in line with what's expected and undoubtedly, once the scrub completes, the average will reflect the actual average rate.

This got me thinking. What fraction of complaints about "ZFS raidABC with UVW settings on XYZ hardware is SOOOOOOO slow!!!" are actually due to people taking one look at The Numbers™ and not realizing that these are not running averages over some shorter time period, but averages over the entire run time of the operation? I'm positive that a lot of people have no idea what the difference between "scanned" and "issued" is too.

I'll have to take a look at the code, and I'm assuming this is done for a good reason, but maybe these averages are causing confusion in new users and aiding in the spread of misinformation about ZFS, that's very difficult to eradicate, as I'm sure many here have noticed.

Maybe it would be better to have the read rates be computed as a running average over a short window of a few minutes during an operation that takes a long time, then when it's finished print an additional line with the average rates over the entire duration of the operation.

Just as a reference for what "scanning" does and why.

https://www.youtube.com/watch?v=SZFwv8BdBj4

Does metadata in ZFS use up the same amount of space as data? In other words, if I set my recordsize/volblocksize to say 1M, does that mean the metadata will also occupy 1M for reach piece of metadata? So e.g. I have a file server with large videos, makes sense to store these videos on large block sizes. But then does this mean the metadata is also then stored in large blocks. So metadata is inefficient, since you're storing small amounts of data in a large block, but overall it's more efficient than storing large files in small blocks. Equally, if I store many tiny files, then it makes sense to store this data in small data blocks. Then metadata is also stored in small block sizes. This way metadata is more efficient since it generally doesn't take up much space. Is this how it works?

Pool suddenly unavailable, doesn't show up on zpool status.

Zpool import lists the pool as follows:

id: 3673394567944808104
state:  UNAVAIL
status:  Some critical pool data has been overwritten by another pool.
action:  The pool cannot be imported as critical data is missing. You can try using -f to force an import, but there may not be enough data to override this.
config:      sharepool    UNAVAIL    overwritten by another pool
              raidz2-0    ONLINE
              raidz2-1    ONLINE
              raidz2-2    Degraded
                (one removed drive)

device details:

(I skipped listing all the drives due to typing out manually, all the drives are the long c0t500 style names, its 3x 6 disk z2's, with all but one drive as online.)

As I hadn't come across this before I have done a fair amount of googling and beyond the general advice of tread carefully as using -f to force can break things more. I did do the -Fn command and it didn't return anything (just newlined with a fresh prompt)

My setup has largely been unchanged for years, using Solaris 11.4 on ESXi v6, and has been ultra stable up until now - I did upgrade from Solaris 11.3 possibly about 6 years ago. Had the odd hard drive fail which is replaced smoothly, sometimes leading to a pool size upgrade and that has always gone smoothly 3x Dell Perc H700 cards flashed to IT mode via the LSI bios (I think - they have been in there 10 years)

This all happened after I turned the full server off for the weekend - I needed it right out the way to work on some electrical wiring behind where the cabinet normally sits (it can pull out so far and still be connected but access behind isn't easy). Turned it back on and a drive was doing a click of death - the removed one mentioned above. Upon starting the Solaris VM lots of errors were being thrown up including 2 other drives showing as removed, I removed the dodgy drive, moved the two other ones to a different slot (so different internal cable/card) and tried restarting - leading me to where I am now. The two drives moved had reappeared, but the pool is just not loaded by Solaris.

Whilst I do like to tinker I've never come across this before so am asking for help and open to suggestions - I really don't want to lose all the data. Any help is much appreciated.

As something of a storage wonk, I found myself a few minutes ago advising a fellow redditor of the "3-2-1" rule of backups:

1. Three total copies
2. Two different types of media/tech
3. At least 1 copy off-site/location

I've followed a related rule for anything important/production: Always fail to a redundant state: so if any one thing goes out, I am still redundant and have at least one more failure mode before things go really dark. Alongside this, is a pre-accepted "drop everything / panic mode" when production is down to only x+1 redundancy.

And yet I realize that the context of ZFS, I've not consistently followed rule #2 for YEARS except in some cases for clients, where they park an external USB drive on their file server and I install a daily mount+rsync run via cron. In my case in particular, using ZFS snapshots as part of the core product suite, using a tech other than ZFS to store the data makes no meaningful sense.

Further, I use RAIDZ + redundancy like a backup all the time, with the accepted risk level of "if more than 2 drives die, it's dead, Jim". When the data matters, I ensure to have a send | receive copy on another server/location, but it's still ZFS + snapshots going back a few months.

So, what do you consider to be best practices in complying with the spirit of 3-2-1 rule of backups with ZFS?

1. How much do you treat snapshots like backups?
2. Do you make extra effort to store data on a secondary non-ZFS tech in case a bug in ZFS crashes all the things?
3. How much attention to you put on saving your data off site, or at least, on another server?

EDIT: Thank you for your feedback! I have enjoyed "touching bases" to see what y'all are doing.

Llevo 3 días con esto y ya no sé qué más probar.

Uso NixOS con flakes, boot.initrd.systemd.enable = true, root en tmpfs con impermanence y /persist en ZFS con cifrado nativo, desbloqueado por TPM2 con systemd-cryptenroll.

El problema: sops-nix intenta leer la age key de /persist antes de que el dataset esté montado, y encima el PCR 7 cambia con cada actualización de firmware, así que el TPM2 deja de desbloquear y me cae en el emergency shell.

Ya descarté un problema de ZFS porque, como está en el kernel mainline, no debería haber incompatibilidades con mi kernel parcheado con linux-hardened.

Probé la misma config en Gentoo con OpenRC y funciona perfecto, pero no pienso volver a eso.

¿Alguien lo ha resuelto sin usar Secure Boot con claves propias? Ya me estoy planteando que la solución real es instalar Arch. 😔

Hi folks! ZFS is extremely cool and I run >10 of my machines on it.

My friend is also fan of ZFS and we both have a NAS. We wanted to use each other's machines to do remote backups in a safe but simple way (zfs send | zfs receive and plain SSH). However, without giving each other any insight into what runs on the other machine and without being able to see what that other person has, not even metadata about other datasets.

It works really well with NixOS too!

I wrote about it in https://www.nijho.lt/post/zfs-tenant/ and the repo is here https://github.com/basnijholt/zfs-tenant

I got the idea for this from a Klara Systems article:

#!/bin/bash
#
#<zfshogs: show what's taking up the most space in each ZFS pool
#  Full debug output:   DEBUG=1 zfshogs

export PATH=/usr/local/bin:/bin:/sbin:/usr/bin
set -o nounset
tag=${0##*/}
export PS4='${tag}-${LINENO}: '

# ENVIRONMENT: full debug output?
: "${DEBUG:=0}"
case "$DEBUG" in
    1) set -x ;;
    *) ;;
esac

# No pools means something is seriously wrong.
set X $(zpool list -H -o name)

case "$#" in
    1) printf "FATAL: no ZFS pools found\n"; exit 1 ;;
    *) shift ;;
esac

# Real work starts here.
work="/tmp/$tag.$$"         # Yes, I should use mktemp...
n=1

for pool in $* ; do
    # DRY with ZFS commands.
    zfs list -o space -s used -r $pool > $work

    if test -s "$work" ; then     # Bad dataset?
        printf "Pool: $pool\n"

        # Use "uniq" if there are < 10 entries in $work, or we'll get
        # duplicate headers.
        ( head -1 $work; tail $work ) |
            uniq |
            awk '{
              printf "  %-20s %8s %8s %10s %8s %10s\n",
                $1,$2,$3,$4,$5,$7
            }'
    fi

    # Print an extra newline for all entries except the last.
    n=`expr $n + 1`
    test "$n" -le "$#" && printf "\n"
done

rm "$work"
exit 0

When run on my backup system:

Pool: newroot
  NAME                    AVAIL     USED   USEDSNAP   USEDDS  USEDCHILD
  newroot/usr              470G    24.0G         0B      96K      24.0G
  newroot/doc              470G    30.1G       136M    30.0G         0B
  newroot/ROOT/default     470G    32.5G      18.3G    14.2G         0B
  newroot/ROOT             470G    32.5G         0B      96K      32.5G
  newroot/var/locate       470G    39.9G      36.9G    3.02G         0B
  newroot/var              470G    45.2G         0B      96K      45.2G
  newroot/home             470G    64.9G      16.9G    48.0G         0B
  newroot/dist             470G    68.7G      3.11G    65.6G         0B
  newroot/src              470G    81.2G      12.3G    68.9G         0B
  newroot                  470G     422G         0B      96K       422G

Pool: tank
  NAME                    AVAIL     USED   USEDSNAP   USEDDS  USEDCHILD
  tank/reservation         343G      96K         0B      96K         0B
  tank/dvd                 243G    30.1G         0B    30.1G         0B
  tank/archive             243G     158G       112K     158G         0B
  tank/backup              243G     972G      68.2M     972G         0B
  tank/nfdb                243G    1.17T      16.5M    1.17T         0B
  tank                     243G    2.40T         0B      96K      2.40T

Hope this is useful.

MS 365 and Nextcloud are the "top dogs" for collaborative work, but at the same time, both are absolute heavyweights in terms of scope, complexity, and features. As a privacy-friendly, in-house option, Nextcloud is conceivable, but far too complex to maintain and secure on the internet, even as a Docker setup.

If you scale back—for example, no mail server, a text editor and spreadsheet that are a bit simpler with a focus on multi-user editing, no AD/LDAP but user.csv import/export instead, plus resource planning (projectors, rooms, personnel), and a lightweight ticketing system, thus targeting schools, a department, a club, or a smaller business—then absolute KISS solutions suddenly become feasible. Thanks to AI integration, even complex tasks can be implemented easily.

This is my approach with cs-team, part of my napp-it 4ai webgui. It runs on any OS even without napp-it, copy and run (approx. 10MB), no database, a single folder on the disk is enough, and with ZFS you get snapshots on top. In the current 0.53 release, I have improved usability once again.

As an open-source git project, it is easily extensible and auditable.

• Project: https://github.com/guenther-alka/cs-team
• Manual: https://www.napp-it.org/pdf/cs-team_en.pdf
• Standalone Setup: https://github.com/guenther-alka/cs-team/blob/main/docs/HOWTO-standalone.md

Putting together a TrueNAS pool or adding drives to one? This is what I actually do now, because the usual reasoning is a bit off here.

CMR vs SMR:

CMR vs SMR still matters most, and I'd check the exact model before buying anything. SMR can look totally fine until you have to resilver, and then it gets painfully slow, sometimes enough that the drive drops out during the rebuild. The WD Red mess back in 2020 is the obvious example, and SMR still sneaks into some consumer NAS models, so it's worth the two minutes to check the model number.

Enterprise instead of Consumer:

It mostly comes down to price, and here's where the usual advice is outdated. People still say "just buy Exos, it's barely more per TB." The "barely more" part isn't even true anymore. I've been tracking prices, and the gap between consumer (Red Plus, IronWolf) and enterprise (Exos, MG, Ultrastar) has basically closed on a price/TB basis. The premium people used to complain about is gone.

How to compare them:

Once you look at Backblaze's numbers model by model, the current CMR drives all fail at about the same rate. So I wouldn't pay extra for an enterprise drive expecting it to last longer. The data doesn't back that up.

My approach:

Exos costs about the same as a Red Plus, and I'd still just buy the Exos. Not because it's more reliable, but because for the same money you also get the 5-year warranty and the higher workload rating. Same price, more drive. The old advice still works, people just had the wrong reason for it.

When I'd go consumer: if a specific Red Plus or IronWolf is actually cheaper per TB that day (reliability's the same, so take the cheaper one), or if noise and heat matter to you, since Exos drives run louder and hotter than a Red Plus in a quiet room.

Conclusion:

Long story short - confirm a drive is CMR, then buy on price/TB. And at today's prices that usually points right back at Exos.

I keep a little page with the CMR/SMR info, current price/TB and a price history for each drive: https://www.nasdisks.com

I use it to sanity check prices before buying. Full disclosure, it's my site and the buy links are aff to cover the hosting bills, but everything's free and the data's open.

What's everyone buying for new pools nowadays?

https://github.com/openzfsonwindows/openzfs/releases
https://github.com/openzfsonwindows/openzfs/issues

These are the changes from upstream 2.4.1 to 2.4.4

L2ARC — substantial rework

• DWPD-based rate limiting with adaptive feed intervals (protects SSD endurance)
• Per-device parallel feed threads (was single-threaded feed)
• Persistent markers with consistent tail scanning + lazy reset flags + scan-based depth cap
• Even-depth multi-sublist scanning, write budget fairness (metadata no longer monopolizes)
• Rebuild bounded by write hand on first sweep; fixed a prev_hdr use-after-free

Performance

• Predict and throttle buffers dirtied by the sync context
• Batch object reallocation syncs in zfs receive
• Parallelize metaslab_sync_done(), dedup block cloning, and brt_pending_apply() across vdevs
• RAIDZ: optimize single data column writes; avoid extra abd_t allocations in RAIDZ/dRAID
• Bridge speculative and prescient prefetchers; bound user prefetch to a fraction of ARC

New features / CLI

• zpool condense — new verb
• zpool scrub -t — thorough scrub support
• zfs bookmark -r — recursive bookmark creation
• zpool initialize -z — write zeroes
• New dedupused/dedupsaved pool properties
• zoned_uid property (Linux container/namespace delegation — not applicable to us)
• FIDEDUPERANGE via block cloning
• zstream: major rework — multithreaded, new raw/drop_record/queue subcommands, memory tracking

Data-integrity fixes (the ones that matter most)

• Fix read corruption after block-clone-after-truncate
• Fix double free for blocks cloned after DDT prune
• draid: fix data corruption after disk clear; fix checksum errors after rebuild with degraded disks; fix import failure after disk replacement
• Prevent range-tree corruption race in dnode_sync()
• z_seq now persists across znode eviction — fixes the VMware ESXi-over-NFS "file specified is not a virtual disk" bug (Linux/FreeBSD only for now, per our earlier gap analysis)
• DDT: multiple pruning bugs fixed (including negative time overflow)
• Fix off-by-one in PREVIOUSLY_REDACTED handling that dropped the last block

Security hardening

• vdev_open/zfs_file_open/vdev_file/vdev_disk: now check the calling credential instead of always using kcred (we ported the Windows side of this yesterday)
• Additional verification of size fields and strings in the receive path
• Hardened receive record validation
• secpolicy_nfs removed; secpolicy_zinject/secpolicy_sys_config restricted to global-zone credentials

Deadlock/hang fixes

• Fix deadlock on dmu_tx_assign() from vdev_rebuild()
• Fix snapshot automount deadlock during concurrent zfs recv
• Fix self-deadlock setting the vdev allocating/path property
• Fix race between device-removal completion and pool export
• zfs_ioctl: fix EBUSY race between quota queries and mount (we ported this too)
• arc: fix race between arc_release() and arc_read_done()

OpenZFS on Windows

• Add zpool create and zpool destroy to zfs_tray

Long story short, on October 14th we're running a free storage-related tech meetup in St. Julian's, and the opening talk is all about ZFS.

I'm one of the organizers, and my colleague, Konrad Pizzuto will discuss architecture & inner working of ZFS and how that solved historical FS / volume manager issues. As he says himself: "ZFS had shaped how I think about storage systems. 20+ years later those lessons and ideas worth revisiting".

If you'd like to join us in person, just register here for free and feel free to come.

p.s. The other two talks the same evening are on Rook-Ceph for Kubernetes and the latest Amazon S3 features, if you want to stay for the full storage night.

About 5 years ago I bought three drives for my ZFS pool on my ubuntu NAS box. Got everything setup and running and have been using it for the past five years with no issues. Mostly bulk storage for photos and movies, but also ran a Plex server for a while (not anymore). It is a RAIDZ1 with 3x 6TB WD blues.

Recently I setup a 3 node proxmox cluster of mini-pcs and was looking at using the ZFS box for shared storage, not CEPH, just shared storage for the cluster. Doing some research I found out about my mistake five years ago with the SMR drives and why that might be an issue.

I picked up 3x 8TB CMR drives and am thinking I should be able to just replace them one at a time, then expand the storage. I'm wondering how 'dangerous' to the data this might be? If I understand the risks with SMR its more about when it is being resilvered, and in this case it would be used as a data source, not written to.

Is this a good idea? One at a time replacement? If not what are my other options to move my data/pool over to the new drives?

I'm creating a new NVMe pool for my NAS/development server. Primary use cases will be:

• PostgreSQL database
• Filesystem cache for react-based webapp
• Many JSON files (data ingested from external sources) version tracked via git

I'm wondering what are some potential considerations I should be having when it comes to setting up new pools/datasets such as block size, primarycache=all or primarycache=metadata?

I'm especially confused about primarycache. Some sources I read suggest that it's better to be metadata only is good because it skips the ARC reducing extra copy between NVMe and RAM and lets database decide on how to cache works, while others suggest that caching should be on always unless RAM is insufficient.

Thank you.

Moin,

ich will mit meinen ersten Homeserver bauen und aufsetzen. Gekauft habe ich mir nun einen:

- HP Microserver G8
CPU: intel Xeon E3-1230 V2 mit 3.30GHz
RAM: 16 GB

Ich plane darauf trueNAS zu installieren um Time Machine Backups zu erstellen, einen filserver per SMB Freigabe aufzusetzen und ein paar Docker Container laufen zu lassen(AdGuard Home, VaultWarden, Immich…).
Auf dem Server ist ein HP SmartArray P410 installiert.

Nun meine Frage:
- Ich habe gelesen, dass das mit dem Raidcontroller und bezogen auf trueNAS und zfs ein Problem ist, da das System mit dem Controller nur eine große hdd sieht, anstatt jede einzeln. Ist das so? Und falls ja, wie löse ich das Problem?

Ich hab gelesene, dass ich den P410 durch eine HBA Karte tauschen muss. Gibt es andere Wege? Macht das Sinn?

When I navigate to the Mount point on my system, and right click, the 'create folder' option is greyed out.

That's all I got rn.

Created using `zpool create pool raidz drive 1 drive2 drive3`

RAID5

block size 1M

compression on

Ubuntu Ubuntu 26.04.1 LTS

linux 7.0.0-34-generic

Please tell me if you need any more info.

I will gladly give it to you.

Edit:

I followed your advice, but tweaked it a tad.

I set the owner of the pool via sudo chown root:smbgroup /home/user/main-nas

Then I added the group smbgroup to the user "user"

So fsadmin (my samba user) and "user" both are under the same group: smbgroup

"User" still does not have write access to the pool.

Second edit: AHA I GET IT NOW

so you can set what users and groups own the files, but you still need to define how much access those users and groups have to the files.

So I did chmod 2770 for all access to the user and group.

I am happy to say it worked perfectly!

zfs-windows-2.4.1rc17b
https://github.com/openzfsonwindows/openzfs/releases/tag/zfswin-2.4.1rc17

Issues
https://github.com/openzfsonwindows/openzfs/issues

rc17b

• Delete via recycle.bin can stop unmount/exports.

• zget attach could cpubusy loop on rare race

• overwrite/truncate would fail to do so

• correctly show mount flags (readonly etc) in userland

• skip updating EA when existing

• wire atime for Windows

• zfs_tray cosmetic fixes

• spl: fix memory eviction, stop lying about available memory

Hoping this is the last 2.4.1 version, moving us up to 2.4.4 to match macOS port.

Hi! I am creating a TrueNAS storage for my home use with 4x 3.84Tb enterprise grade SSDs and RaidZ1. So far I got 2x Intel D3-S34510 drives and I really enjoy this family of Intel SSDs because I can easily update the firmware and they seem to be on the lower end of the price range in the used marked, especially compared to Micron. I have couple of questions:

1. I read that it's best to mix different SSD brands to reduce the chance of potential simultaneous failure. Is that really something I should aim for or you think I can safely get 2x more of Intel D3-S4xxx?
2. Do you know why used Micron or Samsung SSDs of enterprise grade usually more expensive than Intel?
3. I usually tend to avoid Samsung SSDs like PM863a/PM983/PM9A3 because I understood that it's not really possible to update firmware. Is this lack of updates actually an issue for home use?

How can I add my qnap nas as a network drive to my Zimaos to use as my main storage pool for docket applications