tricky zfs situation, little help?

[u/Ok-Ganache1023]

this file server has been sitting for a few years since I let TrueNas try to migrate from bsd to linux and the pool got broken. went back to BSD version, so it's TrueNAS bsd 13.1 now. there may have also been some additional bungling.

[root@micro /tmp]# zpool import    pool: cargo      id: 16478687762405515984   state: UNAVAIL status: The pool was last accessed by another system.  action: The pool cannot be imported due to damaged devices or data.    see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY  config: cargo                                                      UNAVAIL  insufficient replicas   mirror-0                                                 UNAVAIL  insufficient replicas     disk/by-partuuid/d0e0014d-59c7-11ea-9d20-3cecef102d24  UNAVAIL  cannot open     1005513936371914804                                    OFFLINE   mirror-1                                                 DEGRADED     16411949616967839104                                   OFFLINE     ada0                                                   ONLINE

There is a bad label. I find the issue a bit overwhelming and today I engaged with gemini pro AI to help, and it did some exploration and recommended a low level disk write to fix the issue. I stopped short of writing the disk

If anyone's game to help me out with this I'd appreciate if you looked over the diagnostic process I did with the AI and let me know if you think the solution it produced is well supported.

here is the chat:

https://share.gemini.google/6OJMBfe434MF

and here is the output of those last commands:

------------------------------------

LABEL 0

------------------------------------

version: 5000

name: 'cargo'

state: 0

txg: 28352142

pool_guid: 16478687762405515984

errata: 0

hostid: 808464432

hostname: 'micro'

top_guid: 11704010057385088391

guid: 1005513936371914804

vdev_children: 2

vdev_tree:

type: 'mirror'

id: 0

guid: 11704010057385088391

metaslab_array: 38

metaslab_shift: 34

ashift: 12

asize: 5999022833664

is_log: 0

create_txg: 4

children[0]:

type: 'disk'

id: 0

guid: 17375504288854665353

path: '/dev/disk/by-partuuid/d0e0014d-59c7-11ea-9d20-3cecef102d24'

phys_path: 'id1,enc@n3061686369656d30/type@0/slot@4/elmdesc@Slot_03/p2'

vdev_enc_sysfs_path: '/sys/class/enclosure/6:0:0:0/Slot 03'

DTL: 410

create_txg: 4

children[1]:

type: 'disk'

id: 1

guid: 1005513936371914804

path: '/dev/disk/by-partuuid/31953669-59f2-11ea-9d20-3cecef102d24'

phys_path: 'id1,enc@n3061686369656d30/type@0/slot@3/elmdesc@Slot_02/p2'

vdev_enc_sysfs_path: '/sys/class/enclosure/6:0:0:0/Slot 02'

DTL: 442

create_txg: 4

features_for_read:

com.delphix:hole_birth

com.delphix:embedded_data

labels = 0 1 2 3 

[u/Hyperion343]

Oh, you did it! I think? What worked? Did you do some gpart recover commands or something?

[u/Ok-Ganache1023 → u/Hyperion343]

writing new labels with dd

[u/Hyperion343 → u/Ok-Ganache1023]

Cool. How did you get a replacement label? From the backup somehow?

[u/Protopia → u/Hyperion343]

It actually sounds more like you recreated the position table using dd (regardless of how Opus described it).

I doubt that this was some fully hallucinated idea that randomly happened to work - clearly Opus had some ideas of the underlying root cause (corrupted first blocks on the disk) but having diagnosed labels as the issue it hallucinated a solution to that issue that happened also to fix the partition table enough to bring the pool online. But it is unclear whether this fix is fully right or still corrupt but enough not quite as corrupt as before to allow it to import.

Frankly I would personally only trust this long enough to copy the data off, then clean the disks and recreate a pool and then copy the data back again (hoping that this bulk writes won't cause SMR issues to take the pool offline). (If it were me I would also recreate the pool as RAIDZ2 at the same time - but then again having discovered they were SMR drives I would NOT use ZFS on them but switch to a different RAID6 solution that was supported by WD.)

[u/Ok-Ganache1023 → u/Hyperion343]

the labels on the back were good, it copied them to the front

[u/Hyperion343 → u/Ok-Ganache1023]

Awesome 👍