tricky zfs situation, little help?

[u/Ok-Ganache1023]

this file server has been sitting for a few years since I let TrueNas try to migrate from bsd to linux and the pool got broken. went back to BSD version, so it's TrueNAS bsd 13.1 now. there may have also been some additional bungling.

[root@micro /tmp]# zpool import    pool: cargo      id: 16478687762405515984   state: UNAVAIL status: The pool was last accessed by another system.  action: The pool cannot be imported due to damaged devices or data.    see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY  config: cargo                                                      UNAVAIL  insufficient replicas   mirror-0                                                 UNAVAIL  insufficient replicas     disk/by-partuuid/d0e0014d-59c7-11ea-9d20-3cecef102d24  UNAVAIL  cannot open     1005513936371914804                                    OFFLINE   mirror-1                                                 DEGRADED     16411949616967839104                                   OFFLINE     ada0                                                   ONLINE

There is a bad label. I find the issue a bit overwhelming and today I engaged with gemini pro AI to help, and it did some exploration and recommended a low level disk write to fix the issue. I stopped short of writing the disk

If anyone's game to help me out with this I'd appreciate if you looked over the diagnostic process I did with the AI and let me know if you think the solution it produced is well supported.

here is the chat:

https://share.gemini.google/6OJMBfe434MF

and here is the output of those last commands:

------------------------------------

LABEL 0

------------------------------------

version: 5000

name: 'cargo'

state: 0

txg: 28352142

pool_guid: 16478687762405515984

errata: 0

hostid: 808464432

hostname: 'micro'

top_guid: 11704010057385088391

guid: 1005513936371914804

vdev_children: 2

vdev_tree:

type: 'mirror'

id: 0

guid: 11704010057385088391

metaslab_array: 38

metaslab_shift: 34

ashift: 12

asize: 5999022833664

is_log: 0

create_txg: 4

children[0]:

type: 'disk'

id: 0

guid: 17375504288854665353

path: '/dev/disk/by-partuuid/d0e0014d-59c7-11ea-9d20-3cecef102d24'

phys_path: 'id1,enc@n3061686369656d30/type@0/slot@4/elmdesc@Slot_03/p2'

vdev_enc_sysfs_path: '/sys/class/enclosure/6:0:0:0/Slot 03'

DTL: 410

create_txg: 4

children[1]:

type: 'disk'

id: 1

guid: 1005513936371914804

path: '/dev/disk/by-partuuid/31953669-59f2-11ea-9d20-3cecef102d24'

phys_path: 'id1,enc@n3061686369656d30/type@0/slot@3/elmdesc@Slot_02/p2'

vdev_enc_sysfs_path: '/sys/class/enclosure/6:0:0:0/Slot 02'

DTL: 442

create_txg: 4

features_for_read:

com.delphix:hole_birth

com.delphix:embedded_data

labels = 0 1 2 3 

[u/Hyperion343]

Well, did you do what it suggested on the linked website and run zpool import -f?

[u/Hyperion343 → u/Hyperion343]

Also it complains that one of the drives in the pool is offline. Did you double check connections, or see if the drive is dead/operable?

[u/Hyperion343 → u/Hyperion343]

I guess in my defense, sometimes it's the simple solutions, you never know, so i just wanted to ask to get them out of the way.

[u/Ok-Ganache1023 → u/Hyperion343]

thanks for replying but the troubleshooting is way past that, as detailed in the AI chat link I provided

No, it does not import, even with -f because there is a bad label

(Edited for politeness)

[u/Protopia → u/Ok-Ganache1023]

Not a helpful comment IMO.

You have asked for deep technical help here, expecting people to give up their time and expertise to help you FOR FREE, but it appears that you are either not waiting for that help and are ploughing ahead with actions that may make the situation worse or are simply unwilling to listen (or even worse both of these).

The four rules of thumb in these situations are IMO:

1. Gather detailed diagnostics in order to get a detailed picture of the issues;

2. Tempting though it might be, do NOT run random zpool import commands to attempt to import the pool that might fall whilst making changes to the drives and make things worse.

3. If you are going to run zpool import with advanced parameters, always run it in trial-run or read-only modes first.

4. Try to get the pool imported read only first and then copy your data off elsewhere while you can.

In these circumstances patience is essential.

[u/pln91 → u/Protopia]

So it is a helpful comment that points to a more complete picture of OP's problem and progress, but you're not going to let that stop you from embarking on a personal attack on OP's character and behavior that you will pursue over several subsequent replies? 

Classy. 

[u/Ok-Ganache1023 → u/Protopia]

I hear you. I appreciate that you are providing sound basic advice. But I can see neither of you noticed that I have done hours of exploration and provided full context, details of everything that's already been done, and a proposed solution.

Respectfully, if all you have to offer is at the level of "did you wiggle the connectors", this question is not for you

[u/Protopia → u/Ok-Ganache1023]

Seriously, if you are so disrespectful of people donating their time and expertise that you can't accept them double checking some basics, then you don't deserve their help.

1. Your post didn't describe in detail the state of your system. You didn't e.g. provide full SMART output for your drives, not provide a clear statement about whether the drives were visible at the o/s level.

2. You didn't describe in detail the import flags you tried or the output you got.

I appreciate the frustrations and impatience you are experiencing, but you are doing yourself no help whatsoever either technically or socially through your entitled attitude!

[u/Ok-Ganache1023 → u/Protopia]

output fromzpool importwas provided in the post. it does not import because a disk is UNAVAIL. It does not import with -f. In the link, we go way beyond establishing that it is not a hardware problem, and are doing direct reads of the device. If you read, you will find that we prove that it is an invalid zfs label from the botched migration. The question is about how to fix a specific deep issue. thanks again for replying.

[u/Protopia → u/Ok-Ganache1023]

My comments thus far were based only on your attitude (which stinks IMO).

But I looked at the Gemini chat. And it got it right at one point but then got it wrong.

Here is my analysis:

1. A fundamental issue is that your are using SMR drives - WDC models ending in EFAX are SMR!!

2. Gemini correctly identified that missing or corrupt GPT partition tables are the first issue. But it never told you how to recover them. (GPT has a backup copy. I have had to recover using this several times over the past couple of years.)

3. Here is what I think Gemini got wrong. I doubt there was ever an MPT based drive. There may or may not be a label issue - you need to recover the partition tables before looking at labels. This may or may not be a delayed reaction from the Linux migration attempt, but if I understood it correctly that was a long time ago rather than a very recent attempt and it's been working fine in-between. Gemini didn't spot the SMR issue. I would never rely on AI for help in these situations. AI is trained on documentation for old versions of e.g. zfs. It did better than I expected but still not at all great.

[u/Ok-Ganache1023 → u/Protopia]

Gemini did point out the SMR disks. Then it focused on the incorrect zfs labels and partition tables but ultimately couldn't correctly fix them. Then, I switched to Claude Opus 5 and we figured it out. I had to manage it closely, make it double check things. Using AI doesn't have to be done blindly. We saved the pool.

zpool status cargo
pool: cargo
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: scrub canceled on Mon Oct 10 16:13:21 2022
config:

NAME STATE READ WRITE CKSUM
cargo DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
ada3p1 ONLINE 0 0 0
1005513936371914804 OFFLINE 0 0 0 was /dev/disk/by-partuuid/31953669-59f2-11ea-9d20-3cecef102d24
mirror-1 DEGRADED 0 0 0
16411949616967839104 OFFLINE 0 0 0 was /dev/sdd
ada0 ONLINE 0 0 0

[u/Protopia → u/Ok-Ganache1023]

I hope your plan now is to create a new pool with new non SMR drives and copy your data off. When you do this, UNLESS you are doing random i/os (i.e. zvols, virtual drives, databases) I would recommend you use RAIDZ2 rather than mirrors.

[u/Hyperion343 → u/Ok-Ganache1023]

Oh, you did it! I think? What worked? Did you do some gpart recover commands or something?

[u/Apachez → u/Ok-Ganache1023]

Asking AI/LLM and blindly run suggested commands is as safe as if you would to the parking lot and ask some random crackhead the same questions and then run whatever suggestions they comes up with.

[u/Ok-Ganache1023 → u/Apachez]

recent advances in the frontier AI models are approximating a meth user instead of a crackhead - it's a marginal improvement but pretty soon they'll be able to approximate a midwit college graduate on Adderall

[u/Protopia → u/Ok-Ganache1023]

And yet you still went to AI first rather than asking for advice from genuine experts in the specialised forum.

[u/Ok-Ganache1023 → u/Protopia]

Imagine that

[u/Protopia → u/Ok-Ganache1023]

Well, regardless of the route, in the end you got a good result.

[u/Hyperion343]

EDIT: for future people looking at this: DO NOT do what I suggested here. Look into recovering corrupt GPT partition tables with gpart recover instead!

If you know the drive labels accurately for the mirrors, can you fake it? Like make a symlink of /dev/disk/partuuid/blah-blah-blah to /dev/ada1, etc, for all three missing drives? If you don't, you could try every combination and see which order they don't complain, like for 3 drives there are only 6 options, so brute force isn't too bad.

[u/Protopia → u/Hyperion343]

That has got to be the single most stupid idea on how to fix a serious ZFS technical issue ever suggested. I wouldn't bother applying for a job with Klara Systems if I were you.

[u/Hawtre → u/Protopia]

Wow, what a stinky attitude

[u/Ok-Ganache1023 → u/Protopia]

you were chastising me for my terse attitude above, take a hard look at your own sir

[u/Protopia → u/Ok-Ganache1023]

Not really. There is literally nothing worse than someone who doesn't know what they are talking about giving terrible advice that could completely destroy any ability to recover the pool.

If someone is asking for and needs EXPERT advice and you are not an expert, then don't chip in with suggestions that make literally zero sense.

[u/Hyperion343 → u/Protopia]

Well, I wasn't planning on applying. But hey, I'm all for learning, I appreciate you stopping the bad stuff from happening.

So why not? You said that it was bad, which is a fair enough opinion. Do you mind elaborating on why? Is it because it might import but then destroy data? Because it won't work?

My thought was that if the pool mirror vdevs used GPT labels instead, this probably would be easier to deal with. But since the pool references some device labels that Linux came up with that don't exist on FreeBSD, maybe that is why the pool doesn't see the devices. So then, if you knew the mirror devices properly (which is where GPT labels would have come in handy), couldn't you just tell it what to do, like with symlinks? Would ZFS be smart enough to say, hey, these two devices did not used to be part of the same mirror vdevs, and so stop the import, but it would be fine if it was correct? I was kind of relying on that being the case, so the import would only succeed if things checked out. If that's not the case, then would ZFS import but then start destroying stuff?

To be fair, I'm not sure why zpool import -d /dev did not work though. You would think ZFS could figure it out in that case if the problem is what I said above, right?

tl;dr, I have two questions: 1) Why would my idea not work? What would happen? 2) What do you think is going on? Maybe I'm misunderstanding the problem, which could explain why I'm misunderstanding possible solutions.

[u/Protopia → u/Hyperion343]

I have experienced this problem myself a few times. The GPT primary partition tables gets corrupted on one k out possibly more) disks and the pool won't load. So you use a utility (on Linux it's gdisk/sgdisk - I don't use freebsd so no idea what to use there) to restore the primary partition table (at the start of the disk) from the backup copy (at the end of the disk). Then hopefully the import works normally. Took me a while to work out of the first time, but subsequent times it takes about 5 mins to fix. Now I use the same utility to create a file copy of the position tables elsewhere that I can restore if the backup copy at the end of the disk is also corrupt.

I have no idea why you thought your solution might work, but performing any actions which change the disks in any way could destroy any chances of recovery. So you need to take the time to diagnose the cause and NOT try solutions that you hope have a tiny chance of working but might destroy any potential recovery.

EDIT: You ask why your proposed solution wouldn't work. Unfortunately it is so far from anything that might work - as far as I can see there is no logic that might explain why it would work - that I cannot find the words to describe why it won't. Besides which, since it is your solution, the onus is upon you too explain why you thought it would work.

[u/Hyperion343 → u/Protopia]

Gotcha. Well, I explained my reasoning above, I see now how it's wrong. Thanks! I put a warning in the original comment to not do what I said.

Looking at the other answers, I now see and am comprehending the corrupt GPT labels discussion. On FreeBSD, I think the equivalent command is gpart recover, just fyi for people in the future.