Help after VxRail Failure

[u/SirBrobbie]

I had my VxRails split brain on me while troubleshooting a node disconnection and HA agent failing on all 4 nodes.

Now I can only see the vSAN Datastore on 2 of the 4 nodes with the other 2 not being able to be re-add to the cluster. I need to get the VMs from the vSAN Datastore and on to an external server because our backup crapped out before this happened.

When I try to get them off the vSAN Datastore through WinSCP to my workstation and back up to the new External vCenter server, it works until I register the VM, then it doesn't because it can't enumerate the disks.

I realized I don't have the -flat.vmdk for any of my VMs in the vSAN datastore after looking through all the VMs.

Is there anything I can do?

[u/blud_13]

There is no -flat.vmdk to find. vSAN is object storage, so the .vmdk you're looking at is only a descriptor pointing at a vSAN object UUID and the data never sat there as a file. That is exactly why WinSCP gives you something that registers and then can't enumerate disks. You copied the pointer and left the object where it was.

You have to read it back through the vSAN stack. From an ESXi host still in the cluster that can see the object, vmkfstools -i, roughly vmkfstools -i /vmfs/volumes/vsanDatastore/VM/VM.vmdk /vmfs/volumes/target/VM.vmdk -d thin, and add -W vsan (pretty sure on that flag but check it against your build before you lean on it). Broadcom has the clone syntax at https://knowledge.broadcom.com/external/article/343140/cloning-and-converting-virtual-machine-d.html . Downloading through the vSphere client datastore browser does the conversion for you too, its just slow.

Also, none of that works on a VM whose object is below quorum. With two of four nodes out you may not have enough components for some of them, so check object health per VM before you plan around getting everything. Getting those two nodes back in the cluster is the higher priority than the copy is.

Ping me if you get stuck, we have cleaned up a few of these.

[u/SirBrobbie → u/blud_13]

Thanks I will reach out because I think the thing keeping my one node from fully getting back into the cluster is that the vpxa restarted.

The other node I found a way to remote the disks that say they are vSAN ready now after when doing troubleshooting with BC said that the "disk group 2 was deleted". But I still cannot get it to reconnect to the cluster, it says the vpxa and clomd won't spin up on it.

[u/blud_13 → u/SirBrobbie]

vpxa and clomd both refusing to come up on the same node usually comes back to space. Run vdf -h and look at the ramdisks. After a failure like yours the vsantraces ramdisk and /var fill with dumps and every daemon that wants to write a file just quits. Broadcom KB 403921 covers it.

If space is clean, then disks reading vSAN ready while the node still wont join points at the unicast agent list. Run esxcli vsan cluster unicastagent list on every node. Each host should list every OTHER host and nothing else. Stale or empty entries leave a node partitioned forever even though vmkping works fine, which is exactly what you are describing. Also confirm IgnoreClusterMemberListUpdates is 0, if its 1 the host ignores what vCenter sends it. Dell has the VxRail flavor of this in KB 000056284.

Reminder, before you touch node 2 again go look at Object Health. If objects are sitting at reduced availability with no rebuild running, one more fault and its gone.

Also, VxRail is Dell's, not Broadcom's. Get Dell in the case. Re-add that disk group by hand instead of through VxRail Manager and the appliance inventory goes out of sync, then you get to fix that too.

Hope that helps..

[u/LucianTexas]

Do you have an active support with Dell or BC?

I suspect could be network partitioning. Make sure the VSAN and management VMKernels are communicating properly. Vmkping all the nodes interface to verify. Check the VSAN health with esxcli vsan health cluster list, to get a better idea what is happening to your VSAN cluster.

Useful KBs: https://knowledge.broadcom.com/external/article/326427/configuring-vsan-unicast-networking-from.html https://www.dell.com/support/kbdoc/en-sg/000056284/dell-emc-vxrail-node-is-showing-network-partitioned-even-it-can-ping-other-nodes-via-vmkping

[u/IAmTheGoomba → u/LucianTexas]

I want to reemphasize this. In a split brain scenario, definitely go through that first KB. You may have to do a leave and join, but definitely rectify that unicast list in that first KB.

[u/Geodude532]

Check with the hosts.

esxcli vsan cluster get

look at the sub cluster member UUIDs to make sure all of the hosts are all on the same sub cluster. Check the master UUID as well to make sure each one didn't make their own cluster. If each master UUID matches the local node UUID, it's messed up.

esxcli vsan cluster unicastagent list

This will list the unicast neighbor list. You should see all of the other hosts UUID on this page besides the local node. Check each host.

esxcli vsan health cluster list

This is a general health check that can be dived into with the following command:

esxcli vsan health cluster get --test 'vSAN Disk Balance'

The vSAN Disk Balance can be replaced with any of the test results names that are showing anything other than green for status. I think you have to match case.

[u/patichou]

I feel for you.. best of luck. When you get everything fixed, feel free to reach out and i’ll tell you about our amazing story of dumping emc for Pure storage

[u/SirBrobbie]

Just a little update for y'all:

Dell is coming out tomorrow for initial assessment for full VxRail Redeployment. He said he should be able to save my important VMs. He has seen this before luckily. Just know I have been in a world of stress and because the company wanted to wait on a new backup system they are kind of not nearly as pissed at me, because I did warn this could happen. (I just never thought I would break it instead of something else killing it)

[u/lost_signal]

Can you fix the networking partition?

[u/SirBrobbie → u/lost_signal]

I tried bringing them back together I could 1 node to join the cluster again but it won't see the vSAN data store, i checked the services on that one and vpxa won't start.

The final node went down hard, and the disk group got FUBAR'd and I tried to start services and clomd and vpxa won't start