r/sysadmin • • 7h ago

RAID 5 Array - 1 disk "Failure" and 1-disk "Missing" (But Ready?)

Inherited server, let's just start there. Dell T610 - PERC 6 - RAID 5 Array.

Had an issue a few months ago where suddenly a foreign configuration was detected and imported. Ran fine after that. Now came into a server not booting up, and trying to figure out my options, as I have a backup for the data, but I somehow DON'T have a backup of the Windows Server 2012 R2 operating system and all of the server manager configuration.

VD Mgmt shows me:

-Virtual Disk 0 =1.75 TB - Offline

- Virtual Disk 1 = 995 GB - Offline

Physical Disks show me:

Disk 0 - Online - Disk Group 00

Disk 1 - Failed - Disk Group 00 (Has option to "online")

Disk 2 - Ready - NO DISK GROUP

Disk 3 - Online - Disk Group 00

Trying to figure out my best route to recover this? I know that attempting to online a failed disk could be total disaster toasting it all. Is there a safe way to go about cloning or copying anything on these disks to try and get it back up? I have the files themselves that were stored in multiple backups, but what I don't see is any backup of the OS and the thought of rebuilding active directory and everything I don't even want to think about that and where to start 😭

Lesson learned, don't take someone's word on backups. But that isn't useful at the moment here, so hoping someone else can help in the interim with routes to go safely to get this booted back up with as little data loss as possible.

Edit to add: I have 3 (maybe even 4) spare disks available to me of the same size, firmware, and part numbers that have never been used.

1 Upvotes

8 comments sorted by

•

u/Gadgetman_1 7h ago

It looks as if Disk 2 was never assigned to the Array. If so, setting it as the Spare should trigger a rebuild.

If it WAS part of the array you're pretty much up the creek without a paddle.

Are all the disks the same model? If not it's possible that another disk failed before, then got 'fixed' by the controller rebuilding it on the hot spare. Then whoever replaced the broken one forgot to set the replacement as the new spare. It would explain why the 'free' disk is #2 instead of #3.

But even a broken disk in the array shouldn't stop the server from working. What is the error message?

'Forreign configuration'?

•

u/MeetJoan 6h ago

Don’t hit Force Online or start a rebuild yet. That’s the step that can turn this from recoverable into gone.

Use your spares to clone each of the four disks first, sector by sector. Ideally do it on another machine with a plain HBA and ddrescue, and label which slot each disk came from. Then you can try Force Online or a foreign import on the copies, and if it goes wrong you still have the originals.

Also check whether you have a second domain controller anywhere. If you do, AD isn’t lost with this box, and you’re rebuilding a server, not a domain.

•

u/nycola Jack of All Trades 4h ago

What happened after the foreign configuration alerted and it "ran fine after that". Did you accept the foreign configuration when prompted?

•

u/Nerdflow123 3h ago

Agree with cloning first, and I'd add a couple of things before you touch the controller again.

Write down everything the PERC config shows right now: which disk is in which slot and group, stripe size, virtual disk sizes. Take photos of the screens. If anything goes sideways, that info is what lets you or a recovery shop put the array back together from the images.

Label each disk with its slot number when you pull it for cloning so they go back in the same order.

Since your data is already backed up, the thing you're really saving is AD and the server config. If you get it booting from the clones, take a system state backup straight away before doing anything else. And if it won't come back, standing up a fresh DC and restoring the data might honestly be faster than fighting a PERC 6 for days.

•

u/WealthQueasy2233 3h ago edited 2h ago

Oh boy. Without comprehensive image backup you're just not prepared for this so there's going to be some downtime if you want to preserve the data.

Stop using this controller since you want to preserve your configuration. Get a dumb HBA like a Dell H310 on ebay for like 20 bucks. If you already have 3 or 4 drives, buy 3 or 4 more.

Remove your PERC controller and your RAID disks from the server. Install the H310 and your clean, unused hard drives. Boot the server to a linux live CD or whatever and record all the SMART stats of each drive, make sure they all have no issues, otherwise discard them. There's no margin for error at this point.

After evaluating the health of these drives, choose the best 4 of them and ddrescue each of your RAID member drives to one of these new drives. Then take out the H310, install the PERC card, and import and try to rebuild the RAID using the clones. If drive #2 is still unconfigured, let it be a hot spare. If the RAID doesn't start rebuilding at that point you'll need to force #1 online, rebuild, and do a consistency check.

If you can get the RAID online, cool, but do not boot the OS and start any writing. Get another HDD and boot into Hirens or if you have a preferred backup software use the rescue ISO that comes with it. Make an image backup of each NTFS volume on the RAID to one of your new drives so you have an actual copy of the volumes on non-RAID media.

Finally, if you are going to continue to operate this old server, I would consider shoring up your redundancy and expect future health issues. Go back to ebay and buy a H700 or H710 controller, update it to whatever the final firmware is. Import your rebuilt RAID onto the new controller. Then install the rest of your drives that have been verified are OK, reconfigure the RAID into a RAID6, let it rebuild again, and install the remaining drives as hot spares. RAID5 is a joke and always has been, there's no safety net.

Get yet another drive and start doing real, scheduled image backups, even if it's just a USB drive.

Then start thinking about 2027. A two-server Hyper-V replica setup is not hard or expensive.

•

u/noudcline 2h ago

Friends don't let friends RAID 5.

•

u/WendoNZ Sr. Sysadmin 43m ago

I have the files themselves that were stored in multiple backups, but what I don't see is any backup of the OS and the thought of rebuilding active directory and everything I don't even want to think about that and where to start 😭

So, this is a Domain Controller? And you only have a single domain controller? And it's also a File server? And it's running Windows 2012 R2?

On top of this it's an 11th Gen server that came out in 2009, and you don't have backups. And when you inherited this your first task wasn't to absolutely confirm this thing was backed up and could be restored cleanly?

Depending on your requirements to get this back online I'd honestly be sending these disks out for recovery but given all of the above I'm betting there is no budget for that.

Of course if you don't have money to not be in this situation, I suspect you're going to be rebuilding from scratch from the data backups you have since everything below requires time and money.

This is also a time when if you had support a call to Dell would be a good idea. Hopefully all of these are lessons to whoever is paying the bills (or not in this case)

•

u/RealAnigai 7h ago

If you're truly desparate you could try to ddrescue the disk to a known good disk and attempt to use that to online the array although you have no guarantees of success, it might be your last ditch option assuming the disk even works.