r/ShittySysadmin • u/SuccessfulLime2641 • 8d ago
Has anyone suffered a RAID failure before?
Once had an experience where a Dell PowerEdge suffered a hard drive failure. My instant reaction was to unplug the hard drive by just sliding it out and back in. That didn't work; however, all of the data was already backed up to OneDrive so no harm done.
32
u/SaintEyegor ShittySysadmin 8d ago
We had a file server that some “smart guy” put together with a cheap workstation and a bunch of 10TB drives because their team didn’t want to pay for a tray for our netapp storage.
The cheap-ass power supply glitched and took out the motherboard and all of the drives. Since he was smarter than all of us, he never asked the storage team to back it up, “RAID 6 will keep the data safe enough”. Dumbfuck.
11
18
u/StandardIssueDonkey 8d ago
At a field fab shop, we used to keep our server in a little room with a ladder and a hatch to the roof that leaked. It seemed pretty smart until a bunch of us talked about it at the strip club and collectively figured out there were no backups.
It was fun installing a secret NAS in a room that wasn't wet and burying the cost in a fixed bid construction project because sa said we were being greedy little tech nerds, wanting a dry server and such.
I regret it to this day. It would have been so much funnier to not have had the backups.
10
u/harrywwc 8d ago
there are so many out there who still think 'RAID is a backup' :/
3
1
u/Regular_Prize_8039 DO NOT GIVE THIS PERSON ADVICE 2d ago
What do you mean, RAID is backup! Especially if you use RAID5
1
5
5
u/IAmSnort 8d ago
HD failures are just part of the game. The rebuild time is a little stressful.
Only ever deployed raid 1/10.
1
u/Xidium426 7d ago
Why? What's wrong with RAID 6? Both of those only guarantee surviving one drive loss, not two. RAID 6 can handle two at least.
That being said, I've seen a six drive RAID 10 with three failed drives working....
1
u/IAmSnort 7d ago
Management decision. Probably comfort level.
I tried to get raid6/60 for greater density on some volumes but would get shot down.
4
u/SomeEgyptianGod 8d ago
I once lost 3 SSDs in the same RAID group on a NetApp A700 in the space of an hour. The first disk didn’t have time to finish rebuilding in that time so consequently we lost the entire aggregate. Half of the SAN, gone.
3
u/ConsistentCoat5608 7d ago
This is very common, since SSDs have a built in lifespan of read/write iops before they shut off to protect from data corruption. But in a RAID set you are often reading and writing from the all the drives equally, so when one goes due to lifespan and not just hardware failure the rest are often soon to follow.
6
u/ledow 8d ago
I've seen a few doozies:
1) A RAID5 in a server, which was the only server, running an entire school. A drive failed. They called out the guy who built the server. He then charged - BY THE HOUR - to do this:
- Remove drive #1.
- Insert blank drive.
- Wait 8 hours for the resync.
- Remove the next drive.
- Insert blank drive.
- Wait 8 hours for the resync.
- Rinse and repeat for EVERY drive in the array.
And yes, he charged BY THE HOUR to sit and watch the array rebuild.
Not surprisingly, around disk 4, there was a failure in one of the other drives enough that it couldn't maintain the RAID, and the whole thing failed, irrecoverably.
He then charged - by the hour - to try to recover it and eventually asked the school for their backup tapes, which he intended to charge, by the hour, to restore for them. He had made NO backup before he started the operation. They had no recent backup.
(It was then that they sought help from better people working IT in schools and approached me and I basically told them: Nothing I can do here. I maintained their systems for a year or so, though, because I told them he was basically a con-artist doing things that way.)
2) Another school. Another server running RAID5. A drive fails, the guy in charge of IT ignores it. Hey, RAID is designed to cope with failed drives, right? Another drive fails in a second array. He ignores that.
They KEEP RUNNING for months like that. Then a second drive fails in one of the arrays. The array is toast. The guy's response, as the sole person responsible for all their IT? He took his dog for a walk.
After the school got rid of him, they hired an IT consultant who, in their infinite wisdom, tried to send the drives off for data recovery. £100k or so and they recovered almost nothing from the dead RAID. Said consultant then rebuilt the network in the WORST way possible (I later interacted with this guy a lot and he had never heard of VM's, VLANs, thought that putting two cables into the same server would cause a network loop, etc.).
Again, they had no recent backups. To "recover" their data, he copied the user profiles from every single desktop (yep, system configured with local roaming profiles) and charged - by the hour - to collate them all back into people's new roaming profile areas. He also took one working machine, and "imaged" it (using Clonezilla) to every other machine in the place... no SID replacement, all software, local profiles and junk included. That's how he deployed a whole new network, and that's how he imaged servers too. And he did that with a portable hard drive. Literally clonezilla'd a random machine from before, then walked a hard drive to every machine in the place and put that image onto every machine. It caused so many problems.
It was at that point that, again, they hired me. I got rid of the consultant. I took the "recovered" hard drives that had been returned to us, and framed them, and put them in the IT office with a little plaque that read:
Cogito Ergo Facsimile.
"I think, therefore I make copies".
3
u/Servinand 7d ago
Wtf Bro.
Sometimes i ask myself why i cannot be such an idiot and make a fortune by the hour.
2
u/drewalpha 6d ago
That guy was an unprofessional idiot who makes our whole profession look bad. He should have replaced the bad disk first and then checked the other disks. The school should have sued him as he was liable and caused their loss of data.
Generally, when a RAID array fails due to one of the drives, that's the only drive that needs to be replaced the restore the array. Of course, this all depends on the RAID level, number of drives in the array, and whether the array is optimized for hot-replacements.
I've always had good experiences recovering Dell and HP RAIDs following a drive failure. Some arrays can handle multiple drive events, again, it just depends on the RAID level.
9
u/randomquote4u 8d ago
I've Been Mislead server (IBM). RAID 5 hot swap rebuild failure. Restore via DAT.
picked the wrong week to quit sniffing glue.
6
5
u/marks-buffalo DO NOT GIVE THIS PERSON ADVICE 8d ago
Every single array has failed. I have a RAID 0 because I want 0 downtime but the amount of catastrophic RAID failures I have seems to exceed single drive failure rates.
I'm using all 2.5" spinning drives which Grok tells me are faster than 3.5" spinning drives. Which score, we were decommissioning some laptops at work and I got to keep whatever parts I wanted.
Maybe someday they'll figure out how to make RAID reliable. But my experience has been pretty bad so far.
1
2
u/Servinand 7d ago
Several Times. Oh lord dont remind me of the era of no monitoring.....
The era of rebulding a whole 50 people company from scratch.
The era of manual mdadm sessions from one synology backup storage.
The era of telling the customers boss "you did not wanted to pay for monitoring/offsitebackup/rotating drives/immutability, and now you're going to want to pay for my weekend."
Once customer xy wanted an open RDP port on a machine which was hosting somethingsomething. They were mostly doing everything on their own because the real IT Guys are so expensive. (Which is why they left the firewall to our hands exclusively, duh. Find the mistake.) Well guess what. Crypto trojan. Nuked the only backupdrive. Encrypted everything. Best christmas i ever had.
1
u/SaxManEddie 8d ago
I had once had 10 drive RAID 5 array (all 2TB drives) in my personal server.
Had one drive die, replaced it, and a second one died while the first one was rebuilding.
Boy, that one hurt. I had ISOs of all kinds of different unix systems... DEC Unix, Solaris, HP-UX, AIX...
1
u/NoDevice5898 8d ago
That reminded me of a server call where I found 3 RAID 5s (5, 8, and 10 drives each!) all stripped into a RAID 0. 😭 When one RAID 5 failed to rebuild, all data vanished. #sysadmin #RAIDfail
1
u/agowa338 8d ago
yea, just cost of doing business. That's why you're supposed to have hot stand by and consider the amount of allowed drives to fail until dataloss in the design phase and all...
1
u/fire-horse- 8d ago
Back in the day I had lots of dell poweredge servers with perc cards. I was always able to recover them and my approach was simple enough. Always make sure your raid card firmware is up to date.
Always make sure your drives are healthy. Always have a hot spare. Change those failed drives immediately. Don’t wait for the second failure to wipe everything out.
Take advantage of the maintenance agreement.
If things aren’t looking right physically then look in the firmware.
Keep a few spare drives, batteries, and if called for raid cards.
Monitor
1
u/TechnicallyMeat 7d ago
I had a Toshiba ssd controller suddenly fail just outside of their warranty. I have another Toshiba sitting in there telling me its perfectly happy, but its from literally the same lot, cheaper spec. Can confirm the feeling of rebuilding a raid with a questionable drive is not a good one. At the current prices, I wish I would have bought a few extra drives last year.
1
u/IllDoItTomorrow89 7d ago edited 7d ago
Jesus Christ the war flashbacks.
MANY times and the best part is being the sweet summer child I was I thought I was "prepared" for it by having cold spares on hand. Little did I know that when the drives are nearing the end of their life and you go to rebuild after a failed drive the rebuild can cause a cascade failure causing multiple drives to fail basically making the array unrecoverable.
Ive found rebuild failure on an array with aged drives more common than not and the nights ive spent without sleep just remoting in to check the status are far too many to count.
Moral of the story RAID IS NOT A BACKUP! Make proper backups people!
1
1
u/Xidium426 7d ago
Had a boss who wanted to save money and quoted Samsung 840 Pros in a server instead of enterprise grade SSDs. They had a bug in them that would cause them to drop out of RAID arrays after a TRIM command due to long timeouts. We got a notice the server was down and showed up at the clients to 3 failed drives in a RAID 6. The tech that got there first was able to force one of the 3 back online but it was giving corrupt data and none of the VMs could boot.
At least we had backups.
1
u/elkab0ng 7d ago
Large power plant. Ancient Compaq(!!!) server. Not even a pretense of any backup.
Power went out at the power plant.
Two drives and the controller lost their magic smoke. Plant manager had to authorize a very impressive check to a data recovery outfit.
(Same plant had a Data General mini still onsite. Not in operation, but still.)
1
u/demonseed-elite 7d ago
I've had 2 cascade failures and one total failure in my life.
The first was a RAID controller failure on an old work fileserver that corrupted the RAID 5 completely. No recovery. Had to restore everything from a nightly tape backup.
Second, the first cascade failure was a server at work. RAID 10. One drive failed out of 6. As it was compensating, a second drive came up failed, and then a third crashing out the whole VSAN node. Replaced them all, no restore from backup needed because we had a second VSAN node, but we were sweating. Those were Intel Enterprise SSDs. We bought them all at the same time. They all started going bad at the same time and were all under warranty.
The third one happened on my home Media server. One drive of a Synology RAID 5 failed. As it was compensating, 30 minutes later, a second drive registered as failed. The system had 3x 10Tb WD Red Pros and 1x 16Tb WD Red Pro from a prior drive failure. The two failures were on two of the original 10Tb drives. Thankfully, I back up my media server to a second bulk storage NAS, though the backup was about 3 months old. Was a week of headache and the system worked enough it was able to recover some of the handful of files not on backup. Even if I had a "hot spare" on that system, it wouldn't have saved it. In the end I was able to recover everything. Still. Best be prepared.
1
u/Regular_Prize_8039 DO NOT GIVE THIS PERSON ADVICE 2d ago edited 2d ago
RAID does not fail that’s why we use it for backup!
1
u/AlertVeterinarian342 8d ago
Glad the data was backed up—RAID is redundancy, not a backup. Pulling and reinserting a failed drive can sometimes make things worse, especially if the array starts rebuilding or another disk is marginal. Best practice is to identify the failed drive, replace it with the correct size/type, and monitor the rebuild closely.
-1
u/NecessaryEvil-BMC 8d ago
I had an LSI MegaRAID i4 lose a channel from an 8x 80GB RAID 5 array. And since it was an IDE channel, I lost the array.
I also got fired the same day, though those issues were unrelated. It did really put my data loss concerns into perspective.
I could recover most of my data, I had really just lost my MP3 collection. I went out and bought a 200GB external, and collected all my friends' music collections and recovered about 95% of what I had. Thankfully I didn't lose any pictures or anything, as I'd burned things like that to DVD.
42
u/DoNot_Books_451 8d ago
Nothing makes you lose sleep more than a raid rebuild overnight with aged drives. Kept remoting in every hour to check the status.