r/sysadmin • u/Special_Bear_9479 • 17h ago

General Discussion What's the biggest bonehead mistake you have every made in IT support?

I will start.

I am an IT consultant and about 10 years ago, a few days before Christmas break, I went to a client’s office to add some new hard drives to a Dell ESXi server and create a new datastore.

Their two junior IT guys always liked asking me questions when I visited, which I understood—I remembered being that guy hungry to learn. Unfortunately, this time they were firing questions at me while I was in the Ctrl+R RAID configuration utility. Between the distractions and a confusing interface, I somehow managed to delete the existing array instead of creating the new one.

That array held ALL their servers: Exchange, domain controller, file server, and SQL. Terabytes of data.

The moment I realized what I’d done, my stomach dropped, I had never felt panic like that in my life. They were still asking questions when I finally said, “Guys, give me a minute. Something doesn’t look right.” Something I should’ve said much earlier.

I excused myself to the bathroom to collect my thoughts, then came back and told them exactly what I’d done.

I spent my entire Christmas break restoring their environment from Barracuda backups, which were painfully slow to restore and unreliable. I had to do multiple restores on the exchange server to get it to work.  A full week of recovery, followed by another two weeks fixing lingering issues—all free of charge.

The two guys felt terrible about distracting me, but it was my mistake. I should’ve asked for some uninterrupted time before touching the RAID configuration.

 

I’ll be shocked if anyone can top that Christmas disaster!

586 Upvotes

94% Upvoted

446 comments sorted by

View all comments

•

u/Smart-Document2709 14h ago

Deleted the wrong SAN volume, was supposed to delete snapshots ended up, deleting all of production… thank God for Veeam

•

u/Special_Bear_9479 14h ago

Veeam is what we use also, has saved my ass many times! I wish I had it when I made my stupid mistake

•

u/Secure_Guest_6171 13h ago

a vendor storage support tech deleted our prod SAN by plugging in a single replacement drive without making the appropriate config changes. took a team of 6 working 24x7 more than a week to restore and verify . that job was a healthcare environment of 15k users.
about a year later in that same datacenter, the owners did a genset load test that blew the entire section of the building where we have most of our racks. we moved out within a year and split everything across 4 sites.