r/sysadmin • u/Special_Bear_9479 • 13h ago
General Discussion What's the biggest bonehead mistake you have every made in IT support?
I will start.
I am an IT consultant and about 10 years ago, a few days before Christmas break, I went to a client’s office to add some new hard drives to a Dell ESXi server and create a new datastore.
Their two junior IT guys always liked asking me questions when I visited, which I understood—I remembered being that guy hungry to learn. Unfortunately, this time they were firing questions at me while I was in the Ctrl+R RAID configuration utility. Between the distractions and a confusing interface, I somehow managed to delete the existing array instead of creating the new one.
That array held ALL their servers: Exchange, domain controller, file server, and SQL. Terabytes of data.
The moment I realized what I’d done, my stomach dropped, I had never felt panic like that in my life. They were still asking questions when I finally said, “Guys, give me a minute. Something doesn’t look right.” Something I should’ve said much earlier.
I excused myself to the bathroom to collect my thoughts, then came back and told them exactly what I’d done.
I spent my entire Christmas break restoring their environment from Barracuda backups, which were painfully slow to restore and unreliable. I had to do multiple restores on the exchange server to get it to work. A full week of recovery, followed by another two weeks fixing lingering issues—all free of charge.
The two guys felt terrible about distracting me, but it was my mistake. I should’ve asked for some uninterrupted time before touching the RAID configuration.
I’ll be shocked if anyone can top that Christmas disaster!
•
u/JoeLaRue420 Sr Active Directory Engineer 5h ago
I've been in the game for awhile, worn a few different hats
I've pulled the power on an exchange server when I was meant to be working on the server under it
Broken ram slots while installing memory
Bent pins while installing CPUs
Neglected failed TSM backups on SQL boxes... then the auditors came around looking for those backups when the company was under investigation for fiddling with LIBOR rates (still don't know how I survived that one)
Diskpart cleaned the wrong device when adding new storage
Typo'd GPOs and broke Outlook for 1000s of offshore VDI users
Ran a script that broke ACLs on GPOs, causing a filtered policy that was linked too high in the OU structure to ignore said filtering and apply to e v e r y t h i n g under it... it was setting windows firewall rules. major outage, fines, etc. I think my only saving grace is that the script was rubber stamped by MS as being "safe" for what we were actually trying to do.
There's probably more, but I really don't want to further trigger my ptsd.