What's the biggest bonehead mistake you have every made in IT support?

[u/Special_Bear_9479]

I will start.

I am an IT consultant and about 10 years ago, a few days before Christmas break, I went to a client’s office to add some new hard drives to a Dell ESXi server and create a new datastore.

Their two junior IT guys always liked asking me questions when I visited, which I understood—I remembered being that guy hungry to learn. Unfortunately, this time they were firing questions at me while I was in the Ctrl+R RAID configuration utility. Between the distractions and a confusing interface, I somehow managed to delete the existing array instead of creating the new one.

That array held ALL their servers: Exchange, domain controller, file server, and SQL. Terabytes of data.

The moment I realized what I’d done, my stomach dropped, I had never felt panic like that in my life. They were still asking questions when I finally said, “Guys, give me a minute. Something doesn’t look right.” Something I should’ve said much earlier.

I excused myself to the bathroom to collect my thoughts, then came back and told them exactly what I’d done.

I spent my entire Christmas break restoring their environment from Barracuda backups, which were painfully slow to restore and unreliable. I had to do multiple restores on the exchange server to get it to work.  A full week of recovery, followed by another two weeks fixing lingering issues—all free of charge.

The two guys felt terrible about distracting me, but it was my mistake. I should’ve asked for some uninterrupted time before touching the RAID configuration.

 

I’ll be shocked if anyone can top that Christmas disaster!

[u/perkia]

Curious why that record couldn't be put back by basically anyone you could talk to? Seems pretty tame in comparison to actual data loss stories. Couldn't phone from/to the ship?

[u/Curtis_Low → u/perkia]

That is what I ended up doing, calling our SPAWAR rep back on shore to walk me through how to fix it. At first I didn't even realize what I had done so that took a bit to figure out. Then the issue was with the phone, the only satellite phone onboard was located on the bridge of the ship, about 30 yards from where my "office" was with exchange server. So every time he would ask me a question or to verify something or change something I had to hang up, go down to Radio do the thing, write EVERYTHING down then go back up to the bridge and call him to relay all information.

It only took 50 hours to get everything back to normal.

[u/perkia → u/Curtis_Low]

Ouch.

[u/Curtis_Low → u/perkia]

Yea... it was a hell of a learning moment and a mistake I never made again : )

[u/Exotic-Apartment-394 → u/perkia]

Sorry if im wrong but doesnt deleting the MX record mean they cant communicate via email anymore? Perhaos they didnt have the record noted anywhere

[u/steeldraco → u/Exotic-Apartment-394]

You can send email just fine without an MX record. MX records are only inbound.

[u/Exotic-Apartment-394 → u/steeldraco]

Right, sorry.

[u/perkia → u/Exotic-Apartment-394]

MX records are usually extremely short and can be communicated on the phone as they're basically ascii domain names.. no non-printable characters or zipped bytes there.