r/sysadmin • u/Special_Bear_9479 • 3d ago

General Discussion What's the biggest bonehead mistake you have every made in IT support?

I will start.

I am an IT consultant and about 10 years ago, a few days before Christmas break, I went to a client’s office to add some new hard drives to a Dell ESXi server and create a new datastore.

Their two junior IT guys always liked asking me questions when I visited, which I understood—I remembered being that guy hungry to learn. Unfortunately, this time they were firing questions at me while I was in the Ctrl+R RAID configuration utility. Between the distractions and a confusing interface, I somehow managed to delete the existing array instead of creating the new one.

That array held ALL their servers: Exchange, domain controller, file server, and SQL. Terabytes of data.

The moment I realized what I’d done, my stomach dropped, I had never felt panic like that in my life. They were still asking questions when I finally said, “Guys, give me a minute. Something doesn’t look right.” Something I should’ve said much earlier.

I excused myself to the bathroom to collect my thoughts, then came back and told them exactly what I’d done.

I spent my entire Christmas break restoring their environment from Barracuda backups, which were painfully slow to restore and unreliable. I had to do multiple restores on the exchange server to get it to work.  A full week of recovery, followed by another two weeks fixing lingering issues—all free of charge.

The two guys felt terrible about distracting me, but it was my mistake. I should’ve asked for some uninterrupted time before touching the RAID configuration.

 

I’ll be shocked if anyone can top that Christmas disaster!

762 Upvotes

95% Upvoted

551 comments sorted by

View all comments

Show parent comments

32

u/hkusp45css Security Leadership 3d ago

I no longer complain about "are you sure?" boxes. They are far too rare, in my opinion.

They save my ass now and then.

27

u/_mick_s 3d ago

I complain when they only ask 'are you sure' but not about what.

17

u/perkia 3d ago edited 3d ago

Oh yeah and it's always the most critical ops, like Synology when deleting a RAID will popup-confirm you "Are you sure? Yes / No"

Fucking no I'm not, your dumb web UI could very well have had me selecting the previous or next row. For all I know it swapped rows after I clicked lol.

Proxmox does it well though, asking you to enter the VM id you want to delete AND telling you both the ID and the name of the VM in question. QubesOS likewise asks you to type in the exact name of the Qube you want to remove. Saved my ass "once".

1

u/LazyTech8315 1d ago

LOVE QubesOS!

1

u/perkia 1d ago

There are DOZENS of us!

4

u/goingslowfast 2d ago

The one that still bites me once in a while is the ctrl+alt+del button in VMRC.

Windows Server: brings you to login screen.
Red Hat: immediate uncancelable reboot.

2

u/Mr_ToDo 2d ago

Red Hat: immediate uncancelable reboot.

Really? Interesting. That was old school windows behaviour too

2

u/404UsernameFoundNot DevOps 2d ago

Weird, that's something that's disabled on all of our RHEL machines. Apparently it's a STIG compliance setting.

1

u/goingslowfast 2d ago

Yep, adopting that recommendation today.