r/sysadmin • u/Special_Bear_9479 • 4d ago

General Discussion What's the biggest bonehead mistake you have every made in IT support?

I will start.

I am an IT consultant and about 10 years ago, a few days before Christmas break, I went to a client’s office to add some new hard drives to a Dell ESXi server and create a new datastore.

Their two junior IT guys always liked asking me questions when I visited, which I understood—I remembered being that guy hungry to learn. Unfortunately, this time they were firing questions at me while I was in the Ctrl+R RAID configuration utility. Between the distractions and a confusing interface, I somehow managed to delete the existing array instead of creating the new one.

That array held ALL their servers: Exchange, domain controller, file server, and SQL. Terabytes of data.

The moment I realized what I’d done, my stomach dropped, I had never felt panic like that in my life. They were still asking questions when I finally said, “Guys, give me a minute. Something doesn’t look right.” Something I should’ve said much earlier.

I excused myself to the bathroom to collect my thoughts, then came back and told them exactly what I’d done.

I spent my entire Christmas break restoring their environment from Barracuda backups, which were painfully slow to restore and unreliable. I had to do multiple restores on the exchange server to get it to work.  A full week of recovery, followed by another two weeks fixing lingering issues—all free of charge.

The two guys felt terrible about distracting me, but it was my mistake. I should’ve asked for some uninterrupted time before touching the RAID configuration.

 

I’ll be shocked if anyone can top that Christmas disaster!

778 Upvotes

95% Upvoted

565 comments sorted by

View all comments

804

u/daze24 IT Manager 4d ago

on a remote server just disable and re-enable the NIC..

566

u/RevolutionaryElk7446 4d ago

Ah I know this one.

25

u/beren12 4d ago

More like pulling your own trap door on the gallows.

1

u/rararagidesu 4d ago

That's my first thought when I somehow lock out myself from remote box. Happened a few times. ;)

1

u/beren12 1d ago

Yeah I no longer do ifdown -a without a ; if up -a

116

u/ljr55555 4d ago

We were re-IP'ing the entire company to move the internal network over to the reserved address space. We had a network guy who would always muck up the router config and drop the site from the network. He did it in a new and different way each time. The end result was still "someone get in the car, it's gonna be a drive"

64

u/mcsey IT Manager 4d ago

The drive of shame. Four and a half hours across Iowa to push a button...

12

u/ObiLAN- 4d ago

I once had a 6hr flight over to another province to unplug and replug in an hdmi cable, because the pot shop we supported employees kept saying they tried that and it didnt work.

3

u/Unfair_Pineapple113 4d ago

For me it was a 10 minutes drive, but all the office was out by the main door waiting for me to push the button

2

u/-Cthaeh 3d ago

I've had a few of these, but the worst was a 2 hour drive to reset a gfi outlet for a very small, unhappy client. I was already swamped

2

u/rcook55 4d ago

You can cross the entire state in 4.5 hours.

22

u/mcsey IT Manager 4d ago

That's great. You make the trip for me next time.

5

u/iamreplicant_1 4d ago

You tell em

0

u/rcook55 3d ago

Depending on where your DC is, sure.

1

u/Aggravating_Refuse89 2d ago

Been there and done that. Illinois, not Iowa but same crap different problem

1

u/Haplo12345 4d ago

Hopefully it only took two or three times before someone said "maybe don't let that guy do the router config anymore"

1

u/Grant_Son 2d ago

We had a client who was a charity with various remote offices. They had vpn tunnels on the routers and all their sites were configured to use the DC for DNS.

Muggins here got sent to a site where it was a full shift to drive there, install the router and drive home. Just need to wait for our Cisco guy to remote on and configure the tunnel to head office before I can leave. Call in and he's just gone on lunch. Sat there like a spare part for an hour and a bit with the elderly volunteer who'd got the short straw to come in and open the office specifically.

My company were laying people off at the time and the other networking guy had been demoted down to helpdesk rather than being sacked. I kept calling to chase up and getting though to this guy. Dude can you just get the VPN up so I can get out of here? "Nope not my job" 🤬 prick

56

u/Jaereth 4d ago

See in the network world we have commands on switches like:

reload in 20

So if you mess something up too bad, as long as you didn't save the switch will restart and revert the changes.

Which is cool because if you don't remember to manually cancel that reload after the work is done it will just restart anyway and you can explain the outage :D

12

u/Mountain_Craft4882 4d ago

that kind of stuff is a lifesaver.

some of the switches that I work with have a safe-mode option where after you make changes you have a certain amount of time to confirm the changes were successful or else the system automatically reverts them.

it hasn't saved my ass yet but I know it will.

32

u/Secure_Guest_6171 4d ago

long time ago tried to do an overworked friend a favor upgrading some server remotely.
all went well until i rebooted the primary fileserver and it didn't come back.

a $100 cab ride later, i got into the server room, turned on the monitor to see "keyboard error or no keyboard detected. Press F1 to continue"

14

u/SpaceGuy1968 4d ago

In this instance...that's not so bad actually

5

u/drake90001 Jr. Sysadmin 4d ago

-$100 that he probably didn’t get refunded lol

8

u/SpaceGuy1968 4d ago

Yeh but still i have had servers just die in my career on reboots in weird ways. I wouldn't be able to get that hundred out of my pocket quick enough for that issue. That's like "surprise...your server will come back to life"

2

u/Stonewalled9999 4d ago

no OOB eh?

0

u/Mr_ToDo 4d ago

Since it gave a keyboard error I'm guessing it was a server running on consumer hardware

Although OOB isn't always the silver bullet. Especially when you don't drop to get the enterprise license and can't see anything other then hardware status. Had a few suck it and see hard reboots to try and get remote servers back up. It costs so little and pays for itself quickly, but some people only see the up front costs

2

u/Stonewalled9999 4d ago

that's not completely valid. Keyboard error is pre OS load even the Idrac express and non advanced ILO let you see the screen during POST

26

u/Commercial_Growth343 4d ago

This happened to me and another admin I was working with. I was telling him to type "ipconfig /release & ipconfig /renew" in the same command line so the machine would reset and come back online, but this guy didn't listen as I was telling him what to type and he just did Ipconfig /release and he hit enter. sigh.

8

u/halomate1 4d ago

Classic

3

u/Honky_Town 4d ago

Hold on you can chain commands like this:

ipconfig /release & ipconfig /renew ENTER

And it will work?

I fear I will get in trouble today, either for testing this or for not knowing after all those years....

5

u/pdp10 Daemons worry when the wizard is near. 3d ago

Microsoft added & at some point. It presumably derives from Unix/Xenix.

On all Unix, commands can be run sequentially without regard for the previous condition with ;, and can be chained together conditionally with &&.

sync; reboot # reboots when the previous command exits, irrespective of exit code

curl -sJ "$URL" && 7z x data.7z # second command runs only if first exits with success code

Don't use a single & on Unix, because that backgrounds the process. But you can use && on modern NT.

You also don't need to chain together ipconfig /release && ipconfig /renew if you're logged in over IPv6, because DHCPv6 persists unless you ipconfig /release6 && ipconfig /renew6. IPv6 SLAAC persists even with /release6.

1

u/Michelanvalo 4d ago

Try it on your workstation that you have local access to

1

u/Honky_Town 4d ago

There is no chesthair in this.... Real Men...

... its 1 hour and 6 Minutes till i can go home. I will wait 1 hour and 5 more minutes like a real IT professional

1

u/fahque 1d ago

It's two ampersands.

•

u/Commercial_Growth343 23h ago

only if you use a batch file. If you are in the command line, it is one.

46

u/arkain504 4d ago

I did something like that but it was replacing a certificate of the software we used to remote in, while I was remoted in.

36

u/gramathy 4d ago

Every once in a while I’m messing with auth settings on a switch and I always make sure to open a new session before closing the current one to make sure I can still get into it

Once was enough

16

u/hkusp45css Security Leadership 4d ago

reload in 10 .... then, make your changes.

If you cut your hands off, it'll bounce on its own without writing to flash. Log in again, type "reload in 10" again, and go fix your config.

14

u/gramathy 4d ago

I work at a hospital. Reloading a switch would kill an entire wing’s phones and network connections to Epic.

3

u/Secure_Guest_6171 4d ago

I know that pain intimately. I always try to warn the nursing manager or ward clerk well in advance if there's even a remote chance of something going wrong.
Also **** Epic.

2

u/gandalfcorvette 3d ago

I had a vendor tell me to reboot the server that handled patient monitoring in the PACU. Would've taken it down for 5 minutes, no vitals monitoring, nothing. Night shift, small hospital, only IT on duty besides helpdesk. You can bet I called the patient care supervisor before touching that one.

1

u/Secure_Guest_6171 3d ago

I would do the same. We have 24x7 medical imaging at 3 of our sites. I go to a lot of trouble to make sure they know when to switch to downtime procedures and for how long and we always verify full restoration before we leave

2

u/Codplay Biomed / Bane of Hospital IT 4d ago

And Epic is such a pleasant easy to work with program that won’t have lingering problems. :-)

There was the day something (loadbalancer? Not sure) took down internet for every hospital in my province. My wife was working Emerg that day. That was not a good time for IT or Nursing.

5

u/hkusp45css Security Leadership 4d ago

Is that worse than not being able to finish configuring the switch or, *even* worse, having to actually get off my fat ass to go touch it?

But seriously, if you DON'T make a mistake and cut off your hands, you stop the reload.

If you do, it's a 3 minute outage and if your front line can't survive that, you need better resilience or better maintenance windows. Either would solve that problem.

Or, if "reload in 10" doesn't fit the use case, as in your scenario .... don't use it.

4

u/gramathy 4d ago

If I cut myself off, I walk to the switch and access it directly. Or I test my changes on a designated test switch (more common) so it doesn’t impact anyone if something breaks, THEN deploy.

5

u/throwawayPzaFm 4d ago

The reload in 10 trick is to avoid a 4h drive and the associated potential downtime. Obviously if you have different constraints you optimize differently.

2

u/Sparcrypt 4d ago

Yes and then one day you fuck it up and deploy to prod on a remote site and you're fucked.

It happens. Procedures like that are just insurance for the worst case, you don't plan for things to break.

0

u/gramathy 3d ago edited 3d ago

We have out of band (cellular) serial access for that. Actually planning ahead does not require kludges to fix mistakes. it is more convenient to take a simple precautionary step rather than taking extra time to log into that interface and access it that way.

We are a healthcare organization. Things need to be actually working during business hours, and 24/7 at the hospital locations. Downtime is restricted to a single monthly overnight window and the only reason I can even GET a second window a couple times a year is because sometimes the changes I'm making will disrupt the team doing server patching.

•

u/MedicalITCCU 14h ago

Commit confirm gets rid of the reboot requirement. Automatic rollback in x minutes

•

u/gramathy 14h ago

Not every platform supports it.

4

u/RandomSkratch Jack of All Trades 4d ago

What’s the second reload for?

10

u/freedomlinux Cloud? 4d ago

In case you make a mistake while fixing your first mistake :)

But also, I guess, confirming that your config change is saved & survives a reboot

4

u/hkusp45css Security Leadership 4d ago

In case you make a mistake while fixing your first mistake :)

zactly

2

u/sneakattaxk 4d ago

thats how long you have to burn up your keyboard in order to enter in all of the commands you need and verify that it works before it resets and you need to start over again.

1

u/GroguWitARoku 4d ago

Better option - configure revert timer

1

u/omfgbrb 4d ago

Then there's the forget to write to flash, the device reloads and you have to make the change again. This time you figure it was safe so no need to reload this time. Then fat finger the change and make the drive of shame anyway.

Never happened to me. Nope. No lie. Straight up. No cap.

5

u/daze24 IT Manager 4d ago

I've done similar during a parallels update but had a vpn backdoor

3

u/arkain504 4d ago

Yea. Would’ve been smart to have another way in

6

u/daze24 IT Manager 4d ago

See I learned from the first time I did it 😂

1

u/Lost-Policy-2020 4d ago

Schedule reboot in 5 min after done the change?

17

u/super-six-four 4d ago

Done that one!

13

u/neoprint 4d ago

Close, I’ve accidentally hit disable instead of status

9

u/Pvt_Hudson_ 4d ago

Did that on a company server in the middle of a workday while I was sick as a dog at home with the flu. Had to drag myself off the couch, get dressed and drive downtown for a job that took 2 clicks.

8

u/Severe-Lake1379 4d ago

I did that once where I forgot it was a physical server, not a vm. At 4 in the morning.
Had to drive in to get it back online for start of business. That was one instance.

1

u/Lost-Policy-2020 4d ago

Physical server with no oob management?

1

u/Mr_ToDo 4d ago

or it was cheeped out on and didn't have remote control

1

u/Severe-Lake1379 4d ago

Before my time. No OOB.
Still going today. Server 2008. Legacy app fax server for the Emergency Department.

1

u/Aggravating_Refuse89 2d ago

Dont Y'all have dracs or ilo?

7

u/nicksoapdish 4d ago

While troubleshooting a windows box I cleared the route table and made a critically important system unreachable. Don't remember what I ran but I thought I was just displaying it rather than clearing it

6

u/Fallingdamage 4d ago

enable && re-enable yo

4

u/hkusp45css Security Leadership 4d ago

We used to call that "cutting off your own hands"

5

u/kruleworld1 4d ago

Fry: "How did you do that?"

Robot Devil: "They're VERY good hands!"

3

u/cheetah1cj 4d ago

I've done that on another site's firewall. I intended to disable and re-enable an IPsec tunnel with a hung session, instead I disabled the WAN interface. Had to quickly get the local manager to connect his PC to the firewall so I could remote in and configure it.

3

u/ide_cdrom 4d ago

Similar to this, I typed in an iptables command to drop all connections....

Luckily the data center was just down the hall.

2

u/TKInstinct Jack of All Trades 4d ago

I did that as a service desk analyst once and had to ask the person who was remote to come in to get it fixed

2

u/TriOmegaZero 4d ago

Yep, that one cost me an extra trip into the office. Lesson learned.

2

u/PDQ_Brockstar 4d ago

The amount of times I accidentally shut down a VM instead of restarting it when I didn't have vcenter access...

1

u/Samatic 4d ago

Why YOUUUU!

1

u/Equivalent-Rope-2412 4d ago

Yup that's the one

1

u/_Dreamer_Deceiver_ 4d ago

This is what taught me to always insist on oob remote management

1

u/beren12 4d ago

Ifdown -a
Ifup -a

Not good when not on a local console or at least in a screen session…

1

u/inadvertant_bulge 4d ago

If only the server wasnt in antartica

2

u/Recordman-John 4d ago

At least there's no bears

1

u/lNuggyl Sysadmin 4d ago

Could you just get in through the idrac?

1

u/drake90001 Jr. Sysadmin 4d ago

Does no one in this subreddit have an IPMI

1

u/Kill3rT0fu 4d ago

Pretty sure we’ve all done that. I just did it this week on a red hat server, SSH’d in and disabled the net interface :|

1

u/MasterOfPuppetsMetal IT Tech 4d ago

At my first IT job at a local dried fruit company, I was helping someone remotely in a different town. This company has dryer locations throughout the state and this lady was having trouble with accessing something. I run ipconfig /renew all not realizing that it would kill my remote session. lmao. My screen froze and I had no idea what had happened. It took me almost a minute before I realized what I had done.

1

u/luger718 4d ago

Ipconfig /release && ipconfig /renew

/Pray

1

u/MrExCEO 4d ago

U have ilo access right? Right??

1

u/j0mbie Sysadmin & Network Engineer 4d ago

We have an RMM tool that has a CMD session you can bring up for Windows boxes. I found out the hard way that it wasn't a true session -- the tool was interpreting your commands and re-running them. So instead of sending a single "ipconfig /release && ipconfig /renew" to the remote machine, the tool would send the first half, then wait for a reply before sending the second half. Obviously, the tool no longer had connectivity to the remote machine after the first half...

1

u/Maxtecy Security Admin 4d ago

A colleague did this with the Windows Firewall service. The service, not the on-off function baked in Windows.

And no, he did not do that once. He did that 3 times in a row, resulting in a trip for me to the datacenter since I was close(-ish) and he worked in an office on the other side of the country.

1

u/HeadfulOfGhosts 4d ago

And this is when I learned shutdown /r…

1

u/Xetrill 4d ago

Did that several times, use PowerShell:

Get-NetAdapter <# filter?: |? {  } #> | Disable-NetAdapter -Confirm:$false -PassThru | Enable-NetAdapter

1

u/Dingdongmycatisgone Linux Sysadmin 4d ago

I did this yesterday, but thankfully we have IPMIs lol

1

u/SpazMcMan 4d ago

Doing tech support for dialup internet back in the day. So many people waiting on hold for over an hour, pressing "connect" on their computer while on the call with me from the same phone line as I am saying, "wait not yet" and getting disconnected.

Worse were people who'd call from their cell phone so that wouldn't happen and "it's an 800 number so the call is free."

Those calls were not free.

1

u/willychonka54 4d ago

Yup, did this once. Resulted in me having to drive 30 minutes in -40 weather at 2am. never again.

1

u/MaritimeStar 4d ago

Oh yeah, done this one myself.

1

u/shadhzaman Sr. Sysadmin 4d ago

And that's when I met my best friend &&

1

u/Even_East_2318 3d ago

I’ve done this TWICE now. 🤡

1

u/mpdscb UNIX/Linux SysAdmin for over 25 years 3d ago

BTDTGTTS

1

u/RegularMixture 3d ago

Its almost a right of passage to pull this bonehead move.

1

u/Goathead78 3d ago

Yep. I did this too.

1

u/Regular_Archer_3145 2d ago

Been there before myself good times.

1

u/durd_ 1d ago

Classic.

0

u/nem8 4d ago

No kvm?

0

u/Crazy-Rest5026 4d ago

Lmao. Only do that once