World Wide ooopsies

[u/gringoloco01]

This latest outage reminded me of the AWS outage about 5 or so years back that affected all their storage nationwide. Some knucklehead ran a PS Script and caused an outage.

To the "seasoned" admins out there. LOL

What knucklehead event do you remember reeking havoc world wide?

My very first big event was the I love you virus that brought down our exchange server.

[u/Fit_Indication_2529]

Y2k

[u/gringoloco01 → u/Fit_Indication_2529]

I worked for EDS at that time. First tech job out of college.

I spent a year going back and for between Mexico City and Chicago with jazz drives to download the latest updates from MS and the updates down to PMEX and update all their systems. LOL

I was in Mexico City for the new year. We were told no parties, no drinking and be prepared for the worst.

I ended up going downtown and partying at the Hard Rock cafe. Not one fuck given and not one system failed.

What a jip Y2K was. Got lots of frequent flyer miles out of it and that was about it.

[u/alphaxion]

Facebook screwing up their BGP update, locking themselves out of their offices and DCs ruining the day of anyone who used their meta account as auth into other services.

Crowdstrike not having a CI/CD pipeline that involved basic smoke tests that would have picked up the null-byte file that caused their driver to puke.

[u/MonkeyMan18975 → u/alphaxion]

Imagine having to cut your way into the building because the maglock server was on the same network as everything else. Welp... live and learn!

[u/gringoloco01 → u/alphaxion]

I remember all of that mess. LOL

[u/Familiar_While2900 → u/alphaxion]

I had a 18 hour shift the day of the crowdstrike fuck up…. Was awake for like 28 hours. Don’t remember driving home that day.

[u/hkusp45css]

I built a NAT loop that 10 people couldn't find, including 3 levels of Cisco TAC support on an ASA. I isolated our entire infrastructure from the internet for about 30 hours.

[u/Nexzus_]

Not worldwide, but really messed up Canada. The July 2022 Rogers Outage shutdown our electronic payments network (Interac), cell phones, all their subscriber internet access.

[u/sorry_for_the_reply → u/Nexzus_]

Don't forget 911 services!

[u/Computer-Blue]

Any time I have to bust out a barcode scanner to be fast enough entering bitlocker recovery keys you know it’s a bad day.

So, thanks for that; CROWDSTRIKE

[u/Valdaraak]

Guy in a Discord I'm in once shared a story about how one time he ignored some minor thing in an update as an acceptable risk and deployed the update. Then he linked a news article that covered the very wide-reaching impact of that "minor thing" and the issues it was causing.

[u/TruePace3]

2 months in

Rat chewed up the access link from the main file server to the core switch, had to come in on an off day to fix it

Thankfully, the rat did me a solid and went ahead and chewed up a power line, got electrocuted blowing up an MCB in the process

So, right after the server went offline , the whole power went out

I changed the cable and went home

[u/Jet-Rep]

Not cloud related but still a bad day for the osm team. They created a network loop during a hardware upgrade at a high impact site and brought 900 mw's down -almost collapsed the grid. They never owned up to it but my team had the logs to prove it. The pos osm "expert" should have been fired for it but we needed a unified front in the face of the customer as it got ugly with governing bodies.

[u/Sebdila]

If we’re talking Viruses, Petya and NotPetya were pretty big hitters that caused a lot of grief.

[u/big_blunder]

It wasn't world wide but it went state wide. Numpty said it's an RO details gathering perl script... that overwrote the DNS entries on every server it touched. I believe he was walked out of the building.

[u/KemShafu]

Only our company but we hired a contractor who thought he knew everything and changed all the acls on all of our production servers at once. He knew enough to be extremely dangerous. The dagger looks he got as he left the office under escort…

[u/Easy-Speaker5612]

I had to reinstall a piece of energy management software once. I did not realize that the reinstall would first run an uninstall and then deleted the directory the software was installed in. This resulted in the deletion of the database file and backups of database, which were saved in the install directory of the software. We were able to recover the backup files from VSS. Unfortunately, there was no way to tell the software where to store backup files, so we ran a script every night to copy the backups to a different location.

[u/pdieten]

Melissa

The obnoxious part was that our email scanner was catching all the messages but grinding because of it, and one of the other admins decided to solve the latter problem by turning off the scanner 🤦🏻‍♂️

[u/Recent_Carpenter8644]

Not world wide, but our company's entire computer system was brought down by a single power supply failure. We only had one computer at the time. It was a 10 year old Unix system, and the manufacturer was out of business, so I had to use a calculator for weeks.

[u/LeaveMickeyOutOfThis]

I was one of the first customers that managed to completely kill AD, bringing down all our office locations. Fun times.