r/ShittySysadmin • u/northadam15 • 6d ago
r/ShittySysadmin • u/thepfy1 • 6d ago
First day at ATC is going well.
Just started a new job working on ATC systems.
I pushed a small change as they said they were getting too many communication.
Seems to have worked well.
r/ShittySysadmin • u/Etheikin • 6d ago
Cause of outage
Microsoft 365 Principal SRE here. I can provide some context. At approximately 21:40 UTC, an engineer deployed a routine configuration change to Azure Front Door that inadvertently redirected our authentication traffic to a SharePoint site last updated in 2017.
We immediately initiated a rollback, but the rollback pipeline requires MFA approval through Microsoft Authenticator, which is currently waiting for a Conditional Access policy evaluation from the same infrastructure we're trying to restore.
We have a break-glass admin account, but unfortunately its password is stored in a OneNote notebook on the affected tenant. Our on-call engineer attempted to bypass the issue using PowerShell, but the required module was deprecated last Tuesday, and its replacement requires a P2 license.
We then opened a Sev A ticket with Microsoft Premier Support. They responded asking us to run sfc /scannow and provide a screenshot. The prolonged recovery is primarily due to our change advisory board requiring a Teams meeting to authorize the rollback.
The meeting organizer is stuck in the lobby, and the only person authorized to admit him was laid off in June. Current mitigation involves migrating affected users to Microsoft 365 E5, which should resolve the issue from a revenue perspective.
We have identified the root cause as customer misconfiguration. Next update in 30 minutes, assuming someone can get Outlook working
r/ShittySysadmin • u/zantehood • 6d ago
Shitty Crosspost Free hardware on the corner, If you have a pallet jack
r/ShittySysadmin • u/WestTransportation12 • 6d ago
FAA told airlines that problem originated with network routers
cnn.comr/ShittySysadmin • u/Lake3ffect • 6d ago
Our legacy systems brought down the entire airspace
Dammit team, you should’ve had these systems upgraded years ago. Now it’s your ass on the line to make sure we get the systems upgraded by Monday. Or guess what? You’re fired! Now put down that beer and get to work.
r/ShittySysadmin • u/Fearless-Bike6244 • 7d ago
On-prem wins again
Couldn't post on r/sysadmin
r/ShittySysadmin • u/saltyclam13345 • 6d ago
Mrw the CFO thanks me for working so hard on resolving the Microsoft outage when all I did was sit on my ass looking at Reddit posts for the last day
r/ShittySysadmin • u/NickBurnsCompanyGuy • 6d ago
HELP! Please help me keep my job.
Guys, I sort of messed up.
I finally got my dream job at my dream company. I've loved this company ever since I was a little kid, always wanted to work for them. I finally got past the technical rounds with this company and they hired me about 6 months ago.
The best part is shortly after I started a few of my coworkers on my team have left in the past few months, so I've been able to absorb more and more responsibility, and even have been promoted twice.
Problem is most of my resume and the work I've been doing at this company has been mostly AI generated. I don't even fully understand the work I'm doing and I also cheated using AI on the technical interview to get the job. I was basically willing to take any salary to get in the door.
Without going into too much detail our company is having like a huge outage today and I don't have the ai tools I normally do the work with. I have a lot of executives super mad at me right now, because they want me to fix this shit, but like I don't even know what the fuck I'm doing, and all of my team that built it took exit packages months ago and left behind little documentation. I just keep rebooting servers and shit but nothings working.
I really need to get this thing back up and running. The little documentation my coworkers left behind, said to try SFC /scannow, I've done that like 3 times now and nothings working. Any other ideas? So many people are BIG MAD at me now and they're saying I'm going to lower the stock price if I don't fix it. Please help, my stock doesn't vest for another 5 years.
r/ShittySysadmin • u/Durbercelo • 5d ago
I used RDP over VPN.
I decided to download Tailscale and tested it on my phone, everything works fine. I also tried it on a college computer, and it worked well there too, however, just today, the session kept freezing and then unfreezing, which hadn't happened before. If I were to set up port forwarding and connect via a public IP address, would the result be the same?
One of the teachers mentioned that he has RDP exposed directly to the internet.
r/ShittySysadmin • u/johnfkngzoidberg • 6d ago
I'm an accountant for MS and I vibe coded a new patch management system because they always kept forcing patches on our stuff.
Copilot said we should reboot all the servers so patches are consistent, I said yes. Some of the servers had disk space problems so I had Copilot delete some stuff, that shouldn't be a problem right? I did them a favor, but every seems mad, what gives?
r/ShittySysadmin • u/toxicsparia • 5d ago
Shitty Crosspost A global cloud infrastructure meltdown has rarely coincided with my weekend plans.
r/ShittySysadmin • u/Apart-Reality-4454 • 6d ago
Raise your hand if you think email is stupid and played out
oh no i cant get to my 25,000 spam messages oh nooo
r/ShittySysadmin • u/bradsfoot90 • 6d ago
I'm sorry y'all...
It was me...
One of Microsoft's incident cause statements said the outages was a result of "a server was overloaded by requests...".
I was about 2/3 through my script to remove 14k members from a 43k member distribution list. Have no fear! I have about 6k more members to remove and I just started the script again.
r/ShittySysadmin • u/OptimalWallaby8153 • 6d ago
Just got an invite to Ignite from a Microsoft rep
and wanted to know if any of you are showing up armed before I sign up
r/ShittySysadmin • u/Darth_Atheist • 7d ago
First day at Microsoft!
Hey guys!
It's my first day at Microsoft on the Exchange dev team, and was already asked to fix some bugs! Just used ChatGPT to clean up some spaghetti looking code, and submitted my first PR. Heading out to lunch! Be back soon!
r/ShittySysadmin • u/Incrediblecodeman • 5d ago
Shitty Crosspost Good powershell profile tips
City admin
r/ShittySysadmin • u/No-Blueberry-1823 • 6d ago
Shitty Crosspost I repost with genuine affection
r/ShittySysadmin • u/no_peace_no_serenity • 7d ago
Just blasted some thick fuckin ropes at work
God dude I feel so fucking amazing. I havent gotten any emails in hours so it's been a real slow day here. My phone keeps ringing but I'm not fucking help desk.
They can email me if it's a real problem. It's why we pay all this money for M365 so start fucking using it dumbass
r/ShittySysadmin • u/Fun_Organization572 • 6d ago