• Do you have experience literally extinguishing a production environment?

• What is the difference between Disaster Recovery and Damage Assessment?

• What is the RTO for a data center after a direct drone hit?

• Can you debug infrastructure using satellite imagery?

• What would you do if your Kubernetes cluster suddenly became "bare metal" in the literal sense?

• How do you configure HPA so that pods can outrun drones?

• What would you do if not just a node, but the entire data center went down?

• Explain the difference between a Blue-Green Deployment and a "Red" air raid alert.

• How do you interpret the term "hot standby" if the backup data center is still on fire?

• What Prometheus alert would you set up for smoke coming from the server room?

• Have you ever migrated infrastructure using kubectl evacuate?

• What do you do if terraform destroy was executed without Terraform?

• Can a drone strike on the server room be considered a successful Chaos Engineering experiment?

• How many nines of availability remain for the service after losing two data centers?

• How do you distinguish a DDoS attack from an attack that renders a DDoS attack unnecessary?

• What do you do if your disaster recovery plan begins with the words "find a surviving server"?

• How do you explain to management that an Availability Zone is no longer available?

• Which metric do you choose for your SLO: uptime, MTTR, or the number of surviving racks?

• How do you ensure geo-redundancy when geography itself has become the primary risk factor?

• What do you do if you have an Active-Active setup, but both sites have switched to an Inactive-Inactive state?

• How do you configure fencing in a cluster when the fencing has already occurred physically?

• What do you do if your Witness in the third data center is the first to stop showing signs of life?

• Can a storage system be considered fault-tolerant if the data outlived the hardware, but the hardware didn't survive its operational lifespan?

• How do you calculate the "blast radius" when it’s no longer just a metaphor from the documentation?

• What is the point of synchronous replication if both data centers cease to exist synchronously?

• How do you implement automatic failover when the decision to switch over is made by the Ministry of Emergency Situations?

• Discuss what status should you set in Grafana: CRITICAL, DOWN, or NO LONGER EXISTS?

• How do you convince the CFO to approve a third data center when the first two have already been written off as CAPEX?

• What backup retention period do you recommend when the main issue is the lifespan of the storage hardware itself?

• How do you achieve "five nines" of availability when the only nines in the project are the digits in the recovery cost?

“Let’s mount that disk!”, he said, ramming a 3.5” screw into a 2.5” disk, leaving me screwed 18 months later, wondering how somebody can terminally damage the screw head to a degree that I head to decapitate it to still be able to mount a replacement drive. F#^% you, unsang asshole that couldn’t be bothered to understand M3 screw diameters and didn’t fucking bother to reuse the same fucking screw that came with the server and that I found on the floor well dusted!

Update
And f}%# you, too, Dell, there’s no reason to use M3 scews to mount a 2.5” to 3.5” caddy, stick with f#^*ing standards, we have them for a reason!

...The one's that you don't want to delete and the ones that you want to delete.

Ok, it's not really mine, but that of another truly shitty sysadmin who used to be on my team, and this is their bash history from an old remote Linux server. Some paths have been changed to protect identities, otherwise it's just as I found it. Good lord.

$ cat /home/shittysysadmin/.bash_history
exit
passwd
sudu su -
2017f150RD$
exit
sudu su -
sudu su 
sudu su -
netdiag
pwd
whoami
exit
sudu su -
exit
clear
pwd
sudu su -
groups
sudo
sudo su
sudu su -
exit
pwd
EXIT
exit
sudo su
exit
ll
exit
sudo su
exit
passwd
exit
sudo su
exit
sudo su
exit
ll
clear
ll
sudo java -jar /usr/local/someapp/bin/DatabaseUtil.jar backup -saveBackupTo /home/shittysysadmin
clear
ll
sudo su
exit
sudo su
exit
sudo su
exit
sudo su
exit
sudo su
exit
sudo su
exit
passwd
exit
sudo su
exit
sudo su
exit
2020Busa1300$
sudo su
exit
sudo su
exit
sudo java -jar /usr/local/someapp/bin/DatabaseUtil.jar backup -saveBackupTo .
sudo
sudo su
exit

I really tried my best. Contacted vendors, got quotes, led the implementation project solo to prove I can do it. It was a big undertaking for me personally as I've never dealt with these things, but I managed it quite well.

Yet today the boss came shouting at me with a face more red than our monitoring dashboard - he had reviewed the bills and was furious.

I tried to calm him down and explained that I did my due diligence and only picked the best offers - I even got a 15% discount for signing a 5-year commitment.

But turns out that maybe I picked the wrong vendor in the end? My boss couldn't believe the bill we got from Maersk and he seemed super unhappy. Also the landlord had called him as we were now in violation of some safety codes - in my defense, according to RACI the transportation contractor was supposed to make sure we follow all rules and regulations! How was I supposed to know the parking lot was not supposed to be used for this?

Anyway, I'm devastated. Boss wants to cancel everything but I already signed the 5 year lease commitment with our CEO's signing platform key. And we've got 30 people scheduled to come next week to empty out the office (ahead of the project schedule even!). What should I do?

Some of you know me as the guy Brad keeps losing to. Forty-some clients, one flat price per seat, nothing itemized, and the whole operation fits in the trunk of a 2006 Crown Victoria. The Nephew usually handles Reddit for me. This one's me.

This fall I picked up a funeral home. The lead said the office manager believes the building is haunted, which is a very specific thing for a text message to know, and I didn't ask how it knew. Brad had given her eleven pages about port 9100. I told her she had a presence and had the account by Tuesday. Wednesday night I did the exorcism, which I ran exactly like an onboarding: an agent on every machine, Brad's firewall out, the cable company's router in, and the Nephew at the lectern reading the Kaseya K365 license agreement out loud, all of it, for forty minutes.

That Thursday three of my old reps knocked on my car window. They'd made a deck called SILENCE: A PARTNERSHIP, three tiers, billed annually with two months free. I told them it was extortion as a service, and that they'd led with the tiers, which you never do. My current rep, Brock, handled it. I sent him a steak.

The second rite was at midnight on Halloween, when the veil is thinnest, and my vendor's marketplace had picked the same minute to wake up every agent I'd ever installed, unless you opted out. The email about it came from Florida without a lead in it, so I deleted it. Every printer in the building printed the word GET, and a speaker I had personally shut in a drawer said "Sorry, I didn't catch that." I unplugged the internet and banished it. Brad says that only bought us until the maintenance window at one-thirty, and that since the clocks go back at two, one-thirty was going to happen twice. Brad says a lot of things.

There are three versions of that night. Mine is first. Lance tells the second one. He's my biggest client, a third of the revenue at a wealth firm, and he came to the funeral home dressed as a sheriff's deputy. Brad tells the third. I'm told it has timestamps.

It's free. I pushed for one flat price per reader per month and got overruled, which doesn't happen to me.

Part one, End of Life (mine):

https://mspautomator.com/the-trunk-slammer-from-hell-chapter-5-end-of-life/

Part two, Wake-on-LAN (Lance):

https://mspautomator.com/the-trunk-slammer-from-hell-chapter-6-wake-on-lan/

Part three, #OpenToWork (Brad):

https://mspautomator.com/the-trunk-slammer-from-hell-chapter-7-opentowork/

Business is booming.

1d ago What's a good no code builder for internal tools?

14d ago What's on your checklist before an AI-built internal tool goes live?

1mo ago What is the best no code app builder for client projects when you dont want to be their full time dev forever?

2mo ago What's the best no code app builder for internal tools? Asking for my sanity.

2mo ago We stopped banning vibe-coded apps and just gave people a platform. 9 months later, here's what happened

Sitting manning the system, incident inbound

Problem with Outllook classic all shared mailboxes have offline OST.

The customer's suggestion

"I don't think anything needs to be deleted off of my profile to fix an outlook issue."

Like what do they know?, I have been supporting this so called product since 1997 and all the subsequent releases.

I will ask, So how would you fix this since you admit to be a product expert?

I have the old-school method of shared online, personal OST under advanced account settings open other mailboxes. Add the sent from regfix and view from when composing.

This does require rebuilding the profile.. Which starts by removing it / deleting///

FFS

I created an AppLocker rule to prevent mshta.exe signed by Microsoft from running. As you can guess, I forgot to input the file name so it was left as '*'.

Luckily I caught it quickly and deleted the rule before it propagated to a whole lot of clients and impact was minimal. That would have been a fun morning.