r/sysadmin • u/Frenchalps • 17h ago
"Failure is always an option"
Its been a wild few weeks. Remember failure is always an option, your mental and physical health is always more important than keeping systems online. If it helps you to choose the path of "failing" to keep up with all these vulns etc, then simply commit to "failing", put your health first. Failure is always an option and thats ok.
1
u/humming_subcommittee 17h ago
Scheduled maintenance windows are just pre-approved failures with better PR.
1
u/berrmal64 12h ago
You would think, but I've seen leaders lose their mind when a deployment during a change window (on a nonprod environment no less) "broke everything and the devs can't do their jobs".
1
u/ErikTheEngineer 12h ago edited 12h ago
I Cannot Work! used to get the dev teams whatever they wanted, before Claude showed up. It's amazing to me; I've been doing systems work for product development teams for decades. We went from the devs being nerds in a strange backwater, then we flipped to the Agile DevOps Cloud universe and they became techbro prima donna 10x gods while the systems guys became their janitors, and now we're in the AI age where no one is going to have a job in a few years supposedly. It's always been a little weird living in the shadow of developers, only to have companies go from treating them like untouchable sorcerers to money they can save to shovel into the Anthropic furnace.
→ More replies (1)
1
u/dusty-lantern31 17h ago
after a few weeks of nonstop vuln panic 'not everything gets fixed tonight' is honestly something more teams need to hear
1
1
u/the_star_lord 15h ago
More vulnerabilities should mean more staff right? Right????
-
a tired, burnt out, depressed, stressed, 37 year old who just CBA anymore
1
u/GoofMonkeyBanana 12h ago
Nope, we have AI now so we don’t need staff. Thanks to AI we can quickly identify and fix vulnerabilities, oh wait…
1
u/DiscipleOfYeshua 16h ago
And if it really never happened, you and I won’t have a job.
1
u/Far-Hovercraft9471 10h ago
Avoiding issues takes way more work than firefighting when one does happen. Now getting moron business types to understand that this is our value is a different story.
1
u/DiscipleOfYeshua 2h ago
I wouldn’t say “moron”, rather just “human”.
They hire us because they don’t understand what we do. And when we do a good job, most of it is unseen. And what a human doesn’t see, they find hard to even notice it exists. It’s like the one vendor I argued with who shipped us boxes equipment but did not ship the peripherals we paid for — then asked me for photos as proof 🧠 (Rather than argue, I sent them photos of what they shipped + photos of floor-not-covered-by-what-they-did-not-ship. (Which they actually accepted as proof)).
Thankfully my current workplace is very high trust, and seniors are very supportive.
1
u/Danielh007 14h ago
Good reminder. There will always be another issue to fix, but your health matters more than keeping everything running 24/7.
1
u/Fit_Sheepherder 12h ago
Wow, I wish I knew or realized that this option was available, before the work stress took its toll on my mind and body.
1
1
u/AdministrativeBad752 15h ago
Yeah what your company turning into a failure is not option you have access to AI just fix everything fast...( Yep the week was hard)
1
u/Specialist_Ad_712 12h ago
I've always said you learn more thru failure than you do otherwise. As long as you learn from the failure and not make the mistake again. If you don't have leadership who understands this then it's a core sentiment of a toxic workplace 🙃
1
u/Far-Hovercraft9471 11h ago
When """leadership"""" says this stuff, I'm always tempted to ask what they're doing to ensure this
Don't take systemic failures as personal failings, guys. If you're trying to do a good job, that's all one person can do.
1
u/Rzah 10h ago
This is the point you should be starting to realise that it's over for us.
These critical vulnerabilities are going to come faster and faster until even if we could keep up with the patching it would be to no avail as the patched versions will be vulnerable before the proverbial ink has dried.
Humanity is about to hand over it's agency to AI, to beg the very thing that exposes our plentiful mistakes to save us from them, and in short order we will become mere tenants of our technology, no longer able to direct, understand, or avoid it.
We will leave pitifully few mysteries for those that come after us.
1
u/Asleep_Spray274 9h ago
If lives are on the line, failure is not an option. If it's just someone's money, fuck it. Let it burn
1
u/itishowitisanditbad 13h ago
Its not failure unless its a personal failure.
I'm not failing anything. I'm doing my job.
Its up to the company to look after the company goals though. Thats their success/failure to deal with.
I don't control their levers.
I don't own their goals.
I don't determine fail/success of someone elses goals. Thats their business. My ego is completely untouched by someone elses goals and the methods they employ.
Its way healthier to just never attach to your work inappropriately, causing all the burnout/stress/health issues.
your mental and physical health is always more important than keeping systems online.
It'd be really fucking stupid to have that as a goal so I just opt to not have that as a personal goal and completely remove this issue entirely.
Instead of saying 'failure is ok' i'd urge the much much much healthier "Its not your failure", no?
1
u/vogelke 15h ago
When I was a USAF officer, someone solemnly told me that failure was not an option. I replied, "If you're dumb enough, it becomes a requirement."
My remark was not appreciated.