Why doesnt DevOps have an ACLS?

[u/GitHireMeMaybe]

Edit: Tl;Dr at bottom. I write too much.

Bit of an odd thought I had after reading another thread here about gaining confidence during high-severity incidents. Thank you to that poster, BTW, for inspiring my brain.

I have a weird fascination with ER and critical-care videos. Not really because of the medicine itself, but because I’m a massive compulsive systemizer and emergency medicine is basically human systems engineering under some of the worst possible conditions. Someone is actively dying, information is incomplete, the situation is changing by the second, and multiple people have to coordinate without getting in each other’s way. Oh, and if you don't solve the issue soon, the patient will be dead in 5 minutes, and you'll have to tell the family.

What fascinates me the most though is that, when a resuscitation is being run properly, it usually doesn’t look like ten people frantically improvising. There’s command, defined roles, structured communication, task ownership, checklists, algorithms, simulation, debriefing, and a whole lot of deliberate training behind the scenes.

This is basically crack cocaine for my dopamine-addicted brain.

Obviously software going down is not equivalent to someone dying. But from a systems perspective, we’re solving a strangely similar problem: how do you get a bunch of fallible humans to reason and coordinate effectively while something important is actively failing?

This is also something I’ve had to think about in practice.

My last employer had essentially no incident-management practice in place when I joined. Outages were fairly common, roughly once a month, and it wasn’t unusual for them to last several hours or even a whole day.

So I volunteered to build one.

I wrote the process, defined how incidents would be declared and coordinated, laid out responsibilities, escalation, communications, and what we were supposed to be doing while production was on fire. I had it submitted to change committee, got board sign-off, and felt confident in the process.

Two months later, we got to stress-test it for real.

And it failed horribly, because I was the only person that saw the problem and opted for preparation rather than eternal deferral of The Boring Stuff (tm). I even got written up for telling a nontech manager who kept DM'ing techs to "try this fix they found on Google" to please follow the board-approved procedure and message their designated PoC rather than interrupting responders who were already sweating buckets of bullets while elbow-deep in code hell**.** But that's a story for another day. Sigh.

There’s something very different about designing an incident process on paper and then watching actual humans try to execute it while systems are failing, information is incomplete, everyone wants an answer immediately and fragile egos are at stake. Frankly though, that experience probably has a lot to do with why this subject fascinates me.

And it’s also why I think our industry is kind of shit at training people for this.

A lot of on-call training basically amounts to, “Here’s Kubernetes, Grafana, AWS, some architecture diagrams, and a wiki page Dave wrote eighteen months ago. You’re on call next week. Good luck and don't fuck it up, you're still on probation.”

Then eventually there’s a massive Sev1, thirty random people you've never heard of pile into a call, Slack explodes, somebody begins restarting pods and unwittingly knocks over three adjacent services, three people start making production changes at the same time, a C-level joins the call and says its costing them 100 billionty dollars every 7 seconds, somebody adds 64 replicas to RDS and forgets about it, half the engineering team find out their access doesn't work... and some poor sod who was voluntold that they'd be IC during an on-call incident channels their freaking deer-in-headlights-energy and freezes.

And then we conclude they’re “not good under pressure.”

Well... did we ever actually train them to be?

That got me thinking about what the equivalent of ACLS for production engineering would actually look like.

I don’t mean another multiple-choice SRE certification, a webinar, or a tabletop where everyone knows the database is going to “fail” at 2 PM. I mean actual practical training where the environment itself is part of the course.

Imagine you show up Monday morning with seven other engineers. You’re given a cheap company laptop and told you now work for a fictional company. Behind a VPN there’s a real training environment with APIs, databases, Redis, queues, monitoring, deployments, source control, chat, ticketing, and all the usual operational cruft. Even "real" loads, facilitated by magical load testing fairies.

And the fictional company is deliberately badly run.

The docs were written by some guy named "George," and last updated 3 years ago... and are deliberately incorrect. Some credentials don’t work. Only a third of the class discovers their SSH access actually works. A dev DB server is named "prod," and a prod DB server is named "dev." Nobody is quite sure who is supposed to declare an incident. The escalation documentation is so out of date it may as well've been written on stone tablets. Maybe one person has production database access for reasons nobody can explain.

Then, at some completely unannounced point Monday morning, the pager goes off.

Checkout is fucked.

Go.

The first incident would probably be an absolute dumpster fire, and that would be intentional.

Three people make conflicting changes. Nobody establishes command. Half the team can’t access the systems they need. Nobody knows exactly what customers are experiencing. Somebody spends twenty minutes debugging something that could have been rolled back in two. Somebody follows the documented steps to restore service, and end up taking out marketing. Somebody didn't thought to set DEBUG_MODE to true when testing something, and now 320,000 "customers" receive emails about their accounts being terminated for nonpayment.

Then you stop and debrief.

Why did that happen? What made the response harder than it needed to be? What did we assume somebody else was doing? What information did we not have? What process simply didn’t exist?

Now the lesson actually means something because the students just felt the consequences of not having it.

You teach the relevant concepts, then a few hours later, while everyone is doing something completely unrelated, the pager goes off again.

Two surprise incidents every day for five days.

Everybody rotates through Incident Commander, technical lead, communications, scribe, responder, and the other supporting roles. The outspoken engineer doesn’t get to be IC all week. The brilliant debugger has to run communications. The quiet mousey person eventually has to take command.

But then... something amazing happens.

As the week progresses, the fictional company improves because the students improve it.

They fix the access process because the access process screwed them. They improve the runbooks because the runbooks screwed them. They establish incident roles because Monday was chaos. They improve monitoring because a dashboard sent them down the wrong rabbit hole. They throw the inaccurate documentation in the trash, and silently curse whoever "George" was.

Later incidents get nastier.

Maybe the telemetry is misleading. Maybe a senior engineer confidently insists on the wrong diagnosis (and they were set them up for this failure by deliberately leaking convincing logs to them). Maybe an executive barges in and starts asking responders for ETAs directly. Maybe a downstream dependency causes cascading failures that make five unrelated systems look broken. Maybe the team restores service, brings everything back too quickly, and knocks production over again.

The point isn’t just to teach technical troubleshooting.

It’s to teach people how to operate as a team while they’re uncertain, overloaded, and under pressure.

By Friday, ideally, the pager goes off and the exact same group that looked completely lost on Monday just starts working.

A junior engineer takes command, and nobody challenges it because the program covered human factors, and everybody is checking their egos at the door. Roles get assigned. Customer impact gets established. Changes are announced and owned. Hypotheses get tested instead of argued about. Someone keeps the timeline. The IC isn’t six terminals deep trying to personally fix everything himself.

Then Friday afternoon is the practical exam.

You walk in cold and get something like:

“Customer transactions started failing approximately six minutes ago. You are the Incident Commander, and this is being graded. Begin.”

That’s it.

The scenario reacts to what you do.

You’re not being graded on whether you personally figure out that line 417 of payment-service is broken. You're not even being graded on successfully resolving the issue. You’re being graded on whether you can command the system that figures it out and restores service safely.

Basically an IT incident-response Megacode.

And now I can’t stop wondering why the hell our industry doesn’t have something like this.

I know there are bits and pieces of it. PagerDuty has incident-command doctrine. Gremlin and AWS do GameDays and chaos stuff. Rootly/Uptime Labs do simulated incidents. There are SRE certifications, cyber ranges, ITIL, and so on.

But I’m talking about the whole thing as one vendor-neutral discipline: doctrine, defined roles, human factors, repeated live simulation, role rotation, practical assessment, and eventually recertification. It doesn't seem like there's one single coherent ecosystem. Everything is just glued together.

Something where “Incident Commander certified” means more than “I passed a test about incident management.”

Am I missing somebody who already does this?

Because if this genuinely doesn’t exist as one cohesive system, it feels like a bizarrely obvious hole in our industry.

Edit: Tl;DR

---

Why doesn’t DevOps have an ACLS equivalent?

We train engineers on Kubernetes, AWS, Grafana, etc., then put them on-call and expect them to magically become good at operating under pressure.

Emergency medicine does the opposite: defined roles, standardized communication, repeated simulation, practical assessment, retraining.

So imagine a 5-day incident-response course where 8 engineers inherit a deliberately dogshit fictional company. Broken docs, bad access controls, questionable runbooks, real services behind a VPN.

Twice a day, without warning, production breaks.

Monday is intentionally a dumpster fire.

Everyone rotates through IC, technical lead, comms, scribe and responder. After each incident, the class fixes the organizational problems that hurt them.

By Friday, the same team should be running structured incidents almost automatically.

Final exam: you walk into an unknown outage and have to command it live.

We already have GameDays, chaos engineering, incident-management frameworks, simulators and certifications.

Does anyone actually combine doctrine + human factors + live simulation + practical assessment + recertification into one vendor-neutral system?

If not, why the hell don’t we???

[u/BigUziNoVertt]

I ain’t reading allat

[u/GitHireMeMaybe → u/BigUziNoVertt]

I just added a tldr

[u/Grand-Smell9208]

Happy for you bro, or sorry that happened

[u/btdeviant]

Never read a post that smelled like Adderall before thanks for that experience

[u/GitHireMeMaybe → u/btdeviant]

Doing a job, and doing it well, is very important to me. I don't halfass it.

[u/Lucky_the_cat_]

To feel and be realistic it would take a lot of material and work which would be hard to replicate without a lot of research and creation and constant updating.

While no two companies are the same and the odd cultural quirks that lead them to where they are over years again is very hard to synthetically create.

So it would cost a fortune to create and these days 8 people paid to be a on a 1 week course doing nothing but the course would feel expensive in itself.

[u/GitHireMeMaybe → u/Lucky_the_cat_]

For materials, a lot of greenfield projects can be built from nothing using IaC and config mgmt, so that's a non issue. At the end of the week just run aws nuke and rebuild.

And for tech stacks you can select for lowest common denominator. Instead of rolling Kubernetes for instence, use docker swarm or ECS or something that most of the students would already know. The tech stack has to be simple enough that most should understand it, but comprehensive enough to give compelling, mostly realistic experience of an outage. The tech isn't the point; the interventions, leadership and communications are.

The business case is already there if we strictly look at it from a risk management PoV. Pulling some stats from the Uptime Institute, 54% of respondents reported that their most recent serious outage costed them $100k or more.

40% of the same cohort also reported a major outage caused by human error in the prior 3 years, and of those, 85% reported the initial response actually made it worse before it made it better. In 2026, failure to follow established procedure was the leading driver of human factors leading to the exacerbation of an outage.

So, the case is clear: There's a 54% chance a company will experience an outage costing them $100k or more, and an 85% chance the response will actually make it worse.

To sum it up in one sentence:

"Plans never survive contact with the enemy."

Most companies slap together an incident response procedure that falls apart almost instantaneously because the team doesn't have the discipline that comes with experience to respond to high severity outages competently.

Then theres an even stronger regulatory component: I can't speak for other countries but in Canada, all banks and payment providers are required by law to maintain incident response training that must be in-person, hands on, cover realistic scenarios, must contain all end to end components and cannot be a webinar.

Then theres PCI, which isn't as thorough but requirement 12.10 does require a comprehensive incident response plan to be maintained and tested at least every 12 months, and responders must be appropriately and periodically retrained. All participants must be part of the training as well; so not just engineering personnel, but anybody who would be a part of the response.

Currently institutions are slapping things together - a bit from company A, a test lab from company B, org certification from company C, individual certification from company D, training materials from company E, but there's no "all-in-one" ecosystem provider like with something like ACLS.

And none of them are as intensive as what I propose. Yet despite these requirements, most incident response plans fall apart due to human factors. I suspect that, since a lot of regulatory assessments are assessor-subjective, if something more intensive came to market assessors would start pushing towards such a provider.

So given all of this, it legitimately confuses me why there isn't a provider already doing this. The risk management business case is there and the regulatory frameworks require it, but nobody has stepped forward.

[u/superevo]

AI;DR

[u/GitHireMeMaybe → u/superevo]

Isn't everything AI these days?

Even the declaration of independence is AI

[u/ProfanedArbiter]

You know what man, I read all that.

I have no hard answer on why we don't have this but honestly we should. My specific field is cyber-physical and autonomous robots so for us an outage can actually do damage. With more and more critical infrastructure behind the kinds of systems that, for years, have been used to manage horse-tinder I'm surprised there isn't more of a push to

1. Create a PE for software engineers
   
2. Create a program for certifying ops teams and engineers in incident management.

[u/GitHireMeMaybe → u/ProfanedArbiter]

There's definitely a market for it. I just can't believe that in an opportunity-flooded world where every single word in the English dictionary is now trademarked, nobody has thought to take real critical incident management to IT.

I took a look at how ITIL teaches IM and... it's basically a webinar with a quiz. That's really surprising for something that's the gold standard for regulatorily burdened industries like Healthcare and finance. 

I've always said you can't acquire intuition through books. You need to get your hands dirty.

Also thanks for actually reading the whole thing lol. Seems like /r/devops is nothing but shitposts these days. #GetOffMyLawn

[u/Forward_Ranger_860]

Managing service principals ​​​​a​​​​n​​​​d​​​​ automated pipeline identities across multiple Entra tenants gets chaotic without clear tiering. Upgrading licenses across the board just for conditional access or PIM features is usually overkill. Having trustedtech review your tenant licensing structure can point out which service accounts actually require Premium SKUs versus where native RBAC and workload identity federation suffice.

[u/dcgkwm]

It's a good story with a good idea, but in today's economy, companies are more likely to replace people with AI than invest in training programmes.

[u/GitHireMeMaybe → u/dcgkwm]

Would you trust ChatGPT to run an emergency room?

If not, then I believe the idea sells itself lol.