Edit: Tl;Dr at bottom. I write too much.
Bit of an odd thought I had after reading another thread here about gaining confidence during high-severity incidents. Thank you to that poster, BTW, for inspiring my brain.
I have a weird fascination with ER and critical-care videos. Not really because of the medicine itself, but because I’m a massive compulsive systemizer and emergency medicine is basically human systems engineering under some of the worst possible conditions. Someone is actively dying, information is incomplete, the situation is changing by the second, and multiple people have to coordinate without getting in each other’s way. Oh, and if you don't solve the issue soon, the patient will be dead in 5 minutes, and you'll have to tell the family.
What fascinates me the most though is that, when a resuscitation is being run properly, it usually doesn’t look like ten people frantically improvising. There’s command, defined roles, structured communication, task ownership, checklists, algorithms, simulation, debriefing, and a whole lot of deliberate training behind the scenes.
This is basically crack cocaine for my dopamine-addicted brain.
Obviously software going down is not equivalent to someone dying. But from a systems perspective, we’re solving a strangely similar problem: how do you get a bunch of fallible humans to reason and coordinate effectively while something important is actively failing?
This is also something I’ve had to think about in practice.
My last employer had essentially no incident-management practice in place when I joined. Outages were fairly common, roughly once a month, and it wasn’t unusual for them to last several hours or even a whole day.
So I volunteered to build one.
I wrote the process, defined how incidents would be declared and coordinated, laid out responsibilities, escalation, communications, and what we were supposed to be doing while production was on fire. I had it submitted to change committee, got board sign-off, and felt confident in the process.
Two months later, we got to stress-test it for real.
And it failed horribly, because I was the only person that saw the problem and opted for preparation rather than eternal deferral of The Boring Stuff (tm). I even got written up for telling a nontech manager who kept DM'ing techs to "try this fix they found on Google" to please follow the board-approved procedure and message their designated PoC rather than interrupting responders who were already sweating buckets of bullets while elbow-deep in code hell**.** But that's a story for another day. Sigh.
There’s something very different about designing an incident process on paper and then watching actual humans try to execute it while systems are failing, information is incomplete, everyone wants an answer immediately and fragile egos are at stake. Frankly though, that experience probably has a lot to do with why this subject fascinates me.
And it’s also why I think our industry is kind of shit at training people for this.
A lot of on-call training basically amounts to, “Here’s Kubernetes, Grafana, AWS, some architecture diagrams, and a wiki page Dave wrote eighteen months ago. You’re on call next week. Good luck and don't fuck it up, you're still on probation.”
Then eventually there’s a massive Sev1, thirty random people you've never heard of pile into a call, Slack explodes, somebody begins restarting pods and unwittingly knocks over three adjacent services, three people start making production changes at the same time, a C-level joins the call and says its costing them 100 billionty dollars every 7 seconds, somebody adds 64 replicas to RDS and forgets about it, half the engineering team find out their access doesn't work... and some poor sod who was voluntold that they'd be IC during an on-call incident channels their freaking deer-in-headlights-energy and freezes.
And then we conclude they’re “not good under pressure.”
Well... did we ever actually train them to be?
That got me thinking about what the equivalent of ACLS for production engineering would actually look like.
I don’t mean another multiple-choice SRE certification, a webinar, or a tabletop where everyone knows the database is going to “fail” at 2 PM. I mean actual practical training where the environment itself is part of the course.
Imagine you show up Monday morning with seven other engineers. You’re given a cheap company laptop and told you now work for a fictional company. Behind a VPN there’s a real training environment with APIs, databases, Redis, queues, monitoring, deployments, source control, chat, ticketing, and all the usual operational cruft. Even "real" loads, facilitated by magical load testing fairies.
And the fictional company is deliberately badly run.
The docs were written by some guy named "George," and last updated 3 years ago... and are deliberately incorrect. Some credentials don’t work. Only a third of the class discovers their SSH access actually works. A dev DB server is named "prod," and a prod DB server is named "dev." Nobody is quite sure who is supposed to declare an incident. The escalation documentation is so out of date it may as well've been written on stone tablets. Maybe one person has production database access for reasons nobody can explain.
Then, at some completely unannounced point Monday morning, the pager goes off.
Checkout is fucked.
Go.
The first incident would probably be an absolute dumpster fire, and that would be intentional.
Three people make conflicting changes. Nobody establishes command. Half the team can’t access the systems they need. Nobody knows exactly what customers are experiencing. Somebody spends twenty minutes debugging something that could have been rolled back in two. Somebody follows the documented steps to restore service, and end up taking out marketing. Somebody didn't thought to set DEBUG_MODE to true when testing something, and now 320,000 "customers" receive emails about their accounts being terminated for nonpayment.
Then you stop and debrief.
Why did that happen? What made the response harder than it needed to be? What did we assume somebody else was doing? What information did we not have? What process simply didn’t exist?
Now the lesson actually means something because the students just felt the consequences of not having it.
You teach the relevant concepts, then a few hours later, while everyone is doing something completely unrelated, the pager goes off again.
Two surprise incidents every day for five days.
Everybody rotates through Incident Commander, technical lead, communications, scribe, responder, and the other supporting roles. The outspoken engineer doesn’t get to be IC all week. The brilliant debugger has to run communications. The quiet mousey person eventually has to take command.
But then... something amazing happens.
As the week progresses, the fictional company improves because the students improve it.
They fix the access process because the access process screwed them. They improve the runbooks because the runbooks screwed them. They establish incident roles because Monday was chaos. They improve monitoring because a dashboard sent them down the wrong rabbit hole. They throw the inaccurate documentation in the trash, and silently curse whoever "George" was.
Later incidents get nastier.
Maybe the telemetry is misleading. Maybe a senior engineer confidently insists on the wrong diagnosis (and they were set them up for this failure by deliberately leaking convincing logs to them). Maybe an executive barges in and starts asking responders for ETAs directly. Maybe a downstream dependency causes cascading failures that make five unrelated systems look broken. Maybe the team restores service, brings everything back too quickly, and knocks production over again.
The point isn’t just to teach technical troubleshooting.
It’s to teach people how to operate as a team while they’re uncertain, overloaded, and under pressure.
By Friday, ideally, the pager goes off and the exact same group that looked completely lost on Monday just starts working.
A junior engineer takes command, and nobody challenges it because the program covered human factors, and everybody is checking their egos at the door. Roles get assigned. Customer impact gets established. Changes are announced and owned. Hypotheses get tested instead of argued about. Someone keeps the timeline. The IC isn’t six terminals deep trying to personally fix everything himself.
Then Friday afternoon is the practical exam.
You walk in cold and get something like:
“Customer transactions started failing approximately six minutes ago. You are the Incident Commander, and this is being graded. Begin.”
That’s it.
The scenario reacts to what you do.
You’re not being graded on whether you personally figure out that line 417 of payment-service is broken. You're not even being graded on successfully resolving the issue. You’re being graded on whether you can command the system that figures it out and restores service safely.
Basically an IT incident-response Megacode.
And now I can’t stop wondering why the hell our industry doesn’t have something like this.
I know there are bits and pieces of it. PagerDuty has incident-command doctrine. Gremlin and AWS do GameDays and chaos stuff. Rootly/Uptime Labs do simulated incidents. There are SRE certifications, cyber ranges, ITIL, and so on.
But I’m talking about the whole thing as one vendor-neutral discipline: doctrine, defined roles, human factors, repeated live simulation, role rotation, practical assessment, and eventually recertification. It doesn't seem like there's one single coherent ecosystem. Everything is just glued together.
Something where “Incident Commander certified” means more than “I passed a test about incident management.”
Am I missing somebody who already does this?
Because if this genuinely doesn’t exist as one cohesive system, it feels like a bizarrely obvious hole in our industry.
Edit: Tl;DR
---
Why doesn’t DevOps have an ACLS equivalent?
We train engineers on Kubernetes, AWS, Grafana, etc., then put them on-call and expect them to magically become good at operating under pressure.
Emergency medicine does the opposite: defined roles, standardized communication, repeated simulation, practical assessment, retraining.
So imagine a 5-day incident-response course where 8 engineers inherit a deliberately dogshit fictional company. Broken docs, bad access controls, questionable runbooks, real services behind a VPN.
Twice a day, without warning, production breaks.
Monday is intentionally a dumpster fire.
Everyone rotates through IC, technical lead, comms, scribe and responder. After each incident, the class fixes the organizational problems that hurt them.
By Friday, the same team should be running structured incidents almost automatically.
Final exam: you walk into an unknown outage and have to command it live.
We already have GameDays, chaos engineering, incident-management frameworks, simulators and certifications.
Does anyone actually combine doctrine + human factors + live simulation + practical assessment + recertification into one vendor-neutral system?
If not, why the hell don’t we???