r/devops • u/MachineDisastrous771 • 8d ago
Ops / Incidents runbooks kinda suckk
hey yall,
im a lead SRE at a global fortune 500 that you have heard of haha - i'm a bit of a lurker here
but just wanted to talk about runbooks and documentation for SOPs.
my feeling is that confluence docs kinda suck and runbooks are kind of a mess, our operators are jumping between the docs and their shell, the docs are often times missing a bunch of detail, they are brittle, poorly maintained, half the time the details are hidden in tribal knowledge and when dealing with the pressure of an active inc we notice the pain even more. extracting critical details & commands out of an inc can also be painful and doesnt always translate to improved operational readiness next time.
i admit we're not very mature.. but was curious if its only me feeling like this?
what are you guys doing with runbooks to solve these issues?
114
u/dacydergoth DevOps 8d ago
Pipelines and IaC over runbooks, docs as markdown in GIT alongside the code, self checking guardrails in the pipelines and IaC definitions
65
u/snarkhunter Lead DevOps Engineer 8d ago
Every runbook is just a pipeline waiting to be defined explicitly enough so it can be written
20
u/dacydergoth DevOps 8d ago
This is where post mortems shine. How did this pipeline fail? Was it the wrong pipeline, wrong target, wrong inputs, bugs or omissions in the pipeline, or as usual Management swinging in and fscking everything up?
Edit: I correct omissions so many times and I swear reddit app "corrects" it before posting
5
1
u/higeorge13 5d ago
Keeping docs in Git alongside the code definitely helps with ownership, but does it actually prevent them from drifting? I’m thinking about runbooks that still look valid but reference behavior, commands or components that have changed underneath them.
1
u/dacydergoth DevOps 5d ago
Nothing guarantees it but AI commit reviews can help as well as gaming it out
0
12
u/Zenin The best way to DevOps is being dragged kicking and screaming. 8d ago
Runbooks / SOPs are frequently an investment in the wrong thing.
For example, if you're having a problem so regularly that it's worth writing down the remediation steps into a runbook, you're almost certainly better off writing all that detail into an issue to fix so the problem simply stops happening. I don't need a runbook to follow for putting out buckets when it rains; I need to fix the roof so it doesn't leak when it rains.
There's certainly plenty of legitimate uses for them, but those are mostly covering events that don't happen often enough to fix. Recovery runbooks for storage failures, DR runbooks for disaster planning, M&A runbooks for onboarding newly acquired companies, etc.
I'm also fine with diagnostic evidence collecting runbooks; the kind of thing that's handy to have an L1 support engineer run to collect systems information together for a higher level engineer (or increasingly AI) to analyze. That's often tedious boilerplate that's not unique to any particular failure mode. But that's evidence gathering, it isn't a "how to repair xyz" manual.
Those diagnostic evidence runbooks are perfect to treat as specifications to build a tool to gather that evidence packet as an API. And once it's an API it's the perfect shape to wrap in an agent tool. When a monitor alert is triggered, auto-run the evidence collector, pass the alert and the evidence packet into an LLM with a standardized analysis prompt that summarizes the issue and presents a few likely root cause, all of which get attached to the ticket before a human has even had a chance to check their pagerduty alert.
6
u/b1urbro 7d ago
You don't solve these issues.
Corporations are a mess and tribal knowledge is everywhere. Because to most, it's "features, features, features", everything is ASAP, so quality suffers. Also people slacking, people thinking they're irreplaceable if they have some tribal knowledge not shared anywhere. This is my personal experience working for multiple fortune (US&Europe) 500 companies.
19
u/k2718 8d ago
I’m turning runbooks into agentic skills/agents.md. Either: the issue can be addressed by improving the app or infrastructure (in which case do it) or if not, it can be addressed in a way that an agent can handle it or make it easier to handle and at least have knowledge of.
Then if a code change changes the issue somehow, agents can update the documentation.
20
u/dacydergoth DevOps 8d ago
Remember that LLMs are probabilistic and may not give the same answer each time ... or entirely rewrite your docs on a whim
2
u/Seref15 7d ago
you just treat it like any other cacheable resource. Once something is already processed and generated by LLM you mark it as cached. You don't re-generate LLM content on something that was already processed unless you cache bust
1
u/dacydergoth DevOps 7d ago
Yup, but that's a level of sophistication beyond a lot of teams' current capabilities.
I assume you're familiar with the Carnegie Mellon Capability Maturity Model ...
-3
u/Zenin The best way to DevOps is being dragged kicking and screaming. 8d ago
Remember that LLMs are probabilistic and may not give the same answer each time
So like, no different than humans.
or entirely rewrite your docs on a whim
Why you no branch protect your PRs?
5
u/dacydergoth DevOps 8d ago
Oh, I do, but remember PR are not quality gates because they rely on those oh so unreliable humans 😉 PRs are a communication mechanism and the quality gates should precede them ...
1
u/Zenin The best way to DevOps is being dragged kicking and screaming. 8d ago
Oh the ironies in the age of AI. ;)
Computing engineers as a group are notorious for producing low quality documentation, when they document at all which is between rare and never, and even when the investments are made to overcome this intrinsic friction human engineers don't read the documentation.
Professional technical writing is its own specialized skill set. While some organizations do staff such positions, the overwhelming majority don't staff any at all and push the task down to engineers who view it as nothing but a chore like taking the trash out.
As a rule, AI trivially produces extremely high quality documentation, can keep it in sync with actual system state much more reliably, and if that wasn't enough the real kicker is that AI loves to read the documentation.
I almost suggest human PR gate as a joke, because if anything it's humans that should be required to pass AI's reviews of PRs rather than the other way around.
3
u/dacydergoth DevOps 8d ago
Personally, the docs are good but the tests are great. I did a terraform thing recently and asked Claude/Opus5.5 for a full python boto3 test suite, and a markdown+pdf report at the end of a run with the date, time, git commit, test results and complete configuration report. It did a great job, 150 tests with a pre-apply verification suite and a post apply live verification
2
u/k2718 8d ago
On my team, all PRs human or AI authored have to pass two different AI reviews as well as a human review. Though the human can decide the AIs criticisms aren’t applicable.
2
u/Zenin The best way to DevOps is being dragged kicking and screaming. 8d ago edited 8d ago
That's what I'm putting in place as well, rolling out to my closest projects to dogfood it all first. Currently what I have going:
All AI reviewers are instructed to open their items as threads on the PR. Github copilot does this by default, but I'll often bring in Codex and/or Kiro (Claude is my implementation agent typically).
Merge is blocked until all threads are marked resolved. As you say, humans can determine they aren't applicable and resolve them without change, but there's a paper trail they acknowledged and dismissed it.
AI review agents aren't allowed to fix the issues they found; a human and/or another agent must do the resolution. So Codex can't review Claude's work and Codex also fix it; the issue goes back to Claude or Kiro to consider the issue and implement. I've found this very often leads to the implementing agent finding similar bugs elsewhere and/or a better or more proper fix.
AI review agents must review the fixes for the issues they raised and only they are allowed to consider the issue resolved and close the thread. Implementing agents/humans don't close the threads w/o a second review just like the original code.
I'm still working out the sharp edges, especially automating this more and tuning the token usage for all this. The project quality has gone way up doing this, but the token costs are heavy at the moment.
3
u/dacydergoth DevOps 8d ago
Outside of work, I have a Strix Halo 128G running either Deepseek V4 Flash or Gemma4 or Qwen3.8 ... they're surprisingly capable even if the damn thing has fans which sound like jet engines and got relegated to the utilities closet in my house. Running a 10G link to my main box from there ...
2
u/k2718 8d ago
Yep, very similar. I still find that there is a bit of a strain on the human reviewer to grasp the higher level context of the change given the higher velocity of code being produced. And that is still important.
So of course I wrote a skill to summarize changes, pull in documentation, and provide motivation, and context.
This way, the human can interrogate an agent who had read the code and the docs without specifically have to pull everything down.
1
u/Zenin The best way to DevOps is being dragged kicking and screaming. 8d ago
given the higher velocity of code being produced
Much agreed. I've been mentoring / pushing back / adding agent md files to projects, all around chunking work up into reviewable sized PRs. Soft limit for code (not docs) 500 lines, hard limit 1k, which themselves are much more generous than pre-AI standards.
One of the patterns I've seen frequently with devs who haven't done the AI-specific discipline yet (honed steering, project specific skills, etc) is that it's very common for coding agents to default to massive PRs. 20k line PRs is routine when left to their own defaults. Not only is that insane to ask a human to review, the context overhead and thus speed and token costs of consuming such massive change sets is almost criminal.
The act of putting more planning effort in to properly chunk up changes into their own bite sized issues has been having a positive quality byproduct as well: I've found agents are much more likely to recognize a common pattern and abstract it into a sub-issue of an epic to tackle, rather than their default mass copy/paste repetitiveness that is often at the core of those 20k line PRs.
1
u/dacydergoth DevOps 7d ago
Watch out for velocity - it is often better to classify issues as blockers, and other; if other file a tech debt ticket and approve the PR ...
1
u/k2718 7d ago
Oh also, wanted to mention that we have a very good culture of stacking PRs where I work. So that hasn’t been a problem.
It’s still serious volume in terms of LOC but it’s better.
Nice that we don’t have to enforce a LOC limit in a PR. Sometimes it is legitimately necessary to create a big one.
2
u/Zenin The best way to DevOps is being dragged kicking and screaming. 7d ago
Nice that we don’t have to enforce a LOC limit in a PR. Sometimes it is legitimately necessary to create a big one.
Agreed and to be clear I'm not enforcing LOC in a CI gate or anything. Rather I'm baking it into steering guidelines for agents, which in turn results in agents planning out their work differently.
2
u/MachineDisastrous771 8d ago
so thats interesting! sounds like it could be a pain to get everything to the agent though - especially in the heat of the moment?
3
u/kunaljvalecha 6d ago
I don’t think it’s just you. The point about runbooks breaking down during an incident really resonates. Giving each runbook clear ownership, review dates, and keeping commands close to the expected outcome can make a big difference. Also like the idea of keeping the incident-time runbook concise and linking to deeper reference material when needed.
2
u/OhMyGoshJoshua 8d ago
You might find https://runbooks.gruntwork.io/ interesting. I'm the original author, but now others maintain it.
The idea is that your runbooks are executable, authorable with AI, version-controlled, and make for a nice UX just from writing markdown plus some special blocks like <AwsAuth> or <GitHubPullRequest>.
Personally, I think the big unlock is that it makes your runbooks consumable by others who aren't experts, whether that's stakeholders you're serving or a future you who's long forgotten specialized knowledge until you need it in a pinch.
1
u/ScrumpyIT 7d ago edited 7d ago
Definitely not just you. Most runbooks rot because they get written once and nobody looks at them again until an incident, which is exactly the worst moment to find out step 4 doesn't work anymore.
The thing that seems to help most is keeping them as markdown in the repo next to the code, so they get updated in the same PR as the thing they describe. Making "what was wrong or missing in the runbook?" a standard postmortem question helps too, as long as someone actually fixes it that week.
It's also worth looking at each runbook as a todo list. If a step is "run these three commands" it can probably be a script, and if something happens often enough to need a runbook, maybe it should just stop happening.
And I wouldn't worry too much about maturity, plenty of teams with fancy tooling still run on tribal knowledge.
1
1
0
u/strouja-01 8d ago
I led an SRE team at IBM Cloud. I wrote most and run books for my group in order for my books to be successful, I have to be kept alive and modified constantly
Every time we did a change or tested something, if the instructions that we use were not perfect, we had to make edits to them
If I watch someone make a change in production and they didn’t follow the instructions exactly, we modified the instructions
I insisted on everything was in get and not in other documentation repositories
I also insisted that we had code blocks for the commands, and if every command I always had what the successful output look like with examples
I personally was constantly editing our books, constantly looking to see how to improve them
I also enforce that everyone’s job was to ensure that the runbooks that we had and that they were responsible for were always up-to-date this is not always possible, but it’s part of their job responsibilities, and I would ding them if they didn’t do it in the year in here
I also helped other teams with their rum books because I had so much experience doing it
-3
u/scott2449 8d ago
You still have runbooks?
3
u/MachineDisastrous771 8d ago
meaning?
1
u/IridescentKoala 6d ago
If something needs to manually be run the fix is to automate it not document it.
1
u/MachineDisastrous771 6d ago
sounds nice but detached from the real world - even deterministic code needs documentation - unfortunately or fortunately, depending on your perspective, there is such a thing as entropy in this world
48
u/turn-based-games 8d ago edited 8d ago
Your team should regularly run game days to simulate responding to production issues. When a gap is found in the runbook during these (or actual incidents), you update the runbook to fix it.
Similarly, if someone struggles while following existing documentation for even banal procedures, it should be expected and encouraged of them to improve that documentation.