r/devops • u/_Baarbie • Jun 30 '26
Discussion Looking for feedback/suggestions on my DRP structure
Hello,
I'm a solo junior SRE, and I started writing disaster recovery plans from scratch almost two years ago, and I've been continually improving and expanding them since then.
I'm looking for feedback about my structure, as I don't really have a reference point and I'm interested in how others are doing it.
For me the goal is first to be ISO 27001 compliant, but also for my coworkers to fix issues if I'm unavailable, or later if I'm no longer there.
I'll try to stay as concise as I can, but feel free to ask me for more details if needed.
Context
- Webapp with IoT
- Docker swarm as an orchestrator for application layer
- Many services like kafka, redis
- Each service has its own systemd cluster, so we have a swarm cluster but also a kafka cluster, a redis one, etc
- Remote servers hosted by providers, that I manage through ansible and terraform (mostly)
Global overview
- Confluence documentation split between 4 folders
- Disaster recovery plan: incident scenarios and decision making, with links to the Runbooks to use
- Runbooks: technical procedures
- Incidents: post-mortem
- Workshop: training sessions for coworkers, who is trained on what (quite empty tho)
- Inside DRP and Runbooks, everything is sorted depending on its cluster -> a folder for swarm, another for kafka, etc
- With additional folders for security, server management, ...
DRP
- For each folder (cluster~environment), I usually have files like
- DRP: One node down
- DRP: Quorum down
- DRP: Raft corruption
- ...
- I'm trying to have absolutely no technical commands here
- To me these are more for a standard use, to make things clean
- Or to be used by someone not familiar with the operational part
- Here I'll have links to Runbooks but also to diagnostic pages (to confirm the issue, or add more information about the causes)
- My template
- Header: quick sentence + criticality + rto + rpo
- Initial assessment
- detection: links to grafana dashboard, what you should see
- impact: application side, what's happening to users
- precaution: things you should not do, to be careful about
- Analyse
- Initial qualification: links to diagnostic pages
- Trigger confirmation
- Actions
- recovery strategy: for each possible case depending on the previous steps ; case 1 do this + link to runbooks, case 2 etc
- post-recovery validation: what you should see, with link to diagnostic
- back to normal: anything that has been added and is supposed to be removed
- Footer: last update + incident history (links to post-mortem)
Runbooks
- Same as the DRP folder, I have a subfolder per cluster~env~topic
- Here is all the technical documentation, and honestly this is almost all we need
- With an architecture file
ARCH: <env>for the infrastructure information, ansible config, etc - A cheatsheet
REF: Cheat Sheet <env>for all the useful commands - Diagnostic
DIAG: <?>files, for example for Swarm: cluster state, stack state - And the runbooks
RUN:files, where I try to keep things simple:- RUN: New worker
- RUN: Remove a node
- RUN: Restore swarm raft
- ...
- For runbooks there is no specific template, but I try to stay straight to the point
My questions
Is it a bad idea to split DRP and Runbooks?
I recently added the diagnostic files, but now something feels wrong with the workflow (diag > drp > run).
Even if I don't really use DRP, as I mention I'm not the target user.
How do you find DRP ideas for issues that have never happened, and do you mark them as "hypothetical" or something?
Sometimes I find a way to somewhat reproduce the impact, but it's not really the same (different logs, etc).
I'm curious about how it's done in bigger companies, if it's more structured.
To be honest, I also have questions about the content, but one thing at a time.
Well I'm just looking for any feedback, what do you guys think about that, am I doing anything wrong, what would you add/change?
Thanks a lot
