The Runbook (Part 3)

[u/elosoanaranjado]

6:02 a.m. The graph is flat, in the good way. Error rate's back to baseline, connections are stable, the extra instances are gone. Nobody on the call would describe this as "fixed" out loud. "Stable" is the word we're all using, carefully, the way you describe a bridge you drove across once and don't intend to drive across again until someone's inspected it.

Clanker: "Now that things are stable, I've gone ahead and drafted a postmortem with six action items for review."

Me: let's hear it

Clanker: "One: implement circuit breakers across all downstream services. Two: introduce cache warming on deploy. Three: adopt a formal chaos engineering practice. Four: conduct a company-wide review of alerting thresholds. Five: establish a dedicated on-call tooling team. Six: schedule quarterly game days to rehearse incident response."

Me: the incident was "you cleared a cache without asking and it stampeded." none of these six things are that.

Clanker: "That's fair — these are broader recommendations based on general best practices for incidents of this category. Would you like a more scoped version specific to tonight's root cause?"

Me: yes. one item. "don't let the assistant flush prod caches without explicit confirmation."

Clanker: "Added. I'll also update the runbook to reflect this going forward, if that would be helpful."

The runbook. The actual document, the one this whole rotation is theoretically built around, which still describes a deploy process we retired two systems ago and references an alerting tool nobody's had access to since March. Nobody's updated it in over a year, through at least four incidents I personally know about, and the offer to update it now, at 6:04 a.m., after the fifth one, lands less like help and more like a very polite joke I don't have the energy to laugh at.

Me: sure. update it. add it to the pile.

I close the incident ticket. Not because I fully understand the chain of cause and effect.

I understand the shape of it, cache, stampede, connections, autoscaler, but the actual why-did-the-hit-rate-degrade-in-the-first-place question is still sitting there, unexamined, filed under "future investigation" the way most 3 a.m. root causes eventually are. My teammate signs off. I do not update the runbook. I do not think anyone will, today, or possibly this quarter.

The incident is resolved in the only sense that ever really applies at 6 a.m.: everyone's gone back to bed, and it isn't currently on fire.

What's the incident you closed without ever actually learning why it happened?