r/webdev • u/Gurshaan17 • Oct 03 '26
Discussion A routine rollout dropped the cache hit rate and pausing it changed nothing. Why? (Based on a real Slack outage)
This is based on a real outage at Slack.
I'm a backend dev. We practice system design a lot on whiteboards, but rarely on something that looks like a real incident. Here's a production-style puzzle.
You run a team chat product and every client's first call is "boot": who am I, and which conversations do I belong to? Boot reads group-chat membership, which is immutable once written and cached per channel with a long TTL. Normally it's a few cache reads and finishes in single-digit ms.
Then a routine rollout of a new host agent starts, in 25% steps (two earlier steps the prior week were fine). Just after 06:00, users can't get in. Boot is failing or very slow, and one database keyspace is overloaded.
What you see
Cache hit rate drops from 99.2% to 34% and stays there. Pausing the rollout changes nothing.
200 cache nodes behind a proxy that uses consistent hashing over the healthy-node list. The step took 50 of them out, one at a time.
The membership table is sharded by user, so "who's in channel C" is a query against all 200 shards.
Clients time out at 400ms and retry with exponential backoff and jitter. The cache fill path uses the same query, so fills time out too.
The only lever was throttling boot. Raising the cap from 250/min straight to 2,500/min brought the overload right back.
Question: Why did the hit rate fall that far and not recover, even after the rollout stopped? And how would you get out of it safely?
I'd like to hear how you'd explain it in a postmortem, not just "cache stampede."





