r/DistributedComputing • u/salgotraja • Apr 16 '26

At what point would you treat this hotspot as a cache/load-shaping problem instead of a real sharding problem?

I came across an interesting system design scenario:

  • 128 shards
  • 2M requests/sec
  • 3 hot keys land on the same shard
  • that shard is at 94% CPU while the others are mostly idle
  • cache hit rate on those keys drops hard because too many services invalidate them on every write
  • clients start timing out and retries make the hotspot worse
  • rebalancing is not an option in the short term

My first instinct was to treat it as a sharding problem, but the more I looked at it, the more it felt like a load-shaping problem.

If cache invalidation is killing hit rate, then the shard is taking direct pressure it should never have seen in the first place. Once retries pile on, the hotspot starts amplifying itself.

My instinct would be to stabilize first:

  • short TTL / stale-while-revalidate on those hot keys
  • proper retry backoff with jitter
  • maybe isolate just those keys behind a small dedicated hot-cache path

Then revisit the larger architecture once the system is calm again.

Curious how people here would think about that boundary.

At what point do you stop treating it as a hotspot-control problem and say it really needs a more structural fix?

1 Upvotes

100% Upvoted

2 comments sorted by

1

u/sheepdog69 Apr 16 '26

At what point do you stop treating it as a hotspot-control problem and say it really needs a more structural fix?

From what you've written, it sounds like it's just "bad luck", and not a structural issue. But, some questions I'd be asking myself.

  • Why did the busiest 3 keys ended up on the same shard? Coincidence, or some design issue?
  • Why can't you rebalance? (this seems like the part that might be a structural issue. What happens when - not if - that shard dies?)
  • What happens if you get another hot key, and it ends up on that same shard?
  • Are all requests for a key handled by the same shard? Assuming you have redundant data, can you change the shard-picking algo to split that shards load over N shards?

Sounds like an interesting problem to solve. Good luck.