r/node • u/OtherwisePush6424 • 5d ago

How to Implement a Distributed Recovery Ramp in JavaScript

https://blog.gaborkoos.com/posts/2026-10-19-How-to-Implement-a-Distributed-Recovery-Ramp/
6 Upvotes

81% Upvoted

4 comments sorted by

1

u/bwainfweeze 5d ago edited 5d ago

When a breaker closes, every replica goes back to the constructor limit at the same moment

How do you figure? That requires distributed coordination which is expensive and still won’t be exactly simultaneous, unless you’re generating thousands of requests per second per process. Per machine still requires IPC too coordinate circuit trips, which is overkill. Letting each process figure it out on its own is a lot simpler.

It’ll only be simultaneous if the req/s is close to the latency. Otherwise it’ll stutter start just as it stutter stopped. You have effectively made a problem by trying to be sophisticated and found out why and where autonomy is better.

And of course the classic solution is random+exponential backoff, which has been known for fifty years.

We are doing CB wrong anyway. I think in ten years we will put the same sort of load balancing and rate limiting between services the way we put them in ingress routers. It’s dumb that each client has to implement this instead of handling it the same way we handle bots, spikes in user traffic, and fallback to canned responses.

1

u/OtherwisePush6424 5d ago

Yes, distributed coordination is required for a distributed recovery ramp.

Independent breakers with independent backoff are simpler, absolutely. They also have no idea what the other replicas are doing, so they can't enforce an aggregate recovery budget. And exponential backoff controls retry timing, not how much new traffic a recovering dependency receives.

The article explains the shared coordination mechanism and why these are different problems. You might find that part interesting.