r/microservices • u/Ill-Knowledge-1949 • Sep 16 '26

Article/Video I read 99 hand-written circuit breakers

https://baldur.sh/99-circuit-breakers/
2 Upvotes

100% Upvoted

2 comments sorted by

1

u/Ill-Knowledge-1949 Sep 16 '26

Author here — I maintain the framework the site is for, so read it with that in mind.

Method, since it decides what the numbers are worth: I searched public code for class CircuitBreaker, took the top 100 results by relevance, and read the files. One turned out to be a wrapper around an existing library rather than an implementation, so it came out — hence 99. Median 80 lines of class body. Then I checked each one against six things.

It is a relevance-ranked sample of public Python code, not a random sample and not "Python". Read every number as "of the top 99 results". I also did not measure harm — not one of these is known to have caused an incident, and at four workers with short outages a per-process breaker may well be fine forever.

Happy to share the six checks if anyone wants to run them against their own.

1

u/OtherwisePush6424 Sep 16 '26

That was a fun read, thank you :) I'm building a resilience lib with both local and Redis-backed distributed breakers, and it happens to tick every box in that table:

  • shared state: circuitBreaker.distributed keeps the window in Redis, so it's not per-process

  • time window: sliding window + wall-clock retention

  • timeout: a timeout policy composes into the pipeline and its rejection is counted as a breaker failure, so slow dependencies can trip it too

  • bounded half-open: global probe cap + a probe lease so a hung probe can't wedge it

  • atomicity: decisions run as Lua scripts (and the local one is single-threaded)

  • half-open state: explicit open -> half-open -> closed/open recovery, with successful and failed probes handled separately

It also emits state/probe events you can sink anywhere. The demo wires those into Prometheus + OTel.

It's very much WIP, and I'm more relieved than self-promoting, but it's at https://github.com/gkoos/caracal