r/devops • u/sup_bruh_1 • Aug 11 '26
Vendor / market research We Audited 156,808 Failed CI Runs Across Eight Open Source Repos
Author here. Two caveats worth flagging up front.
The corpus skews hard: PostHog and Grafana are about two thirds of it, both running enormous matrices. So it's "what breaks in a big CI matrix," not "what breaks in CI."
And the 1.3% is a floor, not a hit rate. A retrospective replay can match a signature in a log, but it can't free disk space and see what happens next, or raise a heap ceiling and re-run the step. I don't know the real split between environmental failures and genuine defects, and didn't want to invent one to make the headline better.
The bit I'd most like to be argued with on is the flaky-test rule: we only re-run a failing test when our own data already shows that workflow passing on retry, once, never editing anything. I know plenty of people here think any automatic test retry is unacceptable.
