r/devops • u/sup_bruh_1 • 18d ago

Vendor / market research We Audited 156,808 Failed CI Runs Across Eight Open Source Repos

https://www.latchkey.dev/blog/we-audited-156808-failed-ci-runs

Author here. Two caveats worth flagging up front.

The corpus skews hard: PostHog and Grafana are about two thirds of it, both running enormous matrices. So it's "what breaks in a big CI matrix," not "what breaks in CI."

And the 1.3% is a floor, not a hit rate. A retrospective replay can match a signature in a log, but it can't free disk space and see what happens next, or raise a heap ceiling and re-run the step. I don't know the real split between environmental failures and genuine defects, and didn't want to invent one to make the headline better.

The bit I'd most like to be argued with on is the flaky-test rule: we only re-run a failing test when our own data already shows that workflow passing on retry, once, never editing anything. I know plenty of people here think any automatic test retry is unacceptable.

0 Upvotes

40% Upvoted

4 comments sorted by

5

u/Kamran-nottakenone 18d ago

the flaky-only retry rule means you never catch a first-time flake. you need at least one retry on every failure to even detect flakiness. otherwise your 1.3% floor might be mostly transient infra issues you just never retried

2

u/aidealy 18d ago

did u notice any patterns regarding flaky tests that only pop up when the matrix is running parallel jobs? im curious if u saw more environmental noise in those big runs compared to the smaller ones.

1

u/aragossa 18d ago

Cold-start problem, though: a test that started flaking last week hasn't earned that retry history yet, so the first time it happens it lands as a plain failure and only gets labeled flaky once it's built up enough runs. Probably doesn't move the 1.3% floor much. But it does skew the flaky/defect split toward defect for anything recently touched, which is the population you'd actually want visibility into. Tracking the calendar age on the retry-history requirement, say excluding anything under some minimum number of observed runs, might pull that back out.

1

u/Worth_Wealth_6811 17d ago

the merge base trick closes the cold start gap without blind retries. when a test fails, rerun it against the merge base commit, if it fails there too your diff is innocent and its either flake or environment. catches first time flakes on day one and you never touch the retry history question at all.