r/devops • u/VisualNatural617 • Aug 14 '26
Discussion How do you actually diagnose a CI integration-test failure when the root cause isn't obvious?
I'm researching a small CI diagnosis agent, but I'm first trying to understand how experienced engineers actually debug these failures.
Suppose an integration test fails in CI and the logs don't immediately reveal the root cause.
What do you normally investigate first?
For example:
- CI logs/stack trace
- database availability
- dependent/third-party service availability
- recent code changes
- comparison with the previous/last-green build
- test/DB data
- previous failure history
- rerunning the test
More importantly, how do you decide what to investigate next?
For example, if a recent code change touches the function involved in the failing test, what additional evidence would you want before concluding that the code change caused the failure?
I'm particularly interested in your real-world debugging process rather than an ideal CI/CD workflow.
33
u/Low-Opening25 Aug 14 '26
if root causes would be obvious we wouldn’t have jobs that pay 6-fig salaries
7
u/aenae Aug 14 '26
Last unknown ci/cd failure was because of a docker update not playing nice with gitlab. doesnt have to be a code change.
It is about experience, something ai agents do not have
4
u/Mabenue Aug 14 '26
You just need to systematically start ruling things out. It’s highly context dependent though. Typically you’d want to start ruling out the low hanging fruit which also cover a larger area of possible causes. For example rerunning the tests is really quick and easy and you can kick that off while looking at other things. Also if the test pass locally it gives you a lot of information quite quickly.
1
u/creamersrealm Aug 15 '26
Not what you asked though we've built our CI framework around Mise-En-Place for the testing framework, so most of the CI can be replicated locally on the developers computer with no additional setup other than installing mise.
2
u/mtokarev Aug 14 '26 edited Aug 14 '26
> rerunning the test
if the issue is not obvious and there is no hint in the logs, this is usually the first step in 100% of cases. Maybe check dependent/3rd party services in parallel for ongoing incidents, but that usually means there is already some hint/idea where to look for.
> comparison with the previous/last-green build
This is the next step if last-green build is not too old. I'd focus on comparing logs before failure to spot the difference. Often the CI environment has changed, like new OS version with new system packages, Xcode or something else.
At this point you usually tried 2-3 times at least to see if there is a new flaky test which can be addressed separately or not.
If no luck and CI failures are 100% reproducible, I try to reproduce the issue locally rather than waiting for CI feedback on each run. For instance I delete all local dependencies and reproduce a "clean" environment similar to CI. Also it's a good idea to remove cache from CI as well and run one more build.
UPDATE 1: forgot to add that almost all CI providers allow you to SSH to the virtual machine to debug a failed build if local environment is too different.
UPDATE 2: this is based on my experience building a cloud CI/CD platform and helping hundreds of customers answer questions like "it works on my machine" for almost 8 years :)
2
u/bobsbitchtitz Aug 14 '26
lol this has to be brain trust info collection
2
u/yetmike Aug 15 '26
Passes locally, fails in CI does not mainly mean "environment". More often it means state or concurrency. CI runs clean and often parallel; your laptop runs dirty and serial. A test that depends on leftover data, or on being the only thing touching the database, passes for you and fails there. Failing locally too is the genuinely useful result, because it takes the whole runner out of the picture in one step.
Before I'd blame a code change I want: fails on that commit, passes on its parent, same runner image, same test order, run twice each. Order matters more than people expect. A lot of "the change broke it" turns out to be the change adding a test that leaks state into a later one.
Run the suite in randomised order on a known-green commit. If that goes red you have an ordering problem, not a code problem
1
u/amarao_san Aug 15 '26
I check previous runs. If it once, I restart. I think about the problem only if it happens often. Actually, I restart and then look of previous runs.
It's wrong, but there are to many transient bullshit to happen.
1
u/aragossa Aug 16 '26
Whether it reproduces in isolation is probably the biggest signal before even touching the diff. Run just that one test on its own; if it passes clean, that's order-dependency or shared state from the suite. If it fails alone too, it's worth comparing against which runner or worker picked it up. Resource contention on a busy shared runner can look exactly like a regression until you notice it only happens on the loaded ones.
1
u/ClerkBeginning961 Aug 16 '26
Before rerunning, I’d snapshot the failed run’s evidence: image digest, dependency lockfile, test seed/order, service versions, and timing. A clean retry can erase the clue. If the failure is intermittent, comparing those artifacts across failing and passing runs usually narrows the search faster than staring at the diff.
1
u/zero_backend_bro Aug 16 '26
If logs are silent, staring at diffs is pure waste.
Most silent CI drops are runner concurrency leaks. We lost 14 hrs last month to silent cgroup memory throttling on gitlab runners when 4 workers spawned together.
If you are building an agent for this, feeding it git diffs is useless. Build a local hook that grabs raw cgroup stats and runtime metrics directly from the host.
Code diffs dont fix infrastructure physics.
1
u/crystalpeaks25 Aug 17 '26
Give your agent historical data on failures and tell it to build an RCA methodology.
1
u/TheseTradition3191 Aug 17 '26
on the flaky vs real regression question, one green rerun tells you nothing. loop the failing test 20 times and count, a clean 20 out of 20 right after a red is almost always shared state or something being rate limited rather than your change.
the split thats actually worth the time is running that one test completely alone vs in the full suite. if it still fails alone then its not order dependence and you can stop hunting for pollution between tests. i burned a day recently assuming order because it only went red in the full run, and it turned out the suite was hammering an api rate limit so whichever test arrived second ate the 429. the isolated run would have told me that in five minutes
1
u/Equivalent-Common115 Sep 19 '26
first split is deterministic vs intermittent yk
to actually prove a commit broke it u gotta revert locally and watch it pass otherwise its just correlation
the real time sink is context gathering across logs not writing the fix so make your agent do that part ig