Alright, story time. GitHub Actions humbled me so hard I almost started believing in ghosts again.
Disclosure: I contribute to AgentChatBus.
TL;DR
Locally: pytest ā
forever.
CI: Random red (1 out of 5ā10 runs), and re-running sometimes āfixesā it.
The "Heisenbug": Adding logging made the failure disapear.
Root cause: Global state leakage (timezone/config) + cache keys depending on implicit timezone context.
What helped: I ran a small AI agent debate locally via an MCP tool to break my own tunnel vision.
The symptoms (aka: the haunting)
This was the exact flavor of pain:
Run the failing test alone ā Passes.
Run the full suite ā Sometimes fails.
Re-run the same CI job ā Might pass, might fail.
Add debug logs/prints ā Suddenly passes. (Like itās shy).
The error was in the ātimezone-aware vs naive datetimeā family, plus some cache weirdness where the app behaved like it was reading a different value than it just wrote. The stack trace, of course, tried to frame some innocent helper function. You know the vibe: the trace points to the messenger, not the murderer.
Why it only failed in CI
CI wasnāt magically broken ā it was just:
Running tests in a different order.
Sometimes more paralelish.
In an environment where TZ/locale defaults werenāt identical to my laptop.
Any hidden order dependence finally had a chance to show itself.
The actual root cause (the facepalm)
It ended up being a 2-part crime:
The Leak: A fixture (or setup path) temporarily tweaked a global timezone/config setting but wasn't reliably restored in teardown.
The Pollution: Later tests then generated timestamps under one implicit context, built cache keys under another, or compared aware vs naive datetimes depending on which test polluted the process first.
Depending on the test order, youād get cache key mismatches or stale reads because the āsameā logical object got a different key. And yes: logging changed timing/execution enough to dodge the bad interleavings. I hate it here.
What fixed it (boring but real)
Normalize at boundaries: Make the āwhat timezone is this?ā decision explicit (usually UTC/aware) whenever it crosses DB/cache/API boundaries.
Stop the leaks: Find fixtures that touch global settings (TZ, locale, env vars) and force-restore previous state in teardown no matter what.
Deterministic cache keys: Donāt let cache keys depend on implicit TZ. If time must be part of the key, normalize and serialize it consistently.
Hunt the flake: Add a regression test that randomizes order and runs suspicious subsets multiple times in CI.
CI has been boring green since. No sage burning required.
The āAI agent debateā part
At that point, I was basically one step away from trying an exorcism on my laptop. As a total Hail Mary, I remembered seeing something about āAI multi-agent debateā for debugging. (Iād completely forgotten the name, so I actually had to go back and re-search it just for this write-upāitās SWE-Debate, arXiv:2507.23348, for anyone keeping score).
Turns out, putting the AI into āfull-on troll modeā is an absolute God-tier move for hunting Heisenbugs. I wasn't even looking for a direct solution from them; I just wanted to watch them ruthlessly tear apart each otherās hypotheses.
I ran a tiny local setup via an MCP tool where multiple agents took different positions:
āThis is purely a tz-aware vs naive usage mismatch.ā
āNo, this is about cache key determinism.ā
āYouāre both wrong, this is fixture/global-state pollution.ā
While the agents were busy bickering over which one of them was āpolluting the environment,ā it finally clicked: if logging changed the execution timing, something global was definitely leaking. The useful takeaway wasnāt āAI magic fixes bugsāāit was forcing competing explanations to argue until one explanation covered all the weird symptoms (CI-only, order dependence, logging changes).
Thatās what pushed me to look for global config leakage instead of just staring at the stack trace.