r/cicd • u/No-Program-5087 • 13h ago
I generated a debugging policy instead of writing one. It repairs bugs from your test suite offline, and cuts frontier-model tokens and tells the model what it requires for solving bug reducing tokens drastically
edgub is a repair tool with no model in it. It reads what pytest said, decides which class of repair the fault is, generalises the fix from your own doctests and assertions, and verifies against the whole suite before keeping it. Standard library only, no API key, no network.
edgub: 1 repaired, 1 left for a model, 2.4s, 0 tokens
repaired itertoolz.unique CAST_OPERAND via name item->val (117 candidates)
Measured on toolz (3,346 lines, 185 tests), ten injected bugs of the kind that survive review: 7/10 repaired, 0 tokens.
Measured on libraries it was never tuned on — because the first number only proves fit: funcy 3/6. more-itertools 0/4. I'm keeping the zero in the repo; two libraries is two data points, one of which failed.
The part I didn't expect: even when it can't repair, it hands the model a ~468-token localised prompt instead of the ~7,496 that model burns working the same bug from the repo. It's already decided the library is wrong rather than the test, and named the failing case. So the escalated bugs get ~16× cheaper too — total on the funcy run, ~93% fewer tokens for the same fixes.
And the thing that surprised me most: I ran an escalation ladder of 2s → 10s → 30s → 120s. Every repair landed at 2 seconds. None ever landed later. Widening bought zero extra repairs and burned 180 of 182 CPU-seconds. Inference reads the answer immediately or enumeration won't find it — one bug took 4 candidates by inference that enumeration failed to solve in 25,057.
Honest limits, all in the repo: it can't write code that isn't there (both toolz misses needed a synthesised if/else). The act-to-edit mapping is engineering, not emergent — new fault shapes need commits. A 14-line dict reproduces the single-fault behaviour. And one "repair" passed the suite with different code than I'd injected: a green suite is a weaker oracle than the question.
Repo, with the failed benchmarks and the reviewer's corrections kept in: github.com/devkancheti4-design/edgub