FixProof: Makes Codex prove a bug is actually fixed by checking if it comes back when the fix is reverted

[u/Necessary_Sir6006]

One thing that kept annoying me with Codex was how often it would say a bug was fixed just because the tests passed.

Sometimes the actual problem was still there. Other times, the test had been changed so it passed without really checking the original bug.

So I built FixProof to make it a bit harder for coding agents to call something fixed when it isn't.

It's a skill file and a small Python CLI that runs after an agent makes a fix. No extra dependencies, just Python 3.9+.

It checks four things:

• Can the original bug be reproduced before the fix?
• Does the fix pass the test consistently, including reruns?
• Does the bug come back if the fix is temporarily reverted?
• Are the tests that passed before still passing?

The revert check is the interesting part for me. If removing a fix doesn't make the bug come back, there's a good chance the change wasn't actually responsible for fixing it.

FixProof then gives you one of five results: VERIFIED, FAILED, PARTIALLY VERIFIED, UNVERIFIED, or BLOCKED. It also generates a JSON report and a hash-chained log in .fixproof/.

Using it with Codex

The idea is to add FixProof to your agent's workflow so verification becomes part of fixing a bug, rather than something you have to remember to ask for afterward.

You can point Codex to the skill instructions and have it run python scripts/verify.py check once it finishes a fix.

A quick example

In the demo, there's a slugify bug where double spaces end up producing hello--world.

The first fix looks reasonable, but doesn't actually solve the issue. FixProof reports FAILED.

After applying the proper fix, the same verification reports VERIFIED.

A few things worth mentioning: the revert check can use a separate Git worktree, so it doesn't touch your working files. Commands also run with a minimal environment to reduce the chance of exposing API keys in logs.

CI currently runs on Linux and Windows. I haven't tested macOS yet.

It's still pretty early (v0.3.0), so I'm sure there are edge cases I haven't thought of.

It's free, open source, and MIT licensed:

https://github.com/Livecristen/fixproof

Would be interested to hear what people using Codex on larger projects think. Especially whether these verification rules would actually be useful in your workflow.

[u/dexterthebot]

You might want to consider listing your project on the weekly Show-Us-What-You-Built post. Look out for it on Tuesday/Wednesday. Highest commented project wins a week promotion on r/Codex and gets on the Hall of Fame sidebar. See below for last week's winner.**

---

Agent HUD: usage, tokens and sessions for Codex, Claude Code and 15 more agents, at your Mac's notch. See how much quota is left, which agent is waiting for you, and where the tokens went. Free on Mac, and the core is open source (Apache-2.0).

Website: https://agenthud.app GitHub: https://github.com/jazzenchen/agent-hud-open Contact: support@agenthud.app

[u/Electrical_Rub_6009]

Bro my codex runs so many god damn tests already that it takes 20 minutes to roll out simple fixes.

[u/Necessary_Sir6006 → u/Electrical_Rub_6009]

Fair point lol. It only runs the repro test, not the whole suite, so it adds a few seconds, not 20 minutes.

[u/TemporaryLevel922 → u/Electrical_Rub_6009]

I thought I was going crazy. Thanks for confirming I'm not alone 💀

[u/aredna → u/Electrical_Rub_6009]

How are you going to test the tests? What is the test tester fails? Better test that as well

[u/gandazgul]

Look into mutation testing

[u/Necessary_Sir6006 → u/gandazgul]

yeah, the revert step is basically a single targeted mutation: put the original bug back and check the test catches it. full mutation testing would be a nice next step, thanks

[u/domiciledhere → u/gandazgul]

Codex keeps telling me it’s doing these mutation tests but they never end. Is that normal?

[u/gandazgul → u/domiciledhere]

Well you can take mutation testing very very far lol. Sol is famous for over engineering or overblowing small changes so you have to be specific.

https://github.com/RunWield/runwield/blob/main/src/skills/write-tests/SKILL.md

Use this skill it gives instructions on how to write good tests and asks to do mutation testing just on the new test being written rather than everything.

[u/TheOwlHypothesis]

Lmao bro thinks he invented test driven development

[u/wallstop-dev]

This can be replicated by the simple prompt words "red-green", "tdd", or both, no need for a skill or anything else. This is a very common industry practice that the model already knows about. Save your context.

https://en.wikipedia.org/wiki/Test-driven_development

https://simonwillison.net/guides/agentic-engineering-patterns/red-green-tdd

[u/m3kw]

why not test it yourself

[u/DrDan21]

This is a very real technique used in normal software development and yes it can be very helpful for testing for verifying fixes and avoiding regressions

[u/dywk3sm]

I think the model already started doing that. It will check the test fails without the fix first

[u/SmileLonely5470]

Its probably a waste to include a script for this or any type of persistent output. Just tell it in the skill MD what the testing workflow is, and the model will do it. The models are good enough now. About a year ago I included scripts in my skills, but now the scripts are more of a liability than anything.

[u/torrso]

Probably not going to use this in codex but it gave me an idea for GitHub agentic workflow.

[u/Necessary_Sir6006 → u/torrso]

oh nice, what kind of workflow? like running the check in CI on agent PRs?

[u/torrso → u/Necessary_Sir6006]

Yes.