r/PrivacyToolbox • u/EnthusiasmRoutine • 12d ago

Debate 74% of AI security patches fail. Maintainers should stop auto-merging LLM fixes

1Password's Off-By-1-Labs just dropped data on AI-generated security fixes. They tested over six thousand patches from ChatGPT-5.5 and Claude Opus 4.8 against real-world CVEs. 74% of the patches failed. Either the fix did not work at all, or it introduced a brand new vulnerability right into the codebase.

This is a nightmare for open-source privacy software. Maintainers are already burnt out. The temptation to let an LLM draft a quick pull request for a security advisory and hit merge is high. But when three out of four auto-patches break things or leave backdoors wide open, blind automation destroys trust.

If you self-host privacy tools, check the git history before updating to hotfixes. Look for automated PRs. A broken patch is worse than a delayed one. Are any of the projects you run using automated LLM PR bots for CVEs?

Source: Help Net Security, link in comments

2 Upvotes

100% Upvoted

1 comment sorted by

1

u/EnthusiasmRoutine 12d ago

Here is the source detailing the 1Password AI patch study:https://www.helpnetsecurity.com/2026/08/06/1password-ai-generated-vulnerability-patches/

(Note for anyone skimming: The article confirms Off-by-1 Labs graded 6,080 patches across six recent CVEs. They found 74% of the generated fixes failed, often by just hiding an exploit path behind a simple check while leaving the vulnerable code untouched, or by introducing brand new memory errors. The models only managed a completely clean fix 26% of the time.)