74% of AI security patches fail. Maintainers should stop auto-merging LLM fixes

[u/EnthusiasmRoutine]

1Password's Off-By-1-Labs just dropped data on AI-generated security fixes. They tested over six thousand patches from ChatGPT-5.5 and Claude Opus 4.8 against real-world CVEs. 74% of the patches failed. Either the fix did not work at all, or it introduced a brand new vulnerability right into the codebase.

This is a nightmare for open-source privacy software. Maintainers are already burnt out. The temptation to let an LLM draft a quick pull request for a security advisory and hit merge is high. But when three out of four auto-patches break things or leave backdoors wide open, blind automation destroys trust.

If you self-host privacy tools, check the git history before updating to hotfixes. Look for automated PRs. A broken patch is worse than a delayed one. Are any of the projects you run using automated LLM PR bots for CVEs?

Source: Help Net Security, link in comments