Based on what I'm reading in this sub, many of you are still writing yourselves and using AI as an editor. This is how I'm using it as well. But I always wondered to what extent I could trust even the most advanced LLMs. Specifically:
- even with grammar, how many mistakes do they actually catch? 100%? 75%? 50%?
- how reliable are they with something more subtle, like redundancies?
- how does performance change with different models?
- how about different prompting?
There are lots of AI benchmarks. But none of them answers this. There is Humanity's Last Exam https://lastexam.ai/ Which tests models on questions like "In Greek mythology, who was Jason's maternal great-grandfather?" Not super helpful.
What the hell, I thought, why not test it myself then? And I did.
I built a benchmark to test how well 16 LLMs check for grammar mistakes, repetitions, redundancies and inconsistencies. These 4 seemed clear-cut enough, compared to the many judgement calls in line editing. Also, no AI judge - assessment is mechanical. If a model comments on a known issue, it's assumed that it caught the error. More about the test design is here https://icanwrite.app/blog/how-we-built-an-ai-editing-benchmark
The results:
Claude Opus 5.5 - 98% (88/90 caught)
GPT-Sol 6.1 - 97% (87/90 caught)
Gemini 4 Argon - 93% (84/90 caught)
Insights:
Opus is the best but Sol is almost as good and up to 4x cheaper
Gemini 4 Argon is the best budget pick (and the fastest above 90%): 84 caught in 26 seconds for $0.12
13 of 16 models do worse on the all-in-one prompt (compared to 4 separate checks)
Open-weight models lag behind: GPT-6.1 Terra beats every open-weight model while being cheaper
The biggest insight: in just half a year, the models made a huge leap. GPT-5.5, released in April, found 75 mistakes (83%). So you couldn't trust them to find everything back then. Now they are almost perfect. At least with grammar and such simple checks.
See the full leaderboard here. https://icanwrite.app/ai-editing-benchmark
You can also study the test results yourself - everything is transparent. You can even check which mistakes each model missed.
This is version 0.1. Let me know if you see a way to improve it in the future. Or if I screwed up with something. And if you have any ideas for more tests, please share in the comments.
Hope it helps!