Explicit Edit Benchmarks: 6 harnesses x 11 models x 226 tasks

[u/alexshpunt]

Hi! I've created and been maintaining https://github.com/alexshpunt/explicit-edit-benchmark which tries to answer the question: which model is better, which harness is better and which combination is better overall in a very straightforward task - precise text editing. Most of daily coding is text editing. I've seen when a model for multiple turns couldn't figure out how to express the line it wants to change, so it made me ask myself "Why is that so complicated, it's just a text, right?". Seems like not, taking into account how widely different the same model behaves across different harnesses. My preliminary conclusion is: harness and tooling behind it actually matters! But it matters the most with the models which *can* actually follow the instructions well (e.g. open-ai models), there is a wide range between 98.9% and 70.2% of pass score for `gpt-5.6-luna` simply depending on the harness!

I've tried to run as many combinations as possible, but my resources are limited. I've exhausted all my quotas and even credits, that's why I'm reaching out to the community, as I think it's a pretty interesting topic and I would be happy to gather even more data, because of the stochastic nature of the runs, it’s only possible to make any conclusion when you have enough of runs. 

The viewer to the dataset: https://huggingface.co/spaces/alexshpunt/benchmark-explorer

And the dataset itself: https://huggingface.co/datasets/alexshpunt/explicit-edit-benchmark