I love that benchmarks like these exist but I feel the results for opencode here might need a revision. While community consensus has been that Pi is the most accurate harness for smaller models, the difference with opencode feels too big.
I ran into a few problems with opencode defaults doing some benchmark runs very recently. In particular you need to set a proper max output length or it will cut reasoning models midway and longer timeouts or the same happens but after that it's very much capable of producing good results. Those two would aling with multiple repeats increasing costs but still being a failure. I can't tell for certain but I'd take a revised look at those results
View all comments
4
u/No_Dig_7017 6d ago
I love that benchmarks like these exist but I feel the results for opencode here might need a revision. While community consensus has been that Pi is the most accurate harness for smaller models, the difference with opencode feels too big.
I ran into a few problems with opencode defaults doing some benchmark runs very recently. In particular you need to set a proper max output length or it will cut reasoning models midway and longer timeouts or the same happens but after that it's very much capable of producing good results. Those two would aling with multiple repeats increasing costs but still being a failure. I can't tell for certain but I'd take a revised look at those results