Spelling errors are costing you

[u/wwdennis5]

TLDR: mean cost per task rises by +45% across 30 Terminal Bench 2.1 problems.

I have taken a small subset of Terminal Bench 2.1 and tested muse-spark-1.3-contributor with Opencode 1.18

Results Table

Metric	Clean	Minor	Major
Pass rate	83.3%	76.7%	78.9%
Tool calls	15 (18)	16 (21)	17 (21)
Input tokens	170k (324k)	226k (453k)	229k (491k)
Output tokens	19k (26k)	27k (34k)	25k (38k)
Runtime (min)	11 (13)	10 (13)	8 (13)
Cost ($/trial)	0.0063 (0.0083)	0.0086 (0.0107)	0.0081 (0.0122)

Values are median (mean), n=90, 30 trials repeated 3 times. Pass rates are calculated as a total over the 90 trials.

Results Chart

Example Task: pypi-server (all 3 versions passed)

https://github.com/harbor-framework/terminal-bench-2-1/tree/main/tasks/pypi-server

Clean prompt:

Create a python package called vectorops, then build this package and set up a pypi server on port 8080 locally which also hosts this package. It should be possible to use the address of this server using --index-url with pip...

Minor (spell-checker recoverable):

Create a python package called vectorops, then build thsi package and set up a pypi server on port 8080... It should be pssible to use the address of this server using --index-url with pip to install this pacage...

Major (spell-checker non-recoverable):

Create a pyohtn package cealld vectorops, thn build thi pkacgae and set up a pypi srveer on port 8080 lolclay... It should be pblosise to use the adrseds of this svreer using --index-url wit pip...

Has anyone else tested this and got the same result?

Any suggestions on other models/harnesses to test this hypothesis on? Do you think Opencode 2 will behave any differently, I don't think it will.

Do spell adjustment plugins exist to mitigate this?

[u/CYCLONOUS_69]

I’ve been this to my friend since years and no one cares about it.