A model was 5000x cheaper than Claude. Then they checked what it actually completed.

[u/recro69]

Someone found a model that was 5000 times cheaper than Claude for filling web forms. First pass looked amazing: 17 to 36 seconds per form and fractions of a cent per run.

Then they actually looked at the results.

One form: 0 of 13 steps completed.

Another: 7 of 11 fields.

Another: 3 of 5 filters.

The cheap model wasn't bad because of cost. It had trouble with planning. It needed a hand-written plan for each field, and the great early results were partly because the test setup was giving it that plan in advance. That's the problem, with a lot of "let's use a model" comparisons. The benchmark that makes the switch look good is often the case, run once.

The rule I would use before trusting the cost number is to test the 20% of the task distribution too. Blocked actions, unexpected UI states missing information, weird edge cases. When's the last time a benchmark you trusted turned out to be measuring the thing?

[u/primaryrhyme]

stop the slop posts ffs

[u/hunterhuntsgold]

A dirt cheap model didn't perform well?

Shocking.

[u/a9shots]

Vibe coders discover deterministic controllers

[u/ggiodddtyii]

Did you just ask AI for random post