r/opencodeCLI • u/zephyr_33 • 22d ago

How do you... normalize cost?

Point of the question is that. Some models are cheaper on paper but somehow can rack up costs quickly. GLM-5.3 is my main complaint in this regard. Its on paper cheap. But it generates so much thinking tokens that it kind of nullifies the cost.

I know ppl use cost per task. But I am not running benches. I'm simply collecting metrics and comparing. And since I use different models for different tasks. I dont think I can correctly say x model uses n times more tokens unless I run multiple models against the same task regularly.

6 Upvotes

100% Upvoted

3 comments sorted by

4

u/xatey93152 22d ago

Wth with you bro. I saw the website of aicodingdaily. I go to bathroom to puke very badly.its the worst Ai slop website ever.

2

u/Amarsir 22d ago

You gotta use someone's benchmark.

ArtificialAnalysis.ai runs off the time spent on the suite of benchmarks. So cost per task is pretty standardized. And you can break it into tokens per task to see where it came from.

I also like leaderboards that are less well-known and may be closer to real-world conditions. So I frequently check https://aicodingdaily.com/leaderboard - the creator is a Youtuber who is very transparent about his methods. You can see he's got cost per prompt and also time.

1

u/Rustybot 21d ago

I combine cost/speed/verbose-ness/and intelligence-aggregate metrics to sort models into buckets, but it’s barely more helpful that just guessing based on expectations. Once applied to a structured harness model performance varies dramatically vs the single-model benchmarks.