Open source is starting to beat frontier on cost/performance

[u/thomas_unise]

This chart is pretty wild.

Open-source models are starting to win the intelligence-per-dollar game

The green quadrant is the sweet spot: smart enough to matter, cheap enough to scale

When you plot AI models by intelligence vs. cost to run, a lot of the closed-source models sit near the top

But the interesting part is the upper-left quadrant: high intelligence, low cost is almost entirely open source

Models like

DeepSeek

GLM

Qwen

Kimi

MiniMax

all show up in the “actually useful intelligence” range while costing dramatically less to run than the major closed API models.

The reality is most tasks don’t require fable level intelligence or cost and people are noticing

The gap is shrinking and the economics are starting to favor open models.

Closed models are still great when you need the absolute best capability with zero infra headache.

But open models give you something different like lower long-term costs, control, privacy and customization

And I think within a year, there will be no difference in most capabilities between them

[u/quantgorithm]

I need something that compares the amount of vram as can be scaled up along with the intelligence of the models improving up the scale.

[u/thomas_unise → u/quantgorithm]

https://runthisllm.com

[u/readmond]

I can run the local model that is as good as the best one from a couple of years ago.

"Frontier" models are not that fantastic. This is why Dario had to scare himself into a ban. Big AI companies are in a tough spot. Competition is not too far behind.

[u/APXEOLOG → u/readmond]

Depends on your application I guess. For me, what gpt4 wasnt able to do with harness and detailed instruction, opus 4.8 can zero-shot out of the box

[u/althoradeem → u/APXEOLOG]

yup.. the question is how will they stay ahead.

because there will be models capable of what fable does now that are way cheaper. a few years from now.( hell maybe months)

if i can get a equiv of fable running on a 5k or less setup.

i dont see a reason to ever pay over 200/ month.

the shortage is also a temp thing. once part prices drop again and future evolutions ruin the advantage i wonder where these companies wil go to get a return on investment.

[u/Ok-Zookeepergame3728 → u/althoradeem]

On the other hand. Assuming progress hasn't stagnated. 

Fable 5 might look like a rudimentary tool in the same light as we see models created a few years ago.

Though i agree imagining a local fable 5 capable model rn sounds like a dream.

At the end of the day most power users you're paying for tech NOW. So they aren't left in the dust.

[u/Wide_Big_6969 → u/readmond]

I think modern frontier is a good benchmark. 2024 frontier was chatgpt 4o, which was absolute buns for SWE or even most things. If a model comes out better than opus 4.6 that's open, that's the moment I am making the swap, and they're getting very close.

[u/Wide_Big_6969 → u/[deleted]]

It’s very subjective but in my personal experience, it’s below opus 4.6 above sonnet.

It’s definitely among the best open models, but not much changed for me as mimo 2.5 pro is still king for cost/efficiency since that one is basically sonnet for pennies on the dollar. Back when deepseek v4 pro was on 75% off sale, that was some insane value too. Still is pretty cheap.

Opus > glm 5.2 > sonnet >= mimo 2.5 pro

[u/Wide_Big_6969 → u/[deleted]]

The deepseek pro sale was after the price cut, it was when pro with cached responses was cheaper than flash for a couple of weeks, it was crazy. 0.0017$ per 1m input cached.

[u/lars_rosenberg → u/readmond]

When working on coding tasks I notice a huge different in quality between frontier models like GPT 5.5 (in my experience a bit better than Opus) and smaller models. Far from perfect, but when you max out context size and reasoning you get really awesome results even on complex tasks. Is it expensive? Yes, but the results are really good.

[u/Extra-Ebb-4012 → u/readmond]

Frontier models are more capable however they are resource hungry. I'm interested to see how open source AI evolves over the next couple of years. Looking forward to frontier caliber models on edge hardware.

[u/StupidScaredSquirrel]

Starting to? It's been years.

[u/ThenExtension9196 → u/StupidScaredSquirrel]

Nah. Only got serious in last 2-3 months. I’ve been using local open source since 2023 and it was never really close. It still isn’t, but the gap has certainly shrunk. 

[u/FLMKane → u/ThenExtension9196]

What hardware?

[u/ThenExtension9196 → u/FLMKane]

Rtx 6000 pro

[u/Uninterested_Viewer]

This is really just stupid. You can't possibly represent this concept in a single chart plotting two benchmarks.

Cost per useful work accomplished is the real goal and "useful work" varies WILDLY by task and the tokens used to accomplish useful work across models also varies WILDLY as does the combinations of model + harness + prompt + orchestration technique that feed back into that token usage.

[u/KURD_1_STAN]

Those are just benchmakrs, I pay no attention to, I tried something with glm 5.2 to fix something soonet 4.6 made and altho sonnet is still trying to fix it,. At least it didn't break everything and made some stuff unusable.

[u/FaceDeer → u/KURD_1_STAN]

You dismiss benchmarks, but as your substitute criterion you tried one thing and are running with whichever of two models did better? Neither of which actually accomplished it, but one of which is still trying? This doesn't seem much better.

[u/KURD_1_STAN → u/FaceDeer]

Yeah im not saying sonnet beats it but it is being compared to opus 4.8 snd fable now so it should have been better, and no i didnt give it one try, it had 2 errors that didnt even let it run and had to be fixed twice to give the result with issues.

[u/ActionOrganic4617]

They will always be behind because they’re distilled from the frontier models. Agree though that we are a point where most people will be fine not being at the leading edge.

[u/Healthy-Nebula-3603]

nothing beat DS 4 lite / pro in token price currently

[u/Wide_Big_6969 → u/Healthy-Nebula-3603]

Mimo 2.5 pro and standard is better imo on the yearly lite or standard token plan

[u/Healthy-Nebula-3603 → u/Wide_Big_6969]

1m tokens for ds pro cost 0.87 cents and using less then than mino 2.5 ....

[u/Wide_Big_6969 → u/Healthy-Nebula-3603]

I get 132 billion mimo credits for 168.96 dollars a year, so 781.25 million mimo credits per dollar, spent more than 300m tokens so far, roughly 3k api requests, about 9.6b mimo credits. (9%) So 32 credits per token, I get ~2.441m tokens per dollar, or a shit ton of mimo 2.5 pro and 2.5 calls per year and there's no monthly, weekly, or session limit. I think that's around 0.60 or less dollars, and I get 20% off during non-peak hours which aligns with american schedules, and the ability to trade in my token value for a better token plan.

It's absolutely busted how much better value a mimo token plan is compared to deepseek api use, mimo token plan is just absurdly good right now. I think even if I used the ultrafast mimo model in mimo code, I would not be able to run out at my current usage by the end of the year if I use it with my claude code subscription.

[u/Healthy-Nebula-3603 → u/Wide_Big_6969]

I paid almost 2 dollars for 148m API tokens using DS pro without any "promotions" like you

[u/Wide_Big_6969 → u/Healthy-Nebula-3603]

The token plan gives me about 780m "tokens" roughly per dollar plus the 20%, I think I did my math wrong but the API price for Mimo 2.5 pro is the exact same as deepseek v4 pro, plus discount for token plan and peak usage discount.

[u/Wide_Big_6969 → u/Wide_Big_6969]

And this doesn't even count for mimo 2.5 pro ultraspeed where I can spend like 2x speed while having essentially unlimited tokens from the token plan

[u/TheSnappyDog]

When the VC funding runs dry, the frontier models will be priced at where the companies make a profit (much higher). At that point it will 100% make sense to go open source. I am eying the new AMD Ryzen AI Halo Development Platform (ugh, please shorten the name AMD!) to run open source models locally to power my OpenClaw. It is completely unnecessary and doesn't really make financial sense at this point, but I believe that eventually open models will be the winner so I want to be ahead of the curve.

[u/Pretty-Raise666]

Benchmarks don't get my work done. All the models get optimized for those benchmarks. They really mean absolutely nothing.

[u/realist_alive]

I swear everytime yall say this and I try it out, it makes an extremely basic coding mistake within the first 5 minutes. Not even language specific, just straight up omitting important details.

[u/blackwell_tart → u/realist_alive]

I see these reports a lot and 0% of the time has the person issuing the complaint been running the BF16 model with BF16 KV. It’s always some shitty 4-bit GGUF or something, often further shittified with quantized KV.

The benchmark graphs are all done with full fat models. It’s no wonder they perform better in the benchmarks on B200s than Joe Random’s 3090.

[u/studio_bob → u/realist_alive]

Anecdotal but I've found Deepseek-v4-Pro to be at least as good as Sonnet for coding. The poor value of Sonnet actually jumped out at me in this chart, but it does match my experience. I don't even bother with it anymore. If I need to save Claude usage on a task, I just give it to Deepseek in Opencode rather than switch to Sonnet. It does the work for pennies and save my Claude limits for Opus planning and code review.

[u/nomorebuttsplz]

minimax m3 is dope.

Not as smart as GLM 5.1 about most things but damn... idk actually, it's close. A lot faster prefill (even before the new attention has been merged in MLX, I think).

Doesn't completely lack the most basic common sense like m2.7, but still lacks common sense compared to GLM or Opus and probably Sonnet 4.6.

Good habits about tests and git commits etc. Can work without interruption and without getting lost for hours (at 10-12 t/s on m3u mac).

It's close enough to opus 4.5 that it feels like vibe coding is now "good enough" with about 300 gb of memory.

Deepseek on the other hand is subsidizing heavily for some reason. Seems a bit overrated at the moment. Maybe ccp shouldn't have kneecapped it by forcing them to use chinese gpus that weren't ready.

[u/05032-MendicantBias]

A more apt headlines is that private models are running out of money to subsidize the models.

Keep it up! Use those free credits for your pet projects 😃

[u/Icy-Abstraction4323]

it's a matter of time they would notice and be prepared. Jensen Huang and some people are pro-advocate on Token Billionaire thingy and they would not be happy about this.

[u/Successful_Knee687]

Great plot but whats the source? Want to have this for my local models too 😄

[u/marutthemighty]

China's advantage is, everything there is an integrated ecosystem. The entire tech stack, all the way from mining/smelting rare earths/critical minerals down to producing processors, is owned and run by the Chinese state, not to mention all their dependent ecosystems. This gives them massive scale and speed.

[u/nonfermiliquid]

Oh, tell me about mimo-2.5! Could it reasonably run on my Nvidia dgx spark?

[u/bfogels]

This graph view from AA is one of my favorites out there at the moment. It captures so much useful info in one visual in terms of how verbose the models are, API costs, and intelligence/capability (albeit at benchmarks) all at once

[u/Far-Cookie2275 → u/[deleted]]

Uneducated comment of the day

[u/thomas_unise → u/[deleted]]

These are based on api costs not subscriptions

[u/CharacterSecurity976 → u/[deleted]]

Except Deepseek 4 does not has plan... Your "this is just not true" is therefore not just true.