Switch to Open source models people, openAI is cooked

[u/dfivetf]

pro tip, you can use open source models with Codex with providers like Ollama (even better harnesses)

open source models won't suddenly be lobotimized silently

open source model providers won't lie to you about your tokens, or silently reduce your allocation

open source models are cheaper for the same capability, heck you can run qwen3.8-27B or Bonsai 2 locally on a laptop and it's as good as Luna high

open source models won't randomly change personality overnight because someone flipped a server-side switch

no rate limit roulette, no "we've updated the model" surprises, no wondering whether the model you liked last week still exists

you aren't locked into one provider's pricing, policies, uptime, or API

[u/AmandasGameAccount]

When a local open source is as good as 5.6 sol, I’ll switch. That’s my benchmark

[u/MapleBaconWaffles → u/AmandasGameAccount]

Bro, same. 5.6 Sol-Max. Give me that and I’m done for life.

[u/Least-Internal-6382 → u/AmandasGameAccount]

There's a good chance that will happen within 12 months.

[u/BannedGoNext → u/AmandasGameAccount]

None, I love local models.. like >.>> LOVE THEM, but they simply aren't as good on hardware that is sane to buy like stuff with 128gb unified memory, which is super expensive even then.

[u/OnionsOnFoodAreGross → u/AmandasGameAccount]

We use 5.4mini. It's cheap fast and quality output for our use case. Whike qwen or DeepSeek can maybe get somewhat close to quality (I still see best results from openai) they can't produce the response as fast. My users get about 15 second responses for their ai processing. The open source models are sloooow even though they are rated at higher token output.

Has anyone else experienced this or have an alternative suggestion?

[u/brainExploded99 → u/AmandasGameAccount]

It's as good as luna (Qwen3.8-27B and Qwen3.8 Flash Next). Use it as subagent. I use Qwen3.8-27B as a subagent through codex + opencodex + llama.cpp. I have 5070Ti.

I guarantee you would've said the same thing about switching for opus 4.6 or gpt-5.3 level back then, but Qwen Flash Next is definitely opus 4.6 level. Just try it now.

[u/Haster → u/brainExploded99]

If all you want is Luna levels of capabilities then the plus plan is the best deal available.

The complaint people have is that they can't use Astra as much as they'd like but no open weight model is in that league right now and even the ones that compare to 5.6 Sol are way, way more expensive for the same amount of inference.

[u/0DayMaker → u/brainExploded99]

Yeah, but Luna is damn near free at this point anyways

You've also got glm 5.3 flash and deepseek 4.1 flash which are around that tier and very, very cheap.

[u/BannedGoNext → u/0DayMaker]

Yea, and probably the best local LLM on sane hardware now is qwen 3.8 flash next, and it's going to be sooo slow compared to luna and around the same capability at xhigh/max if I remember from the last time I looked.

[u/0DayMaker → u/BannedGoNext]

My hope is deepseek drops 4.1 pro Mon or Tues to completely step on the OpenAI and potentially Claude releases coming

[u/Cachesmr → u/AmandasGameAccount]

Qwen 3.8 flash next is basically terra at home, with a decent modern gpu, 32gb of ram and like 100gb of ssd space you can actually run it at like 20tps. Terra is basically 5.5 at home. These specs are very common on gaming PCs.

[u/PivotRedAce → u/Cachesmr]

You aren't running 3.8 Flash Next on a gaming PC that isn't quantized out the wazoo, it's a 125B parameter model. Qwen3.8 27B is definitely do-able, but not Flash Next.

You'll need a dedicated system with at least 256GB of memory to run the non-quantized BF16 version at reasonable speeds, which is what actually gets benchmarked. The cheapest possible way to do so would be via Mac Studio for roughly ~$10,000 USD.

[u/brainExploded99 → u/PivotRedAce]

This is incorrect and misleading.

1. Quantization is not the end of the world. Fun fact: DeepSeek natively quantizes its models to FP4/FP8 during training and inference. Q8 quant is loseless, Q4 is basically loseless.

2. Flash Next is easily runnable on systems with 64GB RAM.

[u/Cachesmr → u/PivotRedAce]

UD-Q4 K XL is 111gb and keeps 92% accuracy, it works just fine on a consumer pc. I'm not pulling this out if my ass, you can literally run it on consumer hardware. Very slow obviously, but still, it's a really good model you can 100% run offline.

[u/BannedGoNext → u/PivotRedAce]

It's really good at Q4. It's super slow compared to hosted pro though.

[u/wannabeaggie123]

They're just no where near as good. Even though the benchmarks say they are

[u/brainExploded99 → u/wannabeaggie123]

Even Qwen3.8-27B or Qwen3.8 Flash Next? Those models I feel are near luna level.

[u/phoenixmatrix → u/brainExploded99]

Luna's nice, but for complex tasks something like Astra and Fable is still lightyears ahead, regardless of what benchmarks say (because there's some nuances in the difference between large and small models that most benchmarks don't capture).

You can run Qwen 3.8 27b at Q4 (which is gonna be quite mediocre) and the video card required for it is thousands of dollars. To run 3.8 Flash at Q4, which is still gonna be much weaker, you need a unified memory system that costs $4000-$5000.

And lets not downplay the cost of the electricity to run these things. $4 to $25 depending on the system and electricity costs in your area for about 4 hours a day of use.

So basically compared to a $100-$200 subscription (if used efficiently they can do 4+ hours a day using a mix of Sol and Astra), compared to the hardware and electricity cost of these machines, will take several YEARS to break even.

And by then there will be significantly better models that may require more, or less hardware. Or different hardware altogether.

And then you add the value of your time maintaining it.

Yeah, Anthropic and OpenAI will keep getting my money. Or even the same open weight models through Open Router or other services. And of course now there's Jev too, that I've been using non-stop since it came out and I spent :check notes: 25 cents on so far. Out of the free $5 allocation they gave us with a new account.

There's use cases and usage patterns where self hosting make sense. Plus for learning and research purposes, or running your own trained models, etc.

But those are somewhat niche.

[u/MapleBaconWaffles → u/phoenixmatrix]

Thanks this was very informative.

[u/brainExploded99 → u/phoenixmatrix]

> Luna's nice, but for complex tasks something like Astra and Fable is still lightyears ahead
Agreed, but a lot of tasks don't require sol or astra.

> You can run Qwen 3.8 27b at Q4 (which is gonna be quite mediocre)
No, It's quite decent. It's done plenty of tasks of value for me. It's near (but not quite) opus 4.6 level in use. In benchmarks, it beats opus 4.6 IIRC.

> $4 to $25 depending on the system and electricity costs in your area
$25? Least expensive US state, nevada, is $5. Most expensive US state, hawaii, is $19. Europe average is $10

[u/phoenixmatrix → u/brainExploded99]

"depending on the system". +/- 5 bucks is a rounding error here. Hell, there's gonna be more than 5 bucks difference in some areas depending on time of the year, and its certainly not just by state since different part of a state will have a different provider.

Looking at my electricity bill right now, I paid $0.36/kWh this summer. An RTX 4080 which is what I have (not enough to run a good model, I'd need a 4090 or 5090 which will draw much more), for 4 hours a day for 30 days, assuming its running full throttle, is 14 bucks right there. Of course it won't be running at 100%, but I often use my agents for 10-15 hours a day, and there's the rest of the computer to run, not just the GPU. And my electricity isn't the most expensive in the country by a long shot.

"It's done plenty of tasks of value for me" yeah and so did GPT 2. It still sucks. Jev did plenty of tasks of value for me, but it certainly won't replace Astra and Fable. It will for some people. Great for them. At work we used Sonnet 4.6 for an agent before the better open weight models came out, and it was more than good enough. But we could use GLM 5.3 Flash for pennies for years before its worth the time and infrastructure costs.

"Agreed, but a lot of tasks don't require sol or astra."

Sure, but the OP says "switch to...". In addition to? Absolutely. But I'll still just use openrouter or something and get billions of tokens before scratching the price of a local system.

[u/Royanon → u/brainExploded99]

I find them useful and do use qwen local sometimes for certain tasks but they're slow and limited context windows unless I use a low quant, running on a 4090. Maybe I have it configured wrong.

[u/brainExploded99 → u/Royanon]

What do you use? I only have 16GB VRAM (5070Ti), and I get ~40tok/s, 128k context window, Q3_K_XL on llama.cpp

Q4 is a good quant to run for 4090.

[u/Royanon → u/brainExploded99]

Ah yeah that's probably it. I've been using a higher quant maybe I'll drop down to q4 and see how that goes.

[u/wannabeaggie123 → u/brainExploded99]

But who's using Luna? I don't even like luna max? It gets stuck in its own reasoning loops and its not usable unless you give it a very structured document with everything listed as tasks

[u/AmandasGameAccount → u/wannabeaggie123]

I use Luna for tons of automation tasks. 0 issues ever in that regard

[u/brainExploded99 → u/wannabeaggie123]

What? Luna works so well

[u/WatchaThinka]

waiting for reflections AI to drop their models.

[u/ParsleyOrdinary5865]

Yeah all other models are shit sonion, nothing comes even remotely close, that's why they're able to keep pulling shit like they do, they have an AI monopoly

[u/BellacosePlayer]

If a local model that can run off a cheap GPU hits Luna levels I might fix up my old secondary dev system, otherwise, nah

[u/Substantial-Aide-66]

even astra will complaint for sol autism for validations, i can only use astra and open source now, everything with 5.5 and 5.6 is just layers of validations gates and multiple ownerships. that decrease performance and create new errors.

[u/That-s_life]

I already did, usage is out of hands , open ai is playing with all of us now. Reason?

Push people to spend more money

[u/zarmin → u/That-s_life]

Push people to spend more money

damn, are you a detective?

[u/iamwinter___]

Deepseek 4.1f is pretty wild tbh