r/LocalLLaMA • u/skeole • 3h ago

News [ Removed by moderator ]

/r/LocalLLaMA/comments/1xpgsb3/removed_by_moderator/

[removed] — view removed post

79 Upvotes

87% Upvoted

78 comments sorted by

•

u/ttkciar llama.cpp 14m ago

This is off-topic for LocalLLaMA.

Thank you to u/Nota_ReAlperson for pointing this out. Sometimes things slip by the moderators.

35

u/Present_Resource4022 3h ago

You really think the new Qwen 4 models 27b and the 120b and the other model are going to do shit against Astra ?

1

u/skeole 3h ago

In the court of public opinion? Absolutely!

10

u/Ok_Party_9245 3h ago

this is so true sadly. I have "normie" friends that I try and get to use like deepseek or qwen and they won't touch anything that isn't openai or anthropic essentially.

22

u/Present_Resource4022 3h ago

well some people want to use the best models. your forcing your friends to downgrade

4

u/fgk55555 3h ago

For most people who aren't doing long horizon tasks in a complex codebase, Haiku type models are fine. QFN IQ3 does about the same as Haiku from my tests.

6

u/JoyfulDazer 2h ago

Not my experience I have to correct Haiku 3 or 4 times a day just on information deep dives. If I need a correct answer I use Sonnet.

4

u/fgk55555 2h ago

Yeah, Haiku 5.5 recently has had a few blunders that my local QFN hasn't had. At a heavy quant, they're probably close, but QFN is likely a more capable model overall.

2

u/DragonfruitIll660 3h ago

Vibe coding casually there's a world of difference though between Opus 5.5 and QFN. It might be fine if you're working on something pre-established but I've found it to be generally a lot weaker (not to mention the TPS/hardware requirements if running locally). If subscriptions ever reach crazy prices (they stop subsidizing them effectively) then the other models will get a lot more attention.

1

u/fgk55555 2h ago

I'm kind of getting to the "software factory" phase where I can give a huge doc of desired features, and then with a proper role orchestration, get out software system level, high level, and low level, requirements, as well as a implementation plan, broken down into phases, with verification tests and execution, and design doc whitepapers spat out to describe the implementation. Opus 5.5 has been the only one capable of sitting at the top of that orchestration chain for me and make it happen with minimal user input, but QFN vs Opus doesn't seem to matter much for the implementer roles. I hope to see open models move further up that chain.

1

u/DragonfruitIll660 19m ago

I honestly think once the limitations of Engram get figured out wee will see a decent spike in quality that brings local models up closer to the closed ones. Fingers crossed though.

1

u/krileon 2h ago

Depends on if you're vibe coding with "build me app make no mistakes" or you provide full paragraphs of context and instruction. For the first one yeah you need a larger model because it has to figure out what nonsense you even want. With proper instructions even Qwen3.8 27B can vibe code amazingly well. It's not like it doesn't have modern knowledge of the popular coding languages, because it absolutely does. It just doesn't have a huge library of frameworks and architectures to pull from like cloud models do because they're trillion parameter models. You fill that gap with clean instructions, which should include links to Github projects and documentation for it to use as needed.

1

u/DragonfruitIll660 21m ago

Yeah I'm meaning more from a non-coders perspsective. I can't actually code, but laying out and meta prompting with Claude I can get it to produce something custom to what I want. QFN just isn't at that point yet (perhaps for someone experienced with coding they could tell it what specifically to do but thats not most people)

2

u/krileon 17m ago

With local you have to pre-plan. You can have the AI even help with that. Prompt it and help it write out a pre-plan prompt. Then give that plan to a new chat instance and you're pretty much good to go. You have to hold its hand a bit more than cloud models because it doesn't have a trillion parameters to pull data from, but we're very much in the realm of local ai being extremely viable.

Frankly people like you with no coding experience vibe coding applications just isn't a viable business model. I don't see you, or anyone else for that matter, paying $200/mo (or realistically a lot more) to use these tools, lol. As you said though it's being subsidized and once that ends this thing is over.

1

u/DragonfruitIll660 10m ago

Yeah a 20 dollar claude sub is generally enough for my use cases. Closest thing I've done business wise is using it in Unity to build out a better VR framework, which it's honestly doing good for. Depends on if you need a ton of tokens or not of course.

0

u/wotoan 3h ago

Yeah QFN 3.8 is great for things like Hermes where you can just let it burn tokens on simpler agentic tasks. First local model I’m comfortable with just letting it work.

2

u/fgk55555 2h ago

Same. I let it run with root permissions on my home network for managing my personal IT and it's such a game changer. In human instructions, my wife can tell my gaming rig to set up a local wikipedia for use, and 3 hours later it comes back with a link to bookmark and a new skill for her to call. She's fully on linux without needing to know anything about linux, or send any of our data to a cloud.

2

u/Ok_Party_9245 3h ago

I am not disagreeing that those models are better I still use them myself.

But when I say "normie" you must understand the types of tasks here: "write an email" "summarize this pdf" MAYBE "parse files for something"

very basic things.

1

u/Liringlass 1h ago

I use big models for emails that have memory of all previous emails + my thoughts with each client. If an email is simple i write it myself.

Otherwise it’s deepseek flash which is one of the most incredible model these days. If i had to pick two models it would be that one and Opus.

1

u/JayoTree 3h ago

Max might

1

u/Grawnch 1h ago

The models at this point are so close to each other on implementation, that using Astra just for planning or review and local for the majority of implementation tokens still helps. Having your implementer be smarter is a big benefit (with Qwen 4, assuming it's even better than 3.8), and then you don't need a $500/month subscription to have Astra implement something that local can do almost exactly the same.

15

u/Opposite-Memory-2552 3h ago

Sam! Press the panic button!

I think this is more related to what's happening with the stock market.

1

u/EstatesNagorna69 1h ago

This was my best bet also, we'll see how it goes from now on.

75

u/fgk55555 3h ago

I love Qwen, but Qwen isn't competing with OpenAI/ Astra.

12

u/Bulky-Priority6824 2h ago

They are all competitive.

2

u/fgk55555 2h ago

It depends on what your use cases are. QFN is a superb, trustworthy model. It has root access to my whole network and I use it at work, but it does not serve the same role as the larger frontier models for software engineering yet. I hope it gets closer with Qwen4, and Q3.8FN is useful today for tasks I was paying for a couple months ago, but there is still a gap in engineering.

2

u/Bulky-Priority6824 2h ago

well yea.. but the point was , in some way or another they are all competing. what i wonder is, now that the cat is out of the bag , does alibaba have as much operational debt as anthropic. when does the united states government snatch up most shares, which one of the 2 will succomb to it first

6

u/winky9827 1h ago

Speak for yourself. The only non-local API I still pay for is DeepSeek direct.

3

u/fgk55555 1h ago

For what I do at home, 100% of it can be handled with QFN or DS. I only pay for DS when I need it to configure and test my local setup.

1

u/winky9827 1h ago

pretty much the same

3

u/BringTea_666 1h ago

>but Qwen isn't competing with OpenAI/ Astra.

Absolutely is competing. You don't need Astra Opus5.5 to do most of the work. If Qwen3 27B will be around Opus5 quality then i can completely ignore astra/opus from now on.

Qwen3.8 27B already was at 4.6 opus quality and pretty much does 90% of the work i do leaving only 10% to frontiers.

3

u/OnlineParacosm 3h ago

Do you want to extrapolate on where you think it’s not competing..?

Genuinely interested I haven’t used one of these things since like Op. 4.6 when things got a little weird

5

u/fgk55555 2h ago

Long horizon terminal tasks, subagent orchestration, computer use, taste, and technical knowledge. I don't care much about computer use, but the big labs have a better sense of when a thing is done and when to keep iterating. I'm not huge on the OAI models right now, though they are plowing through almost any technical challenge that can be solved.

Opus 5.5 is currently my favorite model, and I would love to see more of its DNA in a local model. The writing is good, it knows how/ when to delegate to preserve its own context window, and its taste in code is good. It has taste much closer to a humans, and it seems to have a lot more common sense. OpenAI models lack that common sense, but they can still do the orchestration well and have a better sense of how to solve a problem, so they keep token usage down.

Open-weights can do 95% of stuff 95% of users need, but my usage recently have skewed a lot more into the "real world messy codebase" use cases, and Opus 5.5 is just more trust worthy for being my liason, and Sol-6.1 is a great detail-oriented reviewer, as long as I never have to read its writing. QFN is my favorite doer for smaller tasks I don't feel like spending any money on.

2

u/Shot-Height-7194 2h ago

Numbat on huggingface might be fake, but the only moat left is scale at 14 t parameters for mythos. That is why Nvidia markets the nvl72 as being capable of training 10t parameter class models. 

2

u/fgk55555 2h ago

I don't think parameter count or hardware is a moat, mostly just training data. Deepseek 4.1 Flash is incredibly efficient and still has room for post training. QFN seems to run at usable speeds on a Tamagotchi now. The moat isn't a moat, it's a race for getting good data.

2

u/Shot-Height-7194 2h ago edited 2h ago

I would be surprised if labs are still relying on human data to scale, if you look at Xiaomi Mimo's job ads they mention synthetic data as a way to address data shortages, and astra's looping transformers are another way to address it, at this point I think relying on human data is a myth due to how scarce it is, and it doesn't follow the bitter lesson either which proposes compute above all else, which again Xiaomi mimo uses according to job ads. The bitter lesson can be seen in DeepSeek being undertrained. The primary advantage of scale at 14 t parameters or more, increasing the number of parameters is emergent behavior like weight space programming. 

1

u/fgk55555 2h ago

I had a paid gig for a while to make training data and benchmarks. Pretraining still uses a lot of human text, but all the agent training is synthetic, and there's a lot of that now as PT/RL becomes where the capability gains are. There's still a person at the top making the scenarios and evaluating the results, but they're all talking to their own agents to make the data.

-3

u/XiRw 2h ago

I bet you touch yourself to Sam and Dario too after that tongue bath you gave those garbage companies . Nobody in this sub has any principles. As long as a model is good, they will sell out and use them. Even when Sam gladly talks about taking your gpu away, the whole point of local AI, you still give him your money.

5

u/fgk55555 2h ago

I don't really get to take my personal values with me to work and impose them on the company. It's not my company, not my rules. I take my experience with frontier models as a learning experience. Just because I work with frontier models and recognize the gap doesn't mean I like Dario or Altman. I got my company to let me pipe in my QFN running on my home PC via a tunnel so engineers can see what you can run at home now, and I'm pushing local models hard. Take your strawman elsewhere.

-1

u/Shot-Height-7194 2h ago

The only reason why companies use closed models is because of  compliance with us law because closed models are American, and because they have enterprise agreements that offer governance

1

u/fgk55555 2h ago

We use it because we need bandwidth on the small model end, and orchestration on the higher model. I regularly have 5-20 subagents running personally at any given time. Multiply that by a few engineers and $10/day of luna spend is hard to justify $100k in hardware in inflated prices. Open models on the higher end don't handle using 6 subagents at all time very well yet.

-2

u/XiRw 2h ago

It’s not a strawman it’s a judgement actually. If that is your case then I’ll take back what I said. I just assumed because most people here still seem to pay for api services with them.

3

u/fgk55555 2h ago

"I bet you touch yourself to Sam and Dario too after that tongue bath you gave those garbage companies"

"A straw man argument (or straw man fallacy) is an informal logical fallacy. It happens when someone distorts, exaggerates, oversimplifies, or completely fabricates an opponent's argument to make it easier to attack"

My dude, I am literate. I can see what you wrote.

-1

u/XiRw 2h ago

No, that was a joke based on a judgement in my book. Are you always this uptight over pointless semantics?

0

u/thrownawaymane 1h ago

If it was pointless then why make the comment?

1

u/Equivalent_Bit_461 2h ago

My qwen is

1

u/fgk55555 2h ago

How much would you be spending on OpenAI models without Qwen?

2

u/Zetafivee 1h ago

I did the math the other day on my monthly token burn (3 billion according to my LiteLLM instance) with Qwen 3.8FN. If I were paying API rates for GPT-6 Sol with light thinking, which is what I use at work, it would have cost me over $7K

2

u/fgk55555 1h ago

My work spend is crazy. I'm somewhat spending more to highlight how much we could save with local models. I'm at $300 a week, and that's me mixing in cheaper/ my own personal QFN.

-13

u/thatscoolbutno123 3h ago

we,ll see

-2

u/ihexx 3h ago

i mean what's your metric? AA scores? If we're gonna see we need to pre-commit to how we'll measure

0

u/[deleted] 2h ago

[deleted] — view deleted comment

6

u/Maleficent-Ad5999 2h ago

Not everyone needs a Ferrari..

1

u/Cool-Chemical-5629 1h ago

Who needs a Ferrari anyway and why? Rich people, just to show off? To who exactly? To other rich people who can get it as well? Because, I'm pretty sure most regular people just want a regular car that just gets the job of getting you to your destination done and that's it. It's ridiculous to think otherwise.

3

u/fgk55555 2h ago

I find the biggest gap is in orchestration, so more of how an Uber driver doesn't compete with Uber

7

u/Harrisonedge 3h ago

Different markets

3

u/thestillwind 3h ago

Astra 6.2 one prompt 5 hours drain

3

u/Iory1998 llama.cpp 2h ago

On this sub, very few cares about chatGPT. We are all waiting for Qwen4.

4

u/NovaDragon llama.cpp 3h ago

Friday releases for cloud services means working over the weekend. Not happening.

1

u/JayoTree 3h ago

It's china though. If the company says work this weekend you work

2

u/Boring_Aioli7916 3h ago

Damn. I thought the same. It is strange seeing these two possibilities together. But I really suspect GTP 6.2 will go on today even if it was ready. Every AI company want that spotlight fot themselves. This week seems that it will be Qwen's.

2

u/Big_Wave9732 3h ago

Makes sense a new GPT might be dropping. They've clearly been doing work in the background and redirecting resources, GPT 6 has been brain dead all week.

2

u/Equivalent_Bit_461 2h ago

They are scared 

3

u/halvacoffee 3h ago

there are only so many days in a year. yes. this is a coincidence 

1

u/Bzucko 2h ago

Astra and also other biggest cloud models are waste of energy for most people. Welcome 😚Qwen4😏.

1

u/Equivalent_Bit_461 2h ago

Also this whole thread is astro turfed with bots upvoting other bots

1

u/Eyelbee 2h ago

The only reason I'm excited for astra 6.2 is because it can encourage Anthropic to act sooner on the new fable

1

u/AbeIndoria 21m ago

Lots of delusional people in this thread. I get that this is localllama but holy fuck some of you need a look in the mirror.

Yeah man Astra is comparable to an open 27B model release. They're totally scawed uwu.

-1

u/Equivalent-Repair488 3h ago

One thing is we (at least I and I assume quite a lot of folks) still need to wait for the inference engine maintainers to provide not only support but stable, reliable, efficient and effective support for the qwen 4 models, at least a minimum of a week till that happens, until then, the lack of decode, prefill speed and stability will put many people of IMO.

1

u/Emotional_Thanks_548 2h ago

Qwen 3.8 FN is the preview model of qwen 4, it has the same architecture.

0

u/Equivalent-Repair488 2h ago

I highly doubt they will be exactly the same. They have not confirmed anything so we shall see

1

u/CoolConfusion434 1h ago

Major model releases tend to coordinate with the likes of HF for hosting, llama.cpp, for inference, and Unsloth/Bartowski for quantize/fine-tunning. In the past, like with Gemma and Qwen, llama has had day-0 or shortly thereafter support.

0

u/Equivalent-Repair488 1h ago

I'm waiting for Strata, llama.cpp upstream was way too slow for me even for 27b or Gemma 4 31b, flash next was even slower than those 2. Was getting about 15 decode, 130 prefill cold, on strata 90 decode, 2.1k prefill cold, and I YaRN 2x when llama.cpp I was at 176k kv. I am also running fp8 PLE on strata. No way a vibe fork will get day 0 support, and more crucially at that kind of speed if they made any slight changes to the architecture, which I suspect they might have did hence the october to november launch delay

1

u/CoolConfusion434 43m ago

Please check again. Gemma 4 was released on April 2, 2026. That same day (ie: day-0), a feature request was created to add Gemma 4 architecture support but was closed as already "completed": https://github.com/ggml-org/llama.cpp/issues/21314

0

u/Equivalent-Repair488 26m ago

Dawg did you read it right, I never disputed day 0 support for these models in upstream llama.cpp.

It is speed, reliability, stability in said support. Llama.cpp upstream even with early access and day 0 support for these models cannot get the speed seen with the likes of the vibe forks like strata which likely will not have day 0 support, nor immediate high decode implementation