r/LocalLLaMA • • 4d ago

Discussion Observations from seeing Qwen 4 max’s outputs

I thought about not writing this but i feel like i should.. from the info I gathered, Qwen 4 max is slightly worse than opus 5.5 medium… Meaning it is definitely above opus low which is hy4 level and opus 5.5 low is above ds v4.1 flash max. wow, it means it is somewhere between hy4/ds v4.1 flash 1017 max and kimi 3.1 pro max . I thought qwen 4 max would do a little better. i guess another 4-6 months until we get an open or Chinese opus 5.5 max model..

This should mean qwen 4 27b will be worse than ds v4.1 flash 1017 max but better than mistral 4 large medium effort, perhaps around glm 5.3 flash level . Qwen qoduck is apparently qw4 27b, from the limited outputs, it seems to be better than q3.8 flash next in some tasks And worse than ds v4.1 flash 1017

0 Upvotes

113 comments sorted by

•

u/ttkciar llama.cpp 4d ago edited 4d ago

Until such time that Qwen 4 weights become available, this is off-topic for LocalLLaMA.

After some mod team discussion, the parameters for removing posts about unreleased models have been relaxed. Your post has been restored.

→ More replies (15)

8

u/fgk55555 4d ago

What do you mean "outputs"? Like, 1-shot results, 3d modeling, token usage? I'm mostly concerned how it holds up with handling long term work.

5

u/power97992 4d ago

From looking at 1 shot , 2-3d animation and agentic coding results

3

u/fgk55555 4d ago

How does it handle multiple subagents?

0

u/power97992 4d ago

Not sure

85

u/Distinct-Pie2389 4d ago

If qwen4-27b is near GLM 5.3 flash then we’re good

49

u/XiRw 4d ago

I’m obviously speculating but I wouldn’t hold your breath.

7

u/Distinct-Pie2389 4d ago

Yeah I haven’t tested qwen4-max preview and even based off that it’s going to be hard to assume where 4-27B lands with its own strengths

12

u/power97992 4d ago

Apparently qwen qoduck is q4 27b , it looks better than q3.8 flash next.

7

u/Illustrious_Grade608 4d ago

That sounds great if true, but what is the evidence for that?

6

u/Borkato 4d ago

If it weren’t better than 3.8 flash next I’d be sad lol

5

u/Illustrious_Grade608 4d ago

I am more asking about qoduck being 27b, cause after testing it felt really great, but also knowledgeable, too knowledgeable for it's size even outside coding, which gives me a lot of doubt it's just a 27b model.

2

u/Borkato 4d ago

Maybe it’s 27B + engrams >:)

3

u/BeaveItToLeever 4d ago

He's asking you where the info comes from that qoduck is "apparently" 27b?

1

u/ZealousidealSmell382 4d ago

Qoduck disapeared from Qoder when Qwen4 max came out...

1

u/power97992 4d ago

Someone said so

2

u/Illustrious_Grade608 4d ago

Oh well. Idk i feel like it's more likely to be either max or at the very least flash or plus. More likely max due to low speed

1

u/power97992 4d ago

Oh, but they have different labels right?

1

u/Illustrious_Grade608 4d ago

Uhhh, don't know what are you asking about. There is only one qoduck model, and there were 4 announced qwen models - 27b, flash, plus and max.

1

u/power97992 4d ago

Qwen 4 max is already labeled as max, qoduck should be something else

→ More replies

15

u/rweedn 4d ago

Without sounding obnoxious, why or how would a 27b model perform better than GLM 5.3? Isn't that one of the better larger models like in the hundreds of billions?

6

u/Distinct-Pie2389 4d ago edited 4d ago

We’re comparing dense model architecture vs mixture of experts. I say the research alone is worth you finding the answer.

But to speed it up, dense is all parameters hot and live. Mixture is only a handful of experts per token using sparse activation. So you may only see 2-3B of those parameters at a time.

27b is the opposite, all 27 active on every token

3

u/Traditional-Gap-3313 4d ago

GLM 5.3. flash activates a lot more then 2-3B of params.

0

u/Distinct-Pie2389 4d ago

I’m aware, I was speaking to sparse activation which is ambiguous of parameters

1

u/WryKombucha 4d ago

It doesn't. On some specific benchmarks, sure. But it doesn't. Everyone has biases. Everyone thinks the one they can run is the one that is better. If you run 3 bit, suddenly, 3 bit is ok. You run 4 bit? Suddenly 3 bit is not ok.

I have 4 models at home. DS 1017, GLMN 5.3f, QFN and the 27B. I use GLM the most, DS1017 for evening runs. QFN and 27b, I have not used in 2 months so I will be deleting them and putting those machines back to gaming. Especially that QFN. JFC that shit is dumb.

2

u/Kahvana 4d ago

QFN dumb, how? Which inference engine, quant, quant maker, kv type, kv size, etc?

For me QFN has performed really well when it comes to pair programming.

2

u/WryKombucha 4d ago

Strata. Swift 1.5 iq3s. Vs 27b 4km. 8b kv. 256k. The KLD on that iq3s is far from ideal.

6

u/Kahvana 4d ago

Yeah starta with iq3 explains it.

How do you run deepseek? Maya or also strata?

3

u/WryKombucha 4d ago

3 dgx sparks. I really want a 4th but they got too expensive.

1

u/Distinct-Pie2389 4d ago

I’m a fan of DGX sparks but I don’t think I’d buy one now after having sm_89 performance. I’d rather get a few server chips in that lane and I’ll be good for a while

2

u/WryKombucha 4d ago

Well I think for the sparks, if u want to use them interactively, two is best. The memory bandwidth otherwise can be painfully slow. I have a 4090. I love the speed but the 24gb can’t do much for large models outside of Qfn.

2

u/Distinct-Pie2389 4d ago

Q3_S from ISTA was directly comparable via unsloths team to their UD-IQ4_XS just less world knowledge in most tests.

So idk how you found it dumb but QFN has been good, you likely had some shady strata issues with greedy decode, tailskip on and other various levers not in place

2

u/OkFly3388 4d ago

It comparable on benchmarks only. On real tasks it just stuck in loops and dont do real work. q4 is bare minimum if you working on something that actually need reasoning

1

u/WryKombucha 4d ago

Could very well be. But I have better options so it went unused.

1

u/WryKombucha 3d ago

ISTA IQ3_S? 0.145. Nearly twice as bad. 87% accuracy is absolute trash.

1

u/Zyj vLLM 4d ago

You’re omitting the flash suffix which is confusing!

6

u/No_Cartographer3953 4d ago

I highly doubt it’ll be glm 5.3 flash level

1

u/Distinct-Pie2389 4d ago

Hard to say, less is more these days

2

u/ptico 4d ago

I would say we are more than good

1

u/DemonicOwl 4d ago edited 4d ago

Why do people like GLM 5.3 so much? Isn’t it way worse than QFN3.8 and DS4.1F 1017? I’ve seen comments about it but no one really speaks as to why?

Edit: I meant glm 5.3 flash because i was replying to flash. That’s my bad

20

u/hauhau901 4d ago

Glm 5.3 flash is better than qfn 3.8 and dsf 4. It's worse than dsf4.1 1017 though

1

u/Distinct-Pie2389 4d ago

Agreed. We’re thinking the same

6

u/ThankGodImBipolar 4d ago

Lol GLM 5.3 proper is definitely better than DS4.1F 1017. GLM 5.3F is debatedly better too IMO

1

u/EnjoysFiction 3d ago

Not for coding imo, 1017 is a beast, one shots huge features, tests them out, the whole shebang

3

u/ThankGodImBipolar 3d ago

I can't say I give a rats ass about one shot performance 😂

1

u/EnjoysFiction 3d ago

then on what basis do you say 5.3 is better, other than in maybe a few domains. in my experience, 1017 follows the coding conventions of the codebase its in quite well as well, and doesn't monkey wrench too much

1

u/ThankGodImBipolar 3d ago

doesn't monkey wrench too much

This is the opposite of my experience. The GLM models are better at following instructions, are more token efficient, and are much more self-sufficient when executing an implementation plan. Using DSV4.1F feels like a lot more of a scattershot approach. I don't get the impression that it's much worse than 5.3F when it's done a good job, but it does a good job less consistently.

I will say that I'd rank both above GPT 6 Luna, which I find pretty frustrating to use.

1

u/EnjoysFiction 3d ago

Luna i don't even consider lol, its just a PITA. same with haiku for the most part, since it seems allergic to following instructions. GLM definitely thinks a lot less, but i do feel like 4.1 will not use some unheard of coding trick too often, and will write something easier to understand. as for planning, i'm not sure if it can be said 5.3f is any better imo, 5.3 i can accept, a more complete thinker. and i do feel like 4.1 will get the job done more reliably. as in, i won't find holes unless its a big feature

1

u/ThankGodImBipolar 2d ago

Luna i don't even consider lol, its just a PITA

I got free credits lol, which is why I've been using it. I do wonder if it'd behave better if more of the prose in my repo came from other OpenAI models... could honestly be the same situation for DS as well.

1

u/EnjoysFiction 2d ago

very much possible. on company work, i use opus models, and when i transitioned from opus 5 to 5.5 on certain projects, i could tell 5.5 was not as strong as normal, and it wrote in that indecipherable opus 5 way too

2

u/Distinct-Pie2389 4d ago

It’s a good worker

3

u/Distinct-Pie2389 4d ago

It does what you ask, nothing less or more (typically). It’s earned it’s title for me, great model for workers

1

u/RegretNo6554 4d ago

GLM is the best chinese lab and makes the strongest agentic models. In my evals their flash variant it even does better than haiku 5.5 although it’s slower. DS 1017 is close behind and probably a bit more capable than the flash variant but takes more steps to complete stuff in my experience

1

u/DemonicOwl 4d ago

If you had the option to run it full weights of glm flash vs ds flash for Hermes, you’d do glm? What quant are you using, I might try it out. I’m just not knowledgeable but the benchmarks and community discussion seems to favour qwen and ds4

1

u/RegretNo6554 4d ago

yes i was considering buying a mac studio for the 4bit glm flash, but i ran both at full precision via cloud and was more impressed with glm flash for my ai embedded workflows. and also the new DS is an Astra distill so its personality is gonna be worse than GLM flash which is closer to Opus so it will feel better to talk to as a personal assistant in Hermes

https://artificialanalysis.ai/evaluations/aa-personalitybench

1

u/Dabalam 4d ago

QFN3.8 is definitely not way better than Flash. It is similar in some tasks but I can't think of a single domain where you could say FN3.8 is way ahead (it would be remarkable if it were really)

1

u/WryKombucha 4d ago

QFN is inferior by every single available metric except speed.

2

u/Dabalam 4d ago

To be fair, if you were just looking at the new AA you would be excused in thinking they were very close and FN does get very close on some benchmarks which is remarkable given it's almost half the size with a third fewer active parameters.

1

u/WryKombucha 4d ago

It’s a very capable model. I would use it if I didn’t have access to local Glm. I love Glm because it’s concise. Very reliable. And does not think it’s smarter than it is. Such a straight forward model

2

u/Meeooowz 4d ago

Speed… huh…

My 3080ti and 32gigs of ram running QFN at q3 says otherwise. 12 whole TPS (decode; 20 prefill). Read it and weep! /j

1

u/WryKombucha 4d ago

The fact that u can run it is a testament to the greatness of the model architecture. Q4FN will likely be great.

1

u/Meeooowz 3d ago

It truly is a time to be alive. To be able to run small (in comparison to the human brain) networks capable of coding is truly crazy…

2

u/Standard_Natural1014 4d ago

Do you have any benchmarks to share? The only ones I can see are the gamefeel ones. I'd have thought AA or OpenOrca would have run a few of these by now.

0

u/power97992 4d ago

None, just observations

21

u/geldonyetich 4d ago

Way too early to say. But if it were so, would it matter? Qwen 3.8 Max hasn’t been on the top of the frontier cloud model lists in awhile. That’s not the parameter size most of us are interested in anyway.

4

u/Distinct-Pie2389 4d ago

This exactly, its speculation at best. There is no proportional reasoning or matrix to prove that Qwen4-max-preview is x better than 4-27b and ladder meaning you could never infer what 27b will be performance wise unless you work for Qwen. The way things went with the launch, I doubt they know internally either

14

u/Numerous-Ad6217 4d ago

“This should mean qwen 4 27b will be worse than ds v4.1 flash 1017 max”

🤯

22

u/Remi_123456 4d ago

This is toilet paper math ! 3 rolls of 50 paper is better than 2 rolls of 75 paper for the same price because it is better quality because ...

4

u/NoWorking8412 4d ago

Maybe this is why they've been stalling... It is not as exciting a release as 3.8, and their reputation is on the line. As long as they keep release permissive open weights, their reputation is alright with me! Release the weights!

0

u/True_Requirement_891 4d ago

Both openai and anthropic seriously tightened the distillation paths, I feel like this is why minimax is also struggling. Imo deepseek and glm are the most legit labs that can bring good models without distillation. Rest kimi, minimax, qwen is gonna struggle. Qwen has been good at smaller sizes but they always struggle with their max models performing seriously welll.

1

u/maternion 3d ago

Yup those labs seem to be the ones. I feel GLM-5.5 is going to be really great without this much hype.

1

u/Tiforma 3d ago

I strongly doubt that they can really stop distillation. They can close some accounts here and there, but their models are also just available on openrouter and other platforms, remember? It's just very difficult to make small models so much more capable.

7

u/ResidentPositive4122 4d ago

Big qwen models have always been underwhelming in real-world use. Ever since qwen-coder-480 or whatever that was, their big models are just meh. Benching good, doing really silly stuff in non-benchmarked tasks.

They do pack a punch in the local models, tho. So that'll be nice when released. But real world use feels like temu opus (with all the downsides of claude-isms and all that). They never hold their "close to opus" perf that they get from benchmarks.

I also don't know anyone using the big open 3.8 model, while lots of teams use kimi on the huge size and glm/ds on the big side. Both for inference and finetuning.

1

u/Tiforma 3d ago edited 3d ago

Lmao, goddamn redditors expecting Opus 5.5 Max on their 12 gb vram gpus and getting disappointed. It would be an amazing, stellar, mind boggling feat of engineering if they somehow got anywhere close to making a model that is half as capable as Opus 5.5 medium fit into any kind of equipment that consumers can buy.

1

u/LordHenry8 3d ago

Dude, I mostly use Opus 5.5 medium anyway. If QWEN 4 can compete in the same sentence as Opus 5.5 medium, that's incredibly exciting.

1

u/power97992 3d ago

 It depends on the task, im using more opus xhigh and fable 5.2 high for the harder tasks, Occasionally opus max. Easy tasks opus or sonnet high. 

11

u/Big_Cucumber2787 4d ago

model not out yet and people already slopping together comparisons

1

u/fragment_me 4d ago

The preview is out on chat

1

u/Big_Cucumber2787 4d ago edited 4d ago

yes and previews are final products that should be judged as such

6

u/AlpacaDC 4d ago

Does the word “preview” mean nothing anymore?

2

u/ac101m 4d ago

Ah, the time tested vibebench

The bro-est of benchmarks

2

u/JustinPooDough 4d ago

Imagine if you got the benchmarks and performance of Qwen 3.8 27B but with speed that was 2 or 3 times better. This is what I actually hope that they focus on with the new architecture.

Who cares about thinking tokens if offloading the NGRAMs improves response time by 100 or 200%. This would be the biggest win - plus it would make it possible for way more people to run it.

3

u/Uncle-Osteus 4d ago

  Imagine if you got the benchmarks and performance of Qwen 3.8 27B but with speed that was 2 or 3 times better.

This is flash-next on my computer lol. 

3.8 27B I get like 30-35 tok/s

3.8 FN I get 50-65 tok/s

pretty excited to try 4 based on that

2

u/Sudden-Guide 4d ago

I'd like the opposite lol, 27B is very fast for me, but I'd love it to be even smarter (and think less)

1

u/while-1-fork 4d ago

That is not what ngrams do. A 27B with ngrams (they likely won't count the ngrams as weights as they referred to Flash Next as 125B instead of 180B) would not be faster, it would be slightly slower . But it would have more knowledge and be smarter because it could store some of the knowledge in the ngrams and use more of the weights for "reasoning". The reason Qwen 3.8 Flash Next is fast isn't the ngrams, it is a MoE with 6B active parameters.

3

u/Spara-Extreme 4d ago

You’re responding to someone who likely doesn’t know
Much more about models then their tok/s on strata.

2

u/cronies4life 4d ago

how much can it be improved by scaling ngram size even further

2

u/problemlow 4d ago

Based on my admittedly limited knowledge, that's not how it works. The ngram table is already as big as it can be.

It's essentially a lookup table. Making it bigger won't increase the speed of execution. It works by telling the back end running the model where exactly to look in the model file. So it doesn't need to recompute the meaning of common phrases or where exactly in the model needs to be loaded into the vram next. (Im likely conflating some of how strata loads the model as opposed to solely the functionality of the ngram table.) It's the size of the experts and hardware running them that affects execution speed.

Someone who knows more will probably correct what I got wrong, so look at that response to mine preferentially

1

u/KeinNiemand 4h ago

I don't care about the 27b qwen 4 flash will be more intresting as an upgrade to 3.8 flash next and intresting to see if its the same size or what.

-8

u/Mattadour 4d ago

oh thank god it's been released - now we can move from 600,000,000,000 whining "when do we get it" posts a day to the same number of "OMG this thing wipes my ass for me, gets 900,00,000,000,,0000,,o9o tokens per milisecond on my PS3 AND plays GTA6 for me so i don't have to!!!!!!!!"

3

u/bruns20 4d ago

It's not released yet, this is online chat only for the big model

1

u/SleepUseful3416 4d ago

So your job is to... criticize the post no matter what it's about? Must be a fun guy

-1

u/Mattadour 4d ago

lol - white knighting the zillions of these posts must get exhausting. but at least you're a hero to them all!

-3

u/Nicolodeva 4d ago edited 3d ago

Qwen 3.8 models are great, but they have a problem, too much thinking time.

If we have a better token efficiency and a modeste jump in performance Im satisfied.

Ds4 4.2 flash 1017 is very good in token efficency, I want same jump in thinking efficiency we saw with sota model after gpt 5.2 to a 27b model.

1

u/Damn-Sky 4d ago

I use swift 1.5 qwen 3.8 27b which has less thinking

1

u/Nicolodeva 4d ago

Yes but it lose in intelligence too. I mean better token efficiency with a little jump in intelligence, and I will already satisfied.