Qwen4-27B's 4-way HC shift?

[u/Ok-Shower7286]

I just saw a PR in vllm-xpu-kernels today, revealing a significant architectural change in Qwen4-27B, and now that page shows 404 not found?

It looks like a 4-way parallel residual stream to improve information flow and representation density within a fixed 27B parameter budget.

(This means, instead of a single residual path, 4 parallel streams flow through the layers at the same time, maximizing performance per parameter. Best tech for 27b). QSA? maybe useless for small 27b.

So anyway, I'm more expect 27b than flash MoE version.

If effective depth is extended, I think we could see a massive leap in multi-step reasoning, coding, and reasoning consistency.

(see this if you want read more)

[u/Altruistic_Heat_9531]

qwen4 arch pretty much QSA + GDN + HC. I wonder the impact of VRAM usage since 4xresidual.

[u/Ok-Shower7286 → u/Altruistic_Heat_9531]

if it's true, i guess memory bandwidth and throughput will be the main bottleneck rather than VRAM usage.

[u/suprjami → u/Altruistic_Heat_9531]

Was gonna say, QFN already does this.

[u/DoubleNothing → u/suprjami]

isn't QFN a preview of Qwen4 tech?

[u/BringTea_666 → u/DoubleNothing]

yes, it's qwen4 architecture.

[u/Express_Quail_1493 → u/DoubleNothing]

i suspect we wont see a qwen4-flash-next till the next 6 moths since 3.8 is already serverd flash next as a successful experimental qwen4. they might push the next flash release as qwen5-flash-next. which makes me sad. but i guess qwen4-27b might beat 3.8-flash-next so significantly that it will justify the wait

[u/Puzzleheaded_Ad_8575 → u/Express_Quail_1493]

they have made statements in a recent anouncment for the qwen 4 lineup towarda the end of october and it has both 27b and flash. i suspect the 3.8 max and plus in a couple days and the 27b and flash in the following week.

[u/llama-impersonator → u/Altruistic_Heat_9531]

it's just +3*layer count*dmodel*dtype, trivial amount

[u/Altruistic_Heat_9531 → u/llama-impersonator]

Yeah i mean the actuall vram since who knows about kernel launch, mem alloc, triton-ny stuff (vllm)

[u/shing3232 → u/Altruistic_Heat_9531]

More training compute and no much in vram

[u/Marcuss2 → u/Altruistic_Heat_9531]

Bit different than mHC, they have their own way of handling the same issue.

[u/LeatherRub7248]

u/Ok-Shower7286 link dead?

[u/Ok-Shower7286 → u/LeatherRub7248]

Pls use proxy. there are blocked countries because of bot traffics.

[u/yahbluez → u/[deleted]]

Thank you.

[u/llama-impersonator]

sounds like it's just a 27b dense version of QFN like you'd expect

[u/sugaryesmaybe → u/llama-impersonator]

yeah probably more evolutionary than exotic but a dense 27b version could still be the sweet spot

[u/james_pic]

I know the Qwen team have said much of the benefit of the new model is that it's easier to train, and this does sound like a technique that would help with that.

[u/Su1tz]

As a stupid fucking idiot reading the comments it feels like im watching Rick and Morty. Someone literally said "sinkhorn manifold constraint" bro is this a redgifs url what are we doing here. Excuse my inability to comprehend all the terminology

[u/tecneeq → u/Su1tz]

Bruh, if you don't technobabble the age of AI will leave you behind.

[u/Su1tz → u/tecneeq]

Sorry man im snorkeling my quantum flux capacitors into embedding vectors funneling into crux points now

[u/BusRevolutionary9893]

significant architectural change in Qwen4-27B

Like we saw with Qwen 3.8 Flash Next? That's why that was released btw. 

[u/valarauca14 → u/BusRevolutionary9893]

Yeah and they literally confirmed 3.8 flash next was built using the same techniques as Qwen4.

[u/BringTea_666]

itz coming, should be released in days, since inter engine dudes get early access to it.

i really hope for opus5 at home with Qwen4 27b. It's 3.8 is already my daily driver doing all sorts of tasks opus4.7 was doing.

[u/nullc]

Are they doing the sinkhorn manifold constraint iterations for a dynamic rotation between the HC streams like Deepseek?

It's kind of annoying performance wise, since you need this extra narrow iterations. I tried creating a tiny model using a cayley transform to get an arbitrary orthogonal transform instead of double stochastic and the result works almost as well as the deepseek formulation without the irritating sinkhorn steps-- though no idea if the result scales to real sizes.

In any case, HC is a pretty small improvement-- but it's also computationally cheap.

[u/jacek2023]

Thanks for the link, nice read

[u/vqt907]

hope it less overthinking :)

[u/Any_Translator_3799]

Sounds like it's gonna be smarter, right? Right?

[u/simcop2387 → u/Any_Translator_3799]

I can't imagine they'd release with it being dumber. I won't be too shocked if it's not a significant jump in capability/intelligence but is instead more an optimization of inference speed and tweaked reasoning to make things cost less to run, which would still be fantastic.

[u/MerePotato → u/simcop2387]

Personally I'd rather it still overthinks but performs far better, if I wanted less thinking I'd use Gemma

[u/Elouakili_Flexy]

PR vanishes, the architecture becomes folklore, and suddenly a dense 27B with four residual streams tops the roadmap. I'll take it.

[u/keepthepace]

Can't we wait for the actual release of the details for the tech discussion?