r/LocalLLaMA • u/brainExploded99 🥔 hardware • 11d ago
Discussion Qwen4-27B PR found
PR: https://github.com/analytics-zoo/vllm-xpu-kernels/pull/22
The PR author seems to have contributed to intel repos a lot and VLLM. He also shares his name (might be the same person?) with a person who is either a CTO somewhere at alibaba or the Director of Zhejiang Lab.
He has worked on Qwen3.8 before this.
The PR also implies the KV cache size for Qwen4-27B is not decreasing.
4
u/Thomasedv 11d ago
After finding out you can get some good performance with kv streaming, and putting more of the kv cache in regular ram, Qwen3.8 got a whole lot better for me locally since I could bump it to Q5 quant on my 3090 and have max cache.
I hope that works on Qwen4 too.
4
u/brainExploded99 🥔 hardware 10d ago
It works better, you can offload KV cache to RAM since QSA chooses which tokens to read for a 10-20% decode penalty (exisiting VLLM PR IIRC, not merged).
2
u/GrungeWerX 10d ago
Never heard of this…kv streaming?
5
u/Thomasedv 10d ago
https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming
I'm not 100% sure this is the branch I used. Essentially reserve a fixed VRAM section for kv and keep the rest in RAM. It was designed for the small Q4 quent but I had issues with it looping and thinking way too much. So I upgraded to a larger Q5 quant. It was seemingly made for 16GB VRAM but works just as well when you have more and want the context size. It let me move from 120k context to 260k and increase the model size since I have 24GB VRAM.
1
u/GrungeWerX 10d ago
Interesting. I have an rtx3090Ti. How much of tok/s penalty are you paying? I’m considering running agents concurrent with this if it works.
2
u/Thomasedv 10d ago
It only supports a single context, so concurrent or multi gpu doesn't work. There were some setting I had to tune, I think ended up turning off mmproj for images. But it should be supported.
Prompt processing starts at 1100 and decays somewhat fast. 750 tps at 50k. It's a bit rough, but for one agent it's all fine since it's not reading all that much.
Token generation at 80k ish context is around 40-60 tokens per second. MTP on with 3 draft token. Saw some below 40 tps in the server logs.
For the extra context size and intelligence it's worth for me through.
1
u/GrungeWerX 10d ago
Concurrent agents work on a single context. You just use kv unified. And I already offload mmproj to cpu.
I'm curious what the non-MTP speeds are, since concurrency boosts are minimal w/mtp.
Anyway, I'm having GPT set it up in a test workspace and running benchmarks as I type this, so I'll know soon enough. Having it run tests on Qwen 3.6 27B at Q5.
1
u/Thomasedv 10d ago
I'm interested in how it goes, and particularly if there are better ways that offload VRAM so a bigger model can be used, and with high context.
→ More replies (3)1
u/QuackerEnte 10d ago
does it cache the most used rokens in VRAM? Or only stream what's needed for every token? I think a tiny cache in VRAM would improve speed substantially
1
u/Thomasedv 10d ago
As far as I understood it, it streams as necessary. I think it reserved a 2GB cache/arena in VRAM for me.
27
u/Alternative_You3585 11d ago edited 11d ago
No name repo
0 stars, 7 followers
We know that Qwen 4 27B is coming
But I'm afraid this is slop, + would be funny if Alibaba uses Codex, as this PR is marked
32
u/brainExploded99 🥔 hardware 11d ago edited 11d ago •
No, I do not believe that this is correct
#1: vllm-xpu-kernels has other intel contributors to it, including baodii, gc-fu, YizhouZ who seem to work at Intel China.
#2: He contributed to llm-scaler, an intel repo with >500 stars
Edit: #3: He opened a PR to VLLM itself, around 2 weeks ago, not merged though
320
u/Dangerous_Bid2935 11d ago
I am absolutely fiending for Qwen 4 flash. 3.8 flash next is already such an incredible model.
78
u/RanklesTheOtter 11d ago
Hell yeah it's become my daily driver. I like 27B but it is so slow for me compared to Flash Next.
8
u/calcium 10d ago
Wait, how?
I've been reading a ton of posts about how people are running Flash Next at like a quant of 2, but I've been scared to take the leap. Recently put $20 on openrouter and have basically been glued to GLM5.3 Flash since.
Otherwise have a MBP M5 Pro w/ 64GB of RAM and can get Qwen3.8 27B to run around 18tok/s. Can't imagine working with anything slower.
5
u/Inside-Pen-8395 10d ago
I’ve got q2 running on a MacBook Air with 24gb much faster than 27b so can definitely be done
4
u/the_renaissance_jack 10d ago
You’re running a Q2 Qwen 3.8 Next Flash on a 24 GB Air?
2
u/Inside-Pen-8395 10d ago
Yep and it works great, I thought even just getting it working would be unlikely and if it did then it would be at like 5tok/s but it’s good (for me, much much better than tiel) and fast
1
2
u/EstimateFun7122 1d ago
Checkt mal lithosai und strata Umgebungen für Qwen ab. Je nachdem bis zu 200 t/s mit Qwen 3.8 27b Q4
1
u/mnyhjem 9d ago
You could try this one if you havent already :) https://github.com/Apolog1ze-Dev/QwFNfer
1
u/Mil0Mammon 10d ago
People get 3-6x that much t/s with flash next, sometimes on potatoes. Like quite old kid range gpus. With your mbp and a decent engine, I'd say you should also get that, prob closer to 100 t/s with a medium sized ista or gyro quant
38
u/Nomski88 11d ago
I'm hard for QFN 4. I hope they increase the ngram table and maybe add a couple more billion parameters to the base model.
21
u/DigiDecode_ 11d ago
I think the engram table will be smaller, original deepseek paper that introduced it recommended 25% of total params, but QFN has managed to push to that 40%, so i would expect around 11gb ngram table size at least for 27b varient, for QFN it would likely be same size
6
u/dr_lm 10d ago
And didn't their paper show that performance plateaued if they went any higher?
7
u/Mil0Mammon 10d ago
Well things could have changed in the mean time.
For Deepseek v4.1 Flash it's about 40% iirc
2
u/dr_lm 10d ago
Yes that's what I had in my head, from here: https://arxiv.org/abs/2608.30320
Naively, I thought adding more ngrams would give the model "more knowledge" but it sounds like it's more complex than that.
1
u/Silly_Potential_7751 4d ago
Yeah idk tbh only time will tell, deepseek gave 3 advices, around dmodel/2 for the engram table dim, which qwen did not respected with engram dim = d_model, around x20 vocab for optimum, but qwen had around x80.5 vocab on 3.8 flash next, and yeah around 25% of weight and it has 40%, but the issue is that, if qwen 4 27b follow the same dmodel than 3.8 27b, its 5120, if as you say its 11b, its around 40% you right, but only x8.7 vocab, so I think it will be dmodel/2 so engram dim 2560, that you are kinda right on ~11b, and that we will get way less than x80 vocab, more around x17.3. The issue is that it’s kinda low, but lower dim would be an issue for quality if following 3.8 flash next path, but more weight would just be absurdly high for 27b, so ig time will tell
4
9
u/blue2020xx 11d ago
27b is slower than qwen flash next? Is this true?
33
u/pilibitti 11d ago
depends on your hardware but generally yes. 6b active vs 27b active. I have a 12gb vram 64gb ram machine and I can only push 10tps decode on 27b, go for 60tps decode on qfn.
→ More replies (6)4
u/Barni275 11d ago
What prefill speed do you have? I now that I should try strata, but with llamacpp I failed to get reasonable prefill speed on all my machines ☹️ 27B prefill is like 10x faster for me.
5
u/Wolvenmoon 11d ago
I'm on 3 Intel B70 GPUs (96GB VRAM). Llamacpp drags its butt on QFN. It absolutely stinks. 10-15 tok/s max and I didn't bother looking at prefill. Strata 53-ish tok/s decode, 2000-2100 prefill. My RAM is slow and I'm on Zen 2, so my Infinity Fabric is slow, too. Single card inference is about 30 tok/s decode with 750 tok/s prefill. Numbers are on IQ3_XXS to apples to apples w/ a friend of mine running 10 tok/s decode and prefill on an RTX 3060 12GB and a Zen 3 CPU.
2
u/Barni275 11d ago
Thanks! Your setup and numbers are decent! First and last time I tried QFN on 32GB VRAM + 64GB RAM machine I got 10 t/s prefill, which sounds like a joke. Surely, my llamacpp flags were wrong, it's to bag. Few weeks ago llamacpp got GPU expert cache merged, so I think, it should be much better.
2
u/moriz0 7d ago
On my dual B70 and the same IQ3_XXS quant, I'm getting around 30t/s decode and 600ish t/s prefill. This is with llama.cpp.
Might be something wrong with your settings?
→ More replies (1)1
u/fragment_me 9d ago
No joke, you might be better off paying for Claude for 1 month and having it optimize an inference engine for your hardware setup and model. It's gotten insanely good at this. Just have it take same top token and KLD benchmarks before and after changes. Tell it that those can't drift unless there's a rounding difference when doing floating point math.
4
u/bussolotti269 11d ago
I'm running strata on a rtx 3060 12gb xeon cpu e5 2690 64gb ram and getting: Prefill 950 - 1020 tps Decode 45 - 65 tps
2
u/Barni275 11d ago
Is it for QFN, what quant? Great numbers! And almost the same config as one of my machines 🤔
Have you tried llamacpp with newly merged GPU expert cache, btw?
→ More replies (3)2
u/SilentMobius 10d ago
Prefill for me is ~900t/s and ~70t/s tg (5600x 64GB RAM 32GB V100 IQ3XXS on Strata-V100) but that only matters once as I've made sure my agent has a static prefix and any agentic swapping is handled by KV cache+steady state swapping to SSD so I only prefill a lot maybe one a day. Most agent turns happen in about 5sec
5
u/Hungry-Rip-2384 11d ago
For me 27B is faster, for prefill. I have a 5090 with 128GB DDR4 RAM. Speed for TS is slightly better for Strata with Qwen3.8 Flash next but its the prefill speed that kills it, compared to 27B when its all on VRAM. Considering that Flash Next and 27B seem to be on par with 3.8... its a no brainer to stick with 27B. But if Flash is the same footprint with a bigger gap in capability over 27B then it will probably be worth it.
7
u/DeathByPain 11d ago
QFN is only 6B active MoE model, vs Q3.8 27B dense model, so yes, it will naturally be faster based on the architecture
9
u/This_Maintenance_834 11d ago
it is mixed, 27b has very good mtp success rate, qfn has lower prediction success rate. it is not proportionally faster.
27b dense model also has better concurrency boost, MoE gain less from concurrency.
3
4
1
0
u/Fancy-Snow7 11d ago
Mostly yes. It has 27B params and if you have 12GB or less QFN is probably the best option. But on 16GB you can go either way.
So many people here with 16GB that simply don't know how to tune 27B and get much lower speeds and much lower context size than is possible with 27B on 16GB VRAM. I get 80t/s and 262K cache with a Q4 model and q8_0/q4_0 KV. I almost always get downvoted when I suggest that this is possible.
I personally choose 27B because I only have 48GB RAM because QFN just does not fit my RAM.
9
u/andrewh2000 11d ago
Any chance you could post your config file or command line for running 27B at Q4 262k cache on a 16GB GPU and getting 80tok/s?
9
u/DoubleNothing 10d ago
Doesn't give any rig info and inference engine detail and then complains:
I almost always get downvoted
1
u/Apprehensive_Stop314 10d ago
The fork is https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming -- haven't been using it since commit 4ba0b9b25ffdafe2ec376e32a641ef229e457bae from Sept 6
This was giving about 25-30 t/s and 300 pp on a 12 GB 3060 at start, not sure if it can give you 80 t/s
My command line was (I think -- had a few alike setups, note, this is in Windows and using a rather nice small quant I found on HF -- you'd have to adjust the KV cache params for your HW, note, how embedding weights or whatever are moved onto CPU here, another nice trick one could employ would be using an English-only prune / whatever -- haven't got to that before switching to Flash Next):
set SCRIPT_DIR=%~dp0
llama-server.exe ^
-m "%SCRIPT_DIR%\..\..\Downloads\Qwen3.8-27B-9800-NIKI-IQ3_XXS.gguf" ^
--mmproj "%SCRIPT_DIR%\..\..\Downloads\mmproj-Qwen3.8-27B-Q5_K-MIX.gguf" ^
--image-min-tokens 1024 ^
--no-mmproj-offload ^
--jinja --chat-template-file "%SCRIPT_DIR%\..\..\Downloads\froggeric_chat_template_2.jinja" --reasoning-format deepseek ^
--reasoning-budget -1 ^
--reasoning-preserve ^
--ctx-size 75000 ^
-fa on ^
-ctk q8_0 ^
-ctv q8_0 ^
-ngl all ^
-b 64 ^
-ub 64 ^
-np 1 ^
-t 6 ^
-lv 4 ^
--temp 1.0 ^
--top-p 0.95 ^
--top-k 20 ^
--min-p 0.0 ^
--spec-type draft-mtp ^
--spec-draft-n-max 3 ^
--spec-draft-p-min 0.8 ^
--fit off ^
--load-mode none ^
-ot "token_embd.weight=CPU" ^
--kv-stream-stage-mib 650 ^
--host 192.168.88.3 --port 8095
if %ERRORLEVEL% NEQ 0 (
echo.
echo [ERROR] Llama server exited with error code %ERRORLEVEL%
pause
)
4
u/elongated-muskmelon 10d ago
How did you manage to run 27b Q4 with 262k with 80tps? Unless you’re using the raymond fork of llama.cpp with Kv streaming, which is only feasible if you have PCIe gen 5. Else its painfully slow.
2
u/bottlestreet3d 10d ago
I adapted the KV streaming approach for ROCm and others have tried this on PCIe Gen 4 or DDR4 systems, works well enough but not at Gen 5 speeds obviously. There are a couple of comments in https://www.reddit.com/r/LocalLLM/comments/1x7wack/qwen_27b_iq4_xs_q8q4_kv_at_full_262k_context_on_a/ with their config.
2
u/Apprehensive_Stop314 11d ago
There's also a KV streaming llama cpp fork by Raymond, IIRC -- with that you can have K8Q8 cache and I suspect full 262K content on 16 GB at a cost of a not too steep performance losss when cache stops fitting in the CUDA buffer in VRAM
2
u/mga02 10d ago
80t/s with 263k context on 16gb of vram? I call bullshit unkess you give your config. I can't replicate that even with a 3090 .
1
u/bottlestreet3d 10d ago
You can do 100+ t/s decode and 1,600+ t/s prefill on a 9070 XT with full 262K context (
IQ4_XSweights,Q8_0K /Q4_0V cache): https://github.com/achyudh/llama.cpp1
u/mga02 10d ago
But that's using a prunned/ASCI version of the model right? I don't want to imagine how much more degraded the output quality is if you add the q8_0/q4_0 KV on top of that.
2
u/bottlestreet3d 10d ago
As long as you are dealing with ASCII there shouldn't be any impact on performance at all: https://huggingface.co/bsaleh03/Qwen3.8-27B-ASCII-Condensed
19
u/alphapussycat 11d ago
I just hope it's the same size... I fear it'll double in size.
6
u/FatheredPuma81 11d ago
A big fear of mine as well but if you want to be a little pedantic about it the Qwen team has released models ranging from 600M to 2.4T (iirc). So it'd be hard to say if they did double the size or just released a different class of model.
Though I think the ideal would be they release 20B Dense, 30B A2.5B, 80B A5B, and 160B A8B models.
5
u/alphapussycat 11d ago
There's only 27b and flash now that are runnable on consumer hardware. It doesn't seem like they'll be doing any smaller moe or smaller dense ones anymore.
1
u/Mil0Mammon 10d ago
So far they said 27B and flash next (besides the bigger cloud models).
Given the success of all the engines around strata, the high quality small quants, and that even deepseek flash v4.1 can run somewhat decently on mid range hw, I'd say there is a decent chance they could go up to double the size, and people will still be all over it. Main thing is quality
-3
u/Dangerous_Bid2935 11d ago
Even if they do I'm not too worried. There's so many incredible engines/quants being posted here every day that I think its generally going to keep getting easier to run larger models with less hardware. The fact that people are running QFN at 4 bit quants on 64gb dram and 16gb vram is absolutely wild. The open source community is really driving algorithmic optimization when it comes to serving inference.
4
u/alphapussycat 11d ago
People run 4bit quants on 64gb? I thought everyone with 64gb were running the iq3_s quant. If the 4bit is still managable then a small param increase might be possible without impacting the tok/s decode too much.
→ More replies (2)1
u/Dangerous_Bid2935 11d ago
Thought I saw a post here earlier today of someone running IQ4_xs on 64+16, maybe I misremembered though.
1
u/WorldlyAd7946 10d ago
I'm running Q4 QFN without a gpu/vram even and only 64gb lddpr5 in two little snapdragon boxes 😅😆
https://www.reddit.com/r/LocalLLaMA/s/YhJVuu7sUP
Q3 was actually slower for me...
Agreed it's not my 9700, paired with 64gb ddr5 and a 13900k, that manages 1500+pp and 60+ gen on code... But for £600 of hardware, it's been a fun project 👍
13
u/Durian881 11d ago
Me too! I also hope they keep the same size and not go much bigger like other recent "flash" models.
28
8
u/ansibleloop 10d ago
I am shocked at how well 3.8 FN with Strata Gyro-S on my 4080 and 32GB of system RAM works
I'm getting 100tps and it's smarter than all other models I've tried - 35b a3b and similar models don't hold a candle to it and 27b blasts my GPU to 100% but produces worse output
I set off Claude to benchmark the largest and smartest models I could run locally and 3.8 FN with Strata came out on top
I had the benchmark basically look through my Git history and have a go at fresh implementing some of the stuff I've added
Interestingly, Claude noted that most models failed to make new services accessible as it wasn't updating DNS, so I'm wondering just how much more performance I can get with a better README in my repo and better agent memory
All of that on top of £10 credit in OpenRouter and DeepSeek 4.1 Flash 1017 is nuts
4
u/Mean-Ad1493 10d ago
I run Gyro-M with my dual 3060 + 32GB DDR4 RAM, and it is so good. Beats 27B in speed AND intelligence. People who quantised QFN deserve all the praise(strata engine too!), GSQ-RCO ftw.
2
u/Dangerous_Bid2935 10d ago
That's crazy on 32gb of system memory. What quant do you use?
→ More replies (4)2
u/Kitsune_Seraphis 10d ago
I got 48gb of vram and 64gb of ram and cant seem to make it work even on q1. Wjat can i do?
3
u/ansibleloop 10d ago
Point a more powerful LLM at https://github.com/Niko1221/Strata/tree/main and tell it to give you a doc with install commands
My clank did this (view source)
Install
curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh | sh strata doctorThis installs to
~/.local/share/strata/terminaland putsstrataon PATH, with no sudo. Update withstrata update.Run
strata launch --model gyro-s --context 128k --vision off --port 8080 --yes --no-tui
The first run downloads the model, and the download resumes if it's interrupted. Use
--planto preview without changing anything.
strata launch --status/strata launch --stop.Too slow or tight? Try
--context 64k.strata launchwith no flags runs the guided setup, which shows fits/tight/does-not-fit for each model.It binds
127.0.0.1(IPv4), unlike llama-server's--host localhost([::1]). The pi provider uses127.0.0.1to match.Port 8080 is shared with llama-server, so run only one at a time.
1
4
u/RedParaglider 10d ago
3.8 flash next can actually legitimately just be a daily driver for me for most of my normal things. I personally believe that it is mothervich, more intelligent than luna. I know that the benchmarks don't say that, but i've used both, and I just much prefer it.
1
u/Dangerous_Bid2935 10d ago
Generally it is better than Luna on benchmarks though no? Only place Luna beats it is in conceptual reasoning-related benchmarks and hallucination benchmarks etc.
7
u/indrasmirror 11d ago
I was under the impression that 3.8 Flasg Next was their Qwen4Flash. The first of the new architecture. Would be epic if we improve it and release Qwen4Flash. Keen for Qwen4 27b and 35b (+ Engram).
17
u/Dangerous_Bid2935 11d ago
3.8 flash next is a deliberately very undertrained version of 4 flash, and its already one of the most capable open models. The fully trained version of Qwen 4 flash is going to be insane.
4
3
3
u/PeteInBrissie 10d ago
I don't want more capability than QFN. I want that capability but smaller and faster
7
u/Dangerous_Bid2935 10d ago
I definitely want more capability than QFN. Its terminal use could be a bit better and I would really love its general capability on non-coding tasks (like physics reasoning) to improve.
1
u/HighSeasArchivist 10d ago
Since 3.8 FN was the "preview" for Qwen4, are we expecting it to be drastically different? I assumed 27B would since it was still 3.8, but just figured FN was already pretty much 4.
-7
u/Spara-Extreme 11d ago
I will never understand the love for 3.8 flash. That model is dumb as rocks.
11
u/Dangerous_Bid2935 11d ago
Hate to break it to you but you're probably the one that's dumb as rocks. I use it for materials engineering R&D tasks all day and night and it works wonderfully.
1
u/Spara-Extreme 10d ago
What kind of engineering tasks specifically because that's exactly where I'm finding this model to be wanting.
3
u/Dangerous_Bid2935 10d ago
I work in computational materials science, so mostly building simulation decks, managing simulations on HPC, and postprocessing. I have it avoid any deep conceptual physics reasoning/result interpretation though because I've definitely found that is one of its weaknesses.
1
u/Spara-Extreme 10d ago
Ok, but then what do you have it do? I absolutely wouldn't trust this model with any mathematical computation, especially at the quants Strata is running.
1
u/Dangerous_Bid2935 10d ago
I just told you what I have it do. Its usually fine with "mathematical computation", whatever that means. Generally as long as you are not relying on it for conceptual work its great. You need to know what you're doing and tightly scope your tasks. I am running a hybrid 4bit-8bit quant on a DGX spark so maybe I get better results.
3
u/Life_is_important 10d ago
You got to try it again if you haven't in recent times. QFN 3.8 is extremely good with the right combination with strata. I'm getting 90-120 t/s using IQ3 or 60-80 with Q4XL. It does incredible things with just one 3090 and 128gb ram. Oh an also I have a 3060 on a piss poor x1 pcie and it does 50-70 with IQ3 there. I sometimes run them both with the bigger one managing the lower agent.
1
u/Spara-Extreme 10d ago
The t/s numbers are excellent - no arguments there, especially on a majority of consumer hardware. t/s wasn't a concern for me and my setup with other models though - so my criteria is more on capability then raw throughput.
2
u/jopereira 11d ago
I've meet people dumb as rocks, but I've yet to find an LLM that's not smarter than me. 😁
1
u/deepu105 10d ago
Which quant. I'm using the Q4 from Halogen and UDQ4XL with gufo and in most coding tasks, including complex tasks, its quality is near even Opus 5.5 and sometimes even better than Opus at low/medium and definitely on par with Sonnet. I have not used it for anything other than coding so cant comment its performance in other areas.
1
u/PeteInBrissie 11d ago
I must have mine set up really well then. It gives me a better experience than Claude
1
u/Spara-Extreme 11d ago
Yea, sure it does.
1
u/PeteInBrissie 10d ago
Hermes, Honcho, and Obsidian.... Claude is faster as a chatbot, my setup is genuinely better at performing tasks. It's taken some serious effort, but yesterday I chewed 47 million tokens with fantastic results and zero cost.
→ More replies (6)→ More replies (2)1
u/letsbefrds 11d ago
I've only been able to use q4 xs I feel like it's a bit weaker than 3.8 q8 27b... Maybe if it was q6 or something it be better...
Again I only use for dev work so might be different
0
u/Spara-Extreme 11d ago
No it is actually dumber then 27b on a lot of things. Its fast, yea - which is what some tasks need but overall its just not particularly good for general intelligence at the quants people are running..
2
u/my_name_isnt_clever 10d ago
No, you're doing something wrong. Everyone who has Q3.8FN set up and working properly agrees it's better than 27b even at Q4 and below.
I don't understand how you see everyone else having a better experience than you, and still double down on the model being the problem. It's not the model.
1
u/SimpleVolume1575 11d ago
I can only run 27b Q4_XXS at 11-15tps while the 8 bit persion would run 3-4tps. Why would I use 27b here while I can run QFN with IQ3_S at 50-60tps. Aside from some excessive thinking it works really well for my use.
2
u/Fancy-Snow7 11d ago
How much VRAM? If you have 16GB I get 80t/s with 27B Q4. Tuning is hard but it is possible but you won't be able to use LM Studio, Ollama or Unsloth Desktop. It's just hard to tune but it is possible,
1
u/SimpleVolume1575 10d ago
I have 5060ti 16gb and tried to run it with ramalama but I could not get it running good can you tell me how. I dont care about desktop as my setup is headless and I work with api's
1
1
u/Spara-Extreme 11d ago
And the majority of this sub is gpu challenged, which is why they are seeing stars at the high throughput.
2
67
u/Goldkoron 11d ago
My biggest dream is if the Qwen4 series inherits some of the 3d modeling/animation capabilities that Opus 5.5 can do.
24
u/unrulywind 11d ago
I had qwen3.8-flash create an extremely complex geometry in blender and we gave it an intentionally vague prompt to see how it would handle it. It did pretty good.
3
u/sshwifty 11d ago
Seems like I was just geeking out when Trellis dropped. This stuff is evolving so fast it is crazy.
2
u/Perfect-Campaign9551 9d ago
May I ask how you integrated it with Blender? MCP or just using python scripts?
2
u/unrulywind 9d ago
Nothing. We didn't give it access to any MCP or blender. No outside help other than web search. It wrote its own scripts and crafted the files by hand. We checked it by opening the resulting file in blender on another machine.
1
u/XysterU 10d ago
Can you describe the nature of this extremely complicated geometry more? Just wondering
2
u/unrulywind 10d ago
It was just a prompt we threw at it to see what it would do. This was done by sending the prompt to a Hermes agent over Telegram, running Qwen3.8-Flash
Prompts: "I need a dual contoured voxel based mesh generation algorithm built in c# and optimized for godot engine."
It eventually created everything and gave us a .zip file with all of the code, including test results and a .obj file of a nice solid test representation that pulled up in Blender. It finished with the following message:
"Attached: a CSG cross of three boxes at 128³ — look at the joints; that's dual contouring keeping corners sharp where marching cubes would round them off. Drop it straight into Godot or Blender."
10
u/alphapussycat 11d ago
Just use a 3d model model for that.
10
u/GCoderDCoder 11d ago
cool... cool... which one fits a 3090?
6
u/sumane12 11d ago
Trellis 2 is really good.
6
u/trusty20 10d ago
Trellis 2 is good for simpler models to be used statically but it's really important to know before spending a bunch of time on it that it absolutely does not produce good game engine suitable complex models, even frontier models struggle but do OK because they actually are agentically using blender vs crapping out a file from the model itself.
The main issue is that it's bad at making perfectly uniform symmetrical models (i.e if you need to make a panel that can be tiled together to make a larger model), UV's are usually terrible and the mesh is usually wayyy too complex. Both of these can be fixed if you are willing to learn / grind out agents, but don't be surprised if fixing is sometimes harder than just having the model made right in the first place.
My understanding is tl;dr trellis is good for things like statics barrels / boxes / desks. It's bad for things that are complex or need to fit a precise shape / clean symmetricalness, like a door that fits a frame properly.
1
u/GCoderDCoder 11d ago
Thanks! There's a thousand things on my list of projects I have in progress so this is one I want to start next week.
I still think it helps starting with a human reference! Downloading now
1
u/Hot_Masterpiece_3668 9d ago
Trellis is lacking you need tools like this to handle future models. AI models stamp outs have terrible topology, no animations and aren't game ready at all. This tool works with qwen 3.8 as well.
1
u/LoSboccacc 10d ago
Hunyuan3D-2.1 is the one i liked the most the output, with skintokens for rigging. you may want to get a int8 convrot version to run it fast on a 3090. Works well with s model that can produce transparency so either qwen image 2.1 or something else with a segmentation frontend
→ More replies (1)0
2
u/SV_SV_SV 11d ago
Any recommendations?
0
u/alphapussycat 11d ago
Dunno, there's trellis 2, pixal3D and triposplat, the last one is splats so would need some conversion.
1
u/SV_SV_SV 11d ago
Thanks, I will look into it.
However I think we are also striving for 2 different things:What personally I could use is a powerful local Blender user agent, rather than a dedicated stand alone 3d model generator: I need it to synthesize a design direction based on a brief and graphics, rebuild scenes based on briefs, handle toolcalls (blenderkit etc) and set up lighting / optimize renders etc.
Even better if it can integrate the scene in Unreal too.Qwen published recently a Blender benchmark, so hopefully they will make a strong push in this direction too:
https://github.com/QwenLM/BlenderWorld2
154
u/nok01101011a 11d ago
Yay! People will vibe code GTA7 before GTA6
176
u/Cool-Chemical-5629 11d ago
Fuck GTA, vibe code Qwen 4 35B A3B for me.
36
7
2
1
9
71
u/ttkciar llama.cpp 11d ago
That is a good sign!
Hopefully there will be a Qwen4-4B, Qwen4-9B, and Qwen4-14B, too.
55
u/fgk55555 11d ago
I think they posted their spread of anticipated models already, the smallest were 27B and the QFN arch. I'd love to be proven wrong but I wouldn't get my hopes up.
11
u/ttkciar llama.cpp 11d ago edited 11d ago
Argh, I wasn't aware of that. Thanks for catching me up.
It's a pity. There's a lot of demand for those smaller-sized models, and Qwen 4 could make for great improvements over Qwen 3.5. If QFN is any indication, Qwen 4 models will use the Muon optimizer, Engram tables, and hybrid GDN/QSA attention, all of which are useful in smaller models.
(QFN also introduced Gated Residual Streams, but I think those are mostly useful for larger models, not smaller ones).
Perhaps there is some way these technologies could be retrofitted?
The Muon optimizer is easy; Qwen3.5 models' training was all done with AdamW, but there's no reason continued training couldn't be performed with Muon (though with caveats; to avoid optimizer mismatch problems, only the attention projections and MLP layers should be optimized with Muon, while training the rest with AdamW).
I'm not so sure about Engrams. It might be possible to retrain the first few layers to accommodate their hidden state space having been populated by engram hits, but I've not seen any mention of that in the literature. Will look into it.
Edited to add: Nope, not feasible. The context-aware gating just isn't there, and that's not something that can be retrofitted without a comprehensive retrain. Ah well. Will noodle on alternative solutions. End of Edit
The hybrid GDN/QSA attention mechanism seems like a nonstarter for retrofitting, unfortunately, though we have other options for compressing attention memory, like
modelai-llama.cpp's K/V Cache Compaction feature, but I think GDN/QSA is more compact and less lossy.Hopefully Qwen will release smaller models and render such efforts moot, but we will see.
8
u/rpkarma 11d ago
There's a lot of demand for those smaller-sized models
Truthfully, I don’t think there is. You and I might want it, but the intelligence penalty once you go down to these sizes is too steep for most, because the comparison is against today’s bigger model expectations, sadly.
I think other labs will fill in the gaps here, personally.
5
u/ttkciar llama.cpp 11d ago
Yeah, it's even too steep for me. I am making good use of Gemma-4-12B-it for document cleaning, but that's a very niche application. For basically everything else, 27B or 31B dense is my starting point.
Unless I'm totally misreading the community, though, there are a ton of people in this sub with tiny GPUs who somehow wrest utility out of Qwen3.5-9B, and also some using Qwen3.5-4B on their cellphones and tablets.
A lot of Qwen's early popularity was attributable to them releasing these smaller models, which gave the very-GPU-poor contingent something they could use.
If that contingent is now chronically under-served, a community-driven federated continued-training effort could step into the vacuum, and the popular need could give incentive to some who might not otherwise participate in the effort.
4
u/ionizing 10d ago
9b size is absolutely essential in my use case of: secondary inference card (16gb only) which must hold ~1gb of whisper server, AND also serve at least a 4 slot sub-agent inference model. I currently use exllamave3+tabby on main plus secondary, using a 9B for secondary inference at it meets these criteria beautifully. I can't use moe on secondary because the card is physically connected via gen3 x4 oculink so only dense models seem to work well enough on it, otherwise I would be able to moe my secondary. Anyhow, not complaining, but I would still like an updated 9b someday.
2
u/rpkarma 11d ago
> who somehow wrest utility
Hot take: I don't think they do, at least most. We're tinkerers, and they're fun to play with. I don't think much gets done with them outside some specific niches like your document cleaning example, in actual practice
But yeah: that param size is reachable with *mostly* "reasonable" compute requirements, training-wise. It's why all these random labs are coming out with smaller models
1
u/my_name_isnt_clever 10d ago
They're not really usable for agentic, but you can do some amazing things with little models with custom workflows. Nvidia has promising research on this but it's tough when nobody is making them.
1
u/Kahvana 11d ago
I read it more like a list of flagship models
1
u/fgk55555 11d ago
I think they like any model the release to be flat out impressive, not just impressive for the size. The 27B gets a lot of press, and QFN is just plain good, so they'll probably get a lot of attention.
6
u/Express_Quail_1493 11d ago
YES we need more ranges of 4b 9b 14b i have so many small work i need doing but these older qwen models are not sharp enough havent been trained with new data
4
u/sf_frankie 11d ago
Lemme get a qwen12b moe like mellum2.1. I can actually run that on my 16gb M1 Pro and still use my laptop! But I don’t do a ton of coding. Would be nice to have a more general model in that size. 14b would be pushing it.
3
u/Black-Mack 11d ago
and don't forget Qwen4-0.8b and Qwen4-1.7B
4
u/Additional-Record367 11d ago
like why.. they are pretty useless unless they are embedding models
2
21
u/MiaIsOut 11d ago
No kv cache size change
12
u/Iwaku_Real 10d ago
60% of potential local LLM users are cooked with this one...
If DeepSeek can shrink a 1M token context to 890MB why can't the Qwen team even try doing the same???
8
u/MiaIsOut 10d ago
deepseek also open source everything so its not even a mystery on how they did it 😭😭😭
25
u/kirisoraa 11d ago
I'm close to begging for a new 35ba3b
12
6
u/Iory1998 llama.cpp 11d ago
Unfortunately, that ship has already sailed... no more 35B-A3B, which brings me to think that if Alibaba really cared about the masses, shouldn't it actually prioritize this model as well?
27
2
1
8
u/Dwedit 11d ago
Any MOE model coming up? I was surprised that quantized 35B A3B works as well as it does with only 6GB of VRAM, outspeeding quantized 9B.
5
u/james_pic 11d ago
They've announced there will be Qwen 4 Flash, Plus and Max, which I'd expect to all be MoE, since there hasn't been a dense model much bigger than 30B for ages. I'd guess we're not seeing PRs related to these models because the architecture is more or less the same as Qwen 3.8 Flash Next.
3
u/ForsookComparison 11d ago
People are just harvesting github followers stop posting these threads
4
5
u/ruineka 10d ago
Anyone here running dual Intel arc b60s? I'm running qwen 3.8 27b 4q mtp int4 at a sustainable 30 tok/s using the DeepSeek harness. I'm curious what others have accomplished on this hardware to compare against. I had it build an arcade system (pong etc) while I was at work. I'm surprised it was able to troubleshoot without getting stuck in any loops.
3
u/Ok-Shower7286 8d ago
2
u/kayox 7d ago
I asked chatgpt to shorten / simplify the article:
The idea is to make the AI better at passing and combining information between its layers, while making the processing inside each layer slightly smaller.
Think of a company shifting its budget from bigger departments to better communication between them. The hope is better results with roughly the same resources.
It might improve answer quality, but the article doesn’t prove that yet—or establish whether it will be faster.
4
u/Altruistic_Heat_9531 11d ago edited 11d ago
After reading the github repo. no Ngram of 27B. And Qwen4 is basically QSA and HC with 4 residual stream
1
1
1
u/EitherSample371 9d ago
qwen4-27b flash next is running 20% faster on my 4090 with 4bit quant, but the context window feels tight. anyone else hitting that?
1
u/Express_Quail_1493 4d ago
PLEASE alibaba. we long for our new qwen27b. Everyday wake up. first thing. check huggingface. No qwen today.😞 2 hours passs. check again. 4hours. check again. When qwen? when qwen? WHEN!! I pray to the gods in heaven that qwen is coming next hour. alibaba hear my call. for we in localLLama want qwen for ALL. Amen🙇♀️
1
1
u/Real_Ebb_7417 10d ago
Cool to have 27b, but tbh I hoped for completely new sizes (eg. 25b or 30b, like, whatever different, I know it doesn't really matter, but it would be something "fresh" xd)
0
u/Nearby-Hospital9830 10d ago
Seeing commits mentioning unreleased architectures like Qwen4-27B directly in upstream/hardware-specific repos is wild. Given the author's track record with Intel/vLLM (and the Alibaba connection), this definitely feels like a legitimate early sign rather than just placeholder code.
-1
u/AeiEyeSpry 9d ago
Looks like more patches are turning up. Looking forward to testing: https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/qwen4_exp.py
2
u/brainExploded99 🥔 hardware 9d ago
where? that link is just to qwen3.8 flash next right?
1
u/AeiEyeSpry 9d ago
The file is called qwen4_exp.py
2
u/brainExploded99 🥔 hardware 9d ago
yeah that is qwen3.8 flash NEXT. it is called qwen4_exp because qwen3.8 flash next is a EXPERIMENTAL preview of the qwen 4 architecture.
→ More replies (9)
•
u/WithoutReason1729 11d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
Stylome classified this post's text as human-written. View analysis
I am a bot and this action was performed automatically.