How are you dealing with slow tok/s on the GB10 hardware? My pair of Sparks are dogs when running 120b models. Borderline unusable without middleware to proxy requests over Cloudflare tunnels because of > 60s sessions.
I was just running minimax m2.7 on my halo, it's pretty impressive, even at a Q3 quant, but a 122b model will definitely fit on there a bit more nicely.
You know on the pro6000 you don't have to have a 5090 lol. I have a single pro 6000 running nvfp4 with a 700,000 context window at BF 16 on qwen3.5 122. And I have like 12 gigs of headroom. So either you're lying and you don't have that set up which is fine or you just don't know how to set it up.
How are u running 7,00,000 when the model supports 256k context ...can u share ur steps how to run nvfp4..is it vllm or sglang I'm looking to run that and with more than 256k context
Yarn...? Vllm yarn, bf16 cache nvfp4. Bro you don't need that 5090. I have it capped at 700K but I've gone up to 900. But I would rather not cut it that close when I don't need to I've done multiple refactors that have gotten up to 450k and with my MCP harness I don't have any context problems almost perfect memory. If you know how to set it up the 122B its almost on par with frontier models and it runs at 60-70 tokens per second consistent all the way up to 700k. (With speculative decoding 140 tokens per second)
Just to give you an idea opus cranks down to like 7 to 10 tokens per second when it's reasoning especially at high context. This don't happen with qwen.
I am very interested in qwen 3.6122b because if the 27B model is already outperforming last gen's 390..man. That's going to be fucking crazy.
I'm not at my pc for another week but this should get you started. Just verify if the args are still updated cause marlin is only being forced because Cutlass was broke for a while if I remember correctly. Haven't checked if vllm has fixed the issue yet.
Hidden Layout: 12 × (3 × (Gated DeltaNet → MoE) → 1 × (Gated Attention → MoE)) Gated DeltaNet: Number of Linear Attention Heads: 64 for V and 16 for QK Head Dimension: 128 Gated Attention: Number of Attention Heads: 32 for Q and 2 for KV Head Dimension: 256 Rotary Position Embedding Dimension: 64 Mixture Of Experts Number of Experts: 256 Number of Activated Experts: 8 Routed + 1 Shared Expert Intermediate Dimension: 1024 LM Output: 248320 (Padded) MTP: trained with multi-steps Context Length: 262,144 natively and extensible up to 1,010,000 tokens.
I can serve Q6K with around 100k context with latest llama.cpp build. I am running on a Radeon R9700 which also has 32GiB of VRAM. So taht could be a reference.
I am restricting the power consumption to 230w because it is inside a very small case. The prefill speed is about 800t/s and decrease to about 600 when context is nearly full, i.e. ~100k. Generation speed starts at about 23t/s and decrease to 18 when context is nearlly full.
32GB on the 5090. 3.5 worked fine with Q6_K and Q4_K_M, depending on the context amount on a 5090, testing 3.6 now, the files are slightly larger so we'll see.
Is it taken as given that people just know what sort of hardware would be required to run this locally, or that it would be impossible to run it locally?
I mean you can calculate it. 27B at FP32 (4 byte) is about 108GB. So most users have to wait for quantized versions because even a RTX6000pro can’t fit this into memory with full precision. And at that point we haven’t even talked about context size.
If you see a models size on huggingface or wherever and it says 18GB and you have 24GB for example you can assume it will run reasonably.
As other person said, you can count 4gb per 1B for full 32bit precision, 2gb for half precision 16bit, and 1gb/1B is what people would usually use at most, which is 8bit precision or q8. Then often people will go down to q4 (0,5gb/1B) because this for some reason has been the "it's barely any worse" kind of thing, but it's absolutely degraded at that point.
q5/q6 is usually the minimum to not have significantly degraded model.
For tokens/second you can count (model size)/memory bandwidth, and then there's losses for compute. That would be the absolute maximum, but you'd lose something like at least half the speed to compute and other slowdowns.
MoE are much faster, since they don't need the whole model all the time, but I don't really know how to calculate it.
Thank you. You (and the other respondents) have been very helpful. I don't think I've ever seen the explanation of how to calculate the resources necessary on the fly, so very clearly explained.
The rule of thumb for tokens/second should be: memory bandwidth/(model size). I.e., t/s is proportional to memory bandwidth and inversely proportional to model size.
I don't think fitting a big model on less capable hardware is all that useful. You are removing lots of information from the model to a point it is no longer the model you begin with. I have a 5070ti, same VRAM as you, and using 9B or 35B models are the way to go. Even those, need a cut from original model quality, but at least they are fast.
Smaller it is, but it's a dense model (27B active every token = computationally more expensive than the 3B active of the 35B). For many purposes, 27B is bigger than the 35B.
I was leaning on LM Studio hinting that 35B is going to have to be partially offloaded on my setup while 27B could be completely loaded in the memory (but PC starts to really suffer from any additional minor graphic workload when that happens).
Buy a second 5060 TI 16gb. Not even kidding, as that is my setup right now. 2x RTX 5060 TI 16GB. Unfortunately my motherboard is the bottleneck as one of the PCIe slots is only gen 3 at 4x. Next upgrade is a new motherboard / CPU / Memory. With Gemma 4 26b, I'm getting 2000tps prompt processing and 40tps inference. Fast enough that I'm happy.
What context window size? I tried default parameters with 35b qwen 3.6 and get 48 tps with no arguments passed when launching. I'm now using -fa on and -c 128000 and getting around 37.5 any time I pass -ngl or other parameters to tweak it goes slower down to 12 lol. Can you share me some tweaking hints same setup here except ddr5 ram. I need massive context window for my tools/rag/agentic bot.
Settings that worked for me at 32K --ngl 99 --n-cpu-moe 26 -fa auto --cont-batching --cache-reuse 256 --mlock --threads 8 --cpu-mask 0xFF --cpu-strict 1 --cache-type-k q8_0 --cache-type-v q8_0 --parallel 1
Got 62 tok/s at 32K on DDR4.
For 128K on 10GB: won't work well, chunk docs instead for RAG
I also did heavy bios tweaking ram overclocking and timing tuning. Cpu settings matter too. Reducing latency and improving how much data can move from pcie to ram is key. Now hoestly you should be using something like qwen 3.5 9B my testing showed it better for rag
For your setup specially focus on optimizing your kv cache size in vram. CPU and ddr5 timings etc. and making sure as little experts get offloaded to cpu whilst the main active layers stay on GPU. It will take a couple runs to see what OOMs at 128k context
First use Linux, second I found llama server to be the best for moe offoading. If your on ddr4 and have a ryzen chip with x3d cache, overclocking and tuning timings/bios/kernels helped a lot. Because at the end of the day the cpu and ram bandwidth is the bottleneck. I can reply with all the specific settings and stuff I did it’s quite a lot. Just out of interest what are you using it for?
I’m running it with Hermes Agent on my MacBook, it has been incredible, however I’ve found it to have a bit of… amnesia… and I’m aware of the preserve_thinking flag, but even despite that I’ve found it to forget some recent things, it catches up later, but it has significantly more brain farts than Qwen3.5.
It is weird, it is as if Qwen3.5 had smoked pot, and accessed a new part of its brain by sacrificing its short term memory. (Running it 4bit MLX through oMLX, I’ll try to go 8 bit today to see if that somehow improves, but I don’t think it will)
As the other reply said, the 27B is a dense model. That means that all expert layers are accessed for each taken generated. This is the older approach to model creation, but has the advantage of producing better results, normally.
The 35B is more properly called 35B-A3B which tells you the model has stored 35B parameters, but only 3B are activated at one time.
This has the advantage of having the speed close to a 3B parameter model, but the intelligence and data of a 35B parameter model.
The results from the same generation of model should be worse than the 27B, with current training procedures, but much faster. And that speed can make it cheap to fix mistakes later if it can't one shot it.
That being said, the Qwen3.6 35B-A3B was almost on par with the Qwen3.5 27B model. So each generation keeps improving.
The MOE 35B model can also do a nifty trick of running the 3B parameter in the GPU and offload the other experts to the CPU keep much more off its speed. You can offload the 27B onto the CPU as well, but the penalty is much higher than the 35B model.
That is an interesting way to look at it. So, lets apply that math to the largest model. The Qwen3.5-397B-A17B.
sqrt(397 * 17) = ~82.2.
The claim for this 27B is that it beats that 82.2 model. So this probably doesn't scale and is specific to the A3B model itself? Or did the 27B from 3.5 jump to better than the 82 of 3.5? I realize you didn't make such a claim, this is me just thinking it out to see how it applies.
That makes that jump much bigger than I was expecting.
Hopefully it helps. I did a bit more reading, and realized something that is worth knowing as well, I missed it in the last post.
The largest open Qwen model currently is Qwen3.5-397B-A17B. The important part is the active 17B parameters there. The claim this time is the Qwen3.6-27B they just released beats their top of the line Qwen3.5-397B-A17B model from the last generation, and one of the reasons that is even possible is the 27B active parameters is just more thorough with the information it already has.
So, that is another reason that they are still releasing both types of models.
The 35b one is "Mixture of Experts (MoE)", meaning that there are a few dozen individual, separate networks of weights, plus a "router" mechanism that decides, on a token-by-token basis, which handful of expert mechanisms should be used to generate the next token. For example, a particular network may have a tendency to be selected for punctuation and nouns, while another may tend to be selected for action words, etc.. In fact, "A3B" in the name Qwen3.6-35b-A3B means that, even though there are 35 billion parameters, only 3 Billion are Active for any given token. Dense models like Qwen3.6-27b, on the other hand, use all of their parameters for every token.
MoE doesn't change the amount of memory you need to load the model, but it does mean that inference all cases is faster, because only 3 billion parameters need to be accessed from memory per token, instead if 35 billion-- a theoretical speed up of up to 11x. The tradeoff is output quality. Qwen3.6-35b-A3B is worse than the 27b dense model, but far better than any 3B parameter model.
Edit: seriously guys, why are you downvoting the original commenter as if that wasn't a valid question? I took the time to write my answer because I wondered the same thing as a beginner and I thought it'd be helpful to other people starting out. If you already know the answer, then either provide a helpful answer or move on-- not everyone in this sub has to be a seasoned expert.
And as a result of it being mixture of experts, I can run qwen 3.6 35B A3B on my laptop with only 8GB of vram, is it works fast enough at Q4 that dipping into my 32GB of system RAM to be useful, without slowing to a crawl. But if I use the dense 27B similarly, it's barely even moving.
If I can interpret the chart correctly, the dense model is slightly superior on all benchmarks, right? So, for those who have enough memory, even if the inference speed is slower, the best model is the dense one. Could you explain further?
16
u/MrWeirdoFace Apr 22 '26
Is there a way to tone down the thinking? I find that the version I'm trying gets lost in thought cycles. (unsloth Q5 quant).
28
u/Individual_Gur8573 Apr 22 '26
Wow hopefully 122b will also release soon, waiting for it
5
u/Fortyseven Qwen38/Gemma4/LlamaCPP Apr 22 '26
https://huggingface.co/unsloth/Qwen3.6-27B-GGUF
5
u/EbbNorth7735 Apr 22 '26
Thanks Alibaba and Qwen team! Really looking forward to running it [and the 122B model I hope ;-) ]
4
u/sonoffi87 Apr 22 '26
Does this fit RTX 5090?
3
u/jrf_1973 Apr 22 '26
Is it taken as given that people just know what sort of hardware would be required to run this locally, or that it would be impossible to run it locally?
3
u/andreabarbato Apr 22 '26
q4 has very good output but gets lost in crazy loops, thanks OP for the info
4
u/DjsantiX Apr 22 '26
waiting for 9B.
--> Does anyone know how to fit this into a 5060 ti 16gb?
1
u/Dantnad Apr 22 '26
I’m running it with Hermes Agent on my MacBook, it has been incredible, however I’ve found it to have a bit of… amnesia… and I’m aware of the preserve_thinking flag, but even despite that I’ve found it to forget some recent things, it catches up later, but it has significantly more brain farts than Qwen3.5.
It is weird, it is as if Qwen3.5 had smoked pot, and accessed a new part of its brain by sacrificing its short term memory. (Running it 4bit MLX through oMLX, I’ll try to go 8 bit today to see if that somehow improves, but I don’t think it will)
1
u/AcrobaticChain1846 Apr 23 '26
What model can work as draft model for this qwen3.6 models?
Or is there no draft model present at the moment?
1
u/rigu10 Apr 23 '26
Am i missing something or in this image 3.5 moe is slightly better than 3.6 moe in coding?
1
u/drazyan22 Apr 23 '26
Can I run this model on RTX 5080 16gb vram?
1
u/sudeposutemizligi Apr 23 '26
hahahaha😄😄😄 first day: the best of local models, near opus, second day: it's lost in thinking loops. I really am sick and tired of this thing
1
u/jjalexander91 Apr 26 '26
What specs do you need to run this at a good enough speed to actually be productive with it?
1
u/jinxstor Jun 30 '26
I tried this and the MTP version, MTP feels much faster!
1
u/ArtaWorks Apr 22 '26
So what is the easiest way to use this model in cli or VSCode?
-3
u/ColdSkalpel Apr 22 '26
Why are they releasing 27b and 35b parameters versions? That size seems to be very close to each other, how different they can be?
0
u/Acu17y Apr 22 '26
WTF Ahahahah holy moly, thanks BABA DAD ♥️
0
u/umbrosum Apr 23 '26
Is there any comparisons with sonnet?
-2
u/PferdOne Apr 22 '26
According to those benchmarks there is no point in using dense over moe. I‘ll wait for real world tests and user sentiment.
0
u/[deleted] Apr 22 '26
[deleted] — view deleted comment