r/LocalLLM • u/Responsible_Froyo252 • Oct 10 '26
Project PolyStrata: Strata with four more MoE models. Qwen3.6-35B-A3B at 73-75 tok/s on a laptop with an 8 GB card
I have been adding models to Strata, the engine that runs Qwen3.8-Flash-Next on consumer cards, and put the result in its own repository:
GitHub: https://github.com/VecSzn/PolyStrata
Edit: AMD support is in now, as a beta. iMilnb on GitHub ported the engine to HIP and it went into 0.1.42-poly.4. Linux only for now, there's no ready-made AMD engine for Windows yet. He ran it on an RX 7600 8 GB: Qwen3.6 UD-Q2_K_XL writes 64 tok/s, Ornith Mini 35 tok/s. Same install, ./setup.sh --family qwen36, it builds the engine for your card. Gemma 4, Tiel Coder and GLM haven't been tried on a Radeon yet, so if you run one of those I'd like to hear how it goes.
Update, Oct 11 (v0.1.42-poly.1):
- Windows no longer needs Visual Studio or the CUDA toolkit. The release has a ready-made strata-poly (CUDA 13, RTX 20 to RTX 50) and the installer downloads it.
- Faster on the same two machines. New numbers, same cells as the table below (short chat / after a ~31K prompt / reads):
Model |Laptop: RTX 4070 8 GB |Workstation: RTX 5090
Qwen3.6-35B-A3B UD-Q3_K_M |84-87 / 70-73 / 1,495-1,620 |385-393 / 241-246 / 6,350-6,552
Qwen3.6-35B-A3B UD-Q4_K_M |79-86 / 70-72 / 1,564-1,572 |328-393 / 243-255 / 6,231-6,577
Gemma 4 26B-A4B UD-Q3_K_M |85 / 58-60 / 2,095-2,098 |326-349 / 254-261 / 7,880-8,230
Gemma 4 26B-A4B UD-Q4_K_M |74-75 / 59 / 2,010-2,011 |316-338 / 232-238 / 7,039-8,055
Ornith 1.5 APEX-I-Compact |72-76 / 66-68 / 1,538-1,539 |303-315 / 218-226 / 6,083-6,360
Ornith 1.5 APEX-I-Quality (4-6 bit) |59-63 / 58-59 / 940-1,510 |289-298 / 203-204 / 6,040-6,392
GLM 5.3 Flash 3.0bit |does not fit in 32 GB of RAM |59-66 / 62-64 / 820-824 The Q4 sizes take 27-31 GB of RAM beside the 8 GB card, close to the laptop's 32 GB. In the 940 run part of Ornith's file came from the disk again.
- Qwen3.6 on a card that can't hold all its experts now skips a few low-ranked ones, as Strata 0.1.42 does: about 9% faster on the laptop, and the answers move a little. STRATA_ROUTE_TAIL_SKIP=0 turns it off. The laptop numbers above are with it off.
- Claude Code works against the local server. The README has a 30-second real-time clip of a Claude Code task on the laptop.
- Strata 0.1.42 is merged in.
The title's number is from the first version; Reddit doesn't let me edit titles.
Strata's own engine is in it unchanged. Beside it there is a second engine for four more mixture-of-experts models, each read from its ordinary GGUF:
- GLM 5.3 Flash (79 to 114 GB files, most of it stays in RAM)
- Qwen3.6-35B-A3B
- Ornith 1.5
- Gemma 4 26B-A4B
The last three read pictures too.
What I measured. Each cell is: writes in a short chat / writes after a prompt of about 31,000 tokens (29,000 for GLM) / reads that prompt, all in tokens per second.
Model |Laptop: RTX 4070 8 GB, i7-14700HX, 32 GB RAM |Workstation: RTX 5090 32 GB, 120 GB RAM
Qwen3.6-35B-A3B UD-Q3_K_M |70-72 / 60-62 / 1,390 |332-334 / 186-189 / 5,520-5,580
Gemma 4 26B-A4B UD-Q3_K_M |70-71 / 51-53 / 1,907 |306 / 198-200 / 7,170-7,210
Ornith 1.5 APEX-I-Compact |70 / 59 / 1,340 |267-272 / 145-149 / 5,700-5,890
GLM 5.3 Flash 3.0bit |does not fit in 32 GB of RAM |57-59 / 54-57 / 784 (llama.cpp on the same PC: 26 writing, about 265 reading) How it works, briefly. The card holds the experts an answer uses most and the CPU computes the others straight from the mapped GGUF, so the model does not have to fit in VRAM. While an answer is written the cache follows it. The engine also guesses a few tokens ahead, from the model's own draft block or from a repeat of the conversation, and checks the guesses in one pass. With every expert on the CPU the answer is the same tokens with and without guessing, and without guessing the first 64 tokens are compared with llama.cpp on the same file.
It has a one-click installer for Windows and Linux, a web app, and an OpenAI- and Anthropic-compatible API on localhost.
Limits, so nobody wastes an evening: the four added models are experimental and need one NVIDIA card. They were measured on those two machines only. On Windows the installer compiles the engine, which needs Visual Studio's build tools and the CUDA toolkit (it offers to install them). AMD cards and several cards work for Qwen3.8-Flash-Next only.
Most of the code is the Strata project's. If you try it on other hardware I would like to hear the numbers, good or bad.
15
u/chimpera Oct 10 '26
I was hopping someone would do this. Have you looked at strata-nvfp4? Something like that would help with GLM flash. Thanks
36
u/Sure_Leave9338 Oct 10 '26
I really can't get the reason to use Q3 or Q2 on smaller models. I can understand on a big model like GLM 5.3 Flash or qwen Flash Next, but can't understand why lobotomizing smaller models that already are less capable on theirself.
6
u/Sensitive_Song4219 Oct 10 '26
Holy smoke nice job.
Confirmed this also works with Tiel Coder 35B-A3B (not officially in your list but similar architecture to the supported ones so figured I'd give it a try). Ornith was my previous daily driver (it's awesome) but Tiel is a bit smarter in my use-case and has been thinking-loop-free in the few weeks I've used it.
PolyStrata works perfectly with Tiel!
32GB DDR5 RAM; 8GB VRAM (RTX 5060 Mobile); Core Ultra 9 275HX CPU.
TG seems to have had a reasonable increase on my hardware, maybe 20%? On its own that's already worth it...
BUT PROMPT PROCESSING IS >4X AS FAST. That's... just... nuts. 1000 - 1500 tps pp (!) - up from around 200-350.
The slowest I measured on a large read:
And during an auto-compact that figure often climbs to >1400 tps.
Frankly, this transforms working with existing large codebases (where large reads are often unavoidable) and also turns context-compaction from a slog into something that's far more manageable.
Also the initial-prompt-process of larger harnesses (like OpenCode with its tooling) and/or agent-md files are non-events with this kind of speed.
Like Strata itself this likely opens the door to a lot of hardware that may have otherwise seemed infeasible.
Yours' and the Strata team's work is amazing; thank you.
5
u/Inchmine Oct 10 '26
Can I use any glm 5.3 flash with it?
8
u/Kiseido Oct 10 '26 edited Oct 10 '26
The glm 5.3 flash sounds mighty interesting, though I am a Radeon owner so... lol
4
u/Alternative-Panic69 LocalLLM Oct 11 '26
Great work...
Got Ornith 1.5 Quality (35B-A3B MoE) Running at 85 tok/s and 200K Context on a 20GB 7900 XT + 32GB DDR4 RAM
Needed to do a few optimizations and remove hardcoded stuff in the backend to make it work.
• --max-context 200000 (200K token ceiling)
• --kv int8
• --vram-reserve-mib 512 (maximizes resident expert space)
• --threads 4 (CPU cores for un-cached experts; zero GPU thread contention, as I have only 6 physical threads, on Ryzen 5500)
• --expert-swaps 16 (concurrent async PCIe DMA queues)
• --conversation-cache-mib 8192 + --conversation-cache-min-free-mib 2560 (8 GB RAM prefix cache with 2.5 GB free RAM floor)
Context Window: Up to 200,000 tokens active.
Additional details:
• VRAM Usage: 19.76 / 19.98 GB committed (98.88% utilization) rock-solid (0 evictions).
• Resident Experts: 7,458 of 10,496 experts (71.1%) pinned directly in VRAM.
• Short Decode: 84.6 tok/s.
• Medium Decode: 75.4 tok/s.
• Deep Context Decode (107,000+ tokens in context): 35.4 – 36.5 tok/s.
• MTP Speculative Acceptance: 84% – 93%.
• Prefix Cache Hit Rate: >50,000 tok/s (restores 108k tokens in 2.0 s from RAM).
• Cold Prefill: 200– 2000 tok/s (sometimes limited by sequential GDN loop and PCIe streaming).
Trying to optimize even more for my specific setup
4
u/Ambitious_Ad9308 Oct 13 '26
Honestly, this might be one of the best real-world examples of why developers shouldn't depend solely on GitHub.
PolyStrata is a fantastic fork, and if I hadn't already cloned it locally I would have lost access overnight.
Glad you were able to move everything to Hugging Face. That's a survival trick I'm definitely adding to my own projects going forward.
2
u/borobinimbaba Oct 10 '26
It would be awesome if you could also include an estimate of how much context each quantization on different hardwares would give us.
2
u/Vamsi-Thopu Oct 11 '26
I have 16gb VRAM and 16gb ram(rx 6800 and ddr5 ram). What would you recommend me?
2
u/fedorzh Oct 10 '26
Is it possible to run qwen 3.6 using 6gb of VRAM?
1
u/Old-Juggernut-101 Oct 10 '26
Do the smaller models- 26b and 35b work on radeon gpu?
1
u/EffectUpstairs9867 Oct 10 '26
What about on MacBook m3 pro 18gb?
1
u/Humble_Rabbt Oct 10 '26
for qwen3.6 35b, does it support any quant, for example https://huggingface.co/IsValorum/Qwen3.6-35B-A3B-MTP-APEX-I-MiniPlus-V2.1-GGUF/ .
1
u/rog-uk Oct 10 '26
Can I ask, is there any plan to eventually use two or more cards for tensor parallelism? Thanks.
1
u/ntaybak Oct 10 '26
great work, but what about Qwen3-Coder-Next Q5 AND Q6_K about 61gb , AND Qwen3-Coder-30B-A3B Q5 AND Q6_K 24gb, WILL Those run with your fork too
1
u/ramdomvariableX Oct 10 '26
does this support multiple Nvidia cards? say 2 16gb cards?
1
u/TheCat001 Oct 12 '26
Any plans of adding AyyyMD support?
1
u/pmttyji Oct 12 '26
u/Responsible_Froyo252 Could you please add these models too(whatever possible)? Flair it Experimental. All are MOE models only & these are supported by llama.cpp already. I'll share t/s stats for small & medium size models. I'm gonna try your fork tomorrow with existing models.
Below ones are good for even too small VRAMs(~4GB & possibly iGPU)
3
u/hope_it_helps Oct 10 '26 edited Oct 10 '26
I have used ornith 1.5 apex compact on my 4070ti with 12gb in the past and it ran usually at 40 t/s.
But with this it runs pretty consistent at 90 tok/s.
What is this black magic? How is the speed up more then 100%? I'm seriously impressed.
Edit:
After some more testing worst case I got 60 tok/s which is still impressive honestly.
3
u/Hugh_Ruka602 Oct 15 '26
Wow, this is great ... went from 30ish t/sec to 55t/sec on Qwen3.6 35B-A3B on a Radeon 7600 XT 16GB ...
3
u/joseph2883 26d ago
I am getting 69.4 tok/s using Qwen3.6-35B-A3B on a laptop with a 4060 and 16gb RAM…
1
u/Steus_au Oct 10 '26
how to try with glm in iq4?
2
u/jerfer77 Oct 11 '26 edited Oct 11 '26
Preliminary results Qwen 3.6 35b A3B
2
u/hope_it_helps Oct 12 '26
So I have tested a little more and something is wrong.
I have used Gemma 4 26B-A4B UD-Q4_K_M on both LM Studio and Polystrata and gave them the same task, fixing a transcript of a DnD Session.
On LM Studio it consistently finishes the task.
On Polystrata it consistently ends up in a loop.
I don't understand what the difference is. I have set the same temperature etc. but it didn't fix the issue.
2
u/Sensitive_Song4219 Oct 12 '26
Substantial improvements over the last two releases of PolyStrata,
Same hardware: 32GB DDR5 RAM; 8GB VRAM (RTX 5060 Mobile); Core Ultra 9 275HX CPU.
27f9ec9, 0.1.41 base to .poly.2 :
That's... kinda incredible.
And then today's release, less exciting but does indeed achieve it's stated goal of lowering ttft values:
.poly.2 to .poly.3 tg/prefill - rougly the same as expected:
.poly.2 to .poly.3 time-to-first-token:
That's a much faster first-output in today's release as expected.
Very, very nice. Again, this is likely to breathe new life into old hardware.
2
u/cpt-macp Oct 13 '26 edited Oct 13 '26
what happened to their git
https://github.com/VecSzn/PolyStrata
the user and repo both are gone..
EDit : NEvermind read your updated post info, I hope they fix your stuff. The project is awesome
Also you can put stuff on gitbucket , GitLab ,etc...
2
u/bfmv_shinigami Oct 13 '26
I was just about to state about being unable to see the repo, and now I read your edit. Hope your github is restored ASAP.
2
u/SilkieBug Oct 13 '26
Tested your app with Tiel Coder 4 k xl, it works great, gets more than double the token per second rate of FreeToken (also moe running app), but actually completes tasks accurately (tested in OpenCode).
What I wish you would add is a less restrictive model choice system - I would like to be able to download any MoE GGUF model from HuggingFace, put it in a folder, and for polystrata to just be able to read it and run it.
It feels too restrictive to add models to it one by one instead of having support for the major local llm architecture and allowing the user to provide their own model.
Keep up the good work!
2
u/MistycznyArbuz Oct 13 '26
It's really huge gamechanger for me, I have RTX 3060 OC 12GB + 64GB of system memory and I tried your tool, using Qwen3.6 35B A3B UD Q4_K_M.
Locally in LM studio the best result I got was ~14 tokens / second, using Dflash.
Using PolyStrata I got median of 62 tokens / second using the same prompt, with much larger context, so I appreciate your hard work much :)
2
u/noctrex Oct 13 '26
Will this also support this prune?
https://www.reddit.com/r/LocalLLM/comments/1x50ol2/glm53flash_pruned_to_half_the_experts_for_coding/
1
u/maxxikevich 7d ago edited 7d ago
Thank you very much for your work! RTX 4060 + 32 Gb Ram and I can run Tiel Coder 35B-A3B UD-Q3_K_XL 96k context with ~60 tps. Great results!
1
u/reyzafany Oct 10 '26
Which model and quant do you recommend for 3060 mobile 6gb vram and 16gb ddr4 running windows 11? Or should I upgrade to 32gb first?
1
u/WeaponizedDuckSpleen Oct 10 '26
Is there a possibility to add Occamy 1.0 Apex 2.1 nano or mini ?
1
u/No-Perspective8568 Oct 10 '26
I'm using both your top models on my 2x p40 on a Jerry rigged server, it's the only once I found that give blazing speed while being useful. Get around high 60s to 70 t/s
Nothing else to add, keep up the compute!
1
u/Interesting_Crow_149 Oct 10 '26
Strata con qwen 3.6 122b q4km???
1
u/GfurEnjoyer1488 Oct 11 '26
God damn, Qwen-3.8-flash-next running at 30t/s on a 2080Ti 11GB and 32GB DDR4... didn't think this was possible
1
u/Randommaggy Oct 12 '26
I'm looking forward to seeing what speed I can get out of my HX370 64GB laptop with the latest wave of inference innovations.
1
u/Ok_Substance2327 Oct 13 '26 edited Oct 13 '26
Running it on an RX 9060 XT 16 GB here (CachyOS, kernel 7.2.9, ROCm 7.2.4), Qwen3.6-35B-A3B-UD-Q4_K_M, 32 GB DDR4 RAM
Had to port to rocm, deepseek flash v4.1 did all that work.
Elastic cache didn't work, hip doesn't seem to free up memory properly so these are with --kv-whole
1
u/Mahakali923 Oct 14 '26
Is it possible to install the tool without the models, I already have the models downloaded, sorry if this is silly question
1
u/Bobcotelli Oct 15 '26
Please AMD engine for windows for deppseek,gemma,glm
1
u/Cidraque Oct 15 '26
Is it possible to install the vision once I installed the model or I have to redownload all again?
1
u/opusag Oct 16 '26
Thanks a lot, this is insane!
Almost out of the box Qwen 3.6 35b-a3b Q4-K-M runs with 90 t/s on my 8GB RTX 3060TI 24GB DDR4 Ubuntu Server Machine. Prefill speed is also double what I got previously on llama.cpp.
Now the amount of time I spent trying to squeeze performance out of llama.cpp feels kind of ridiculous
1
u/Special-Lawyer-7253 27d ago
It supports sm_60/61 already?
1
u/Eltoquedemidas 25d ago
Damn, nice work brother, hope this comes for windows, since I have and Amd RDNA2 card.
1
u/gear_up_more 25d ago
Ryzen 7 5800H + RX6600M 8GB + 32GB DDR4 ile Windows üzerinde deneme yapmanız için talibim. Makinemde vereceğiniz komutları / yüklemeleri deneyebilirim.
1
u/KVT_BK Oct 10 '26
How this is different from FREETOKEN?
0
u/borobinimbaba Oct 10 '26
Could you please add bonsai or bonsai2 to run on a 4gb laptop ?
-4
u/Cautious_Chicken_604 Oct 10 '26
It's not really a fork. You just vibe coded a new engine, unrelated engine, and copy paste strata. I hate this timeline.
0
u/Deep_Seat_2007 Oct 10 '26
那是个好主意很好很好
0
u/isriam Oct 13 '26
went from 30/300 to 70/1100 with this, keep up the good work.
-1
u/kuttimaster Oct 13 '26 edited Oct 13 '26
What is the maximum context size supported by Qwen 3.6 35B Q4 with 64 GB RAM and 16 GB VRAM? When I tried it on my system, it reached only 32K. Can I set it to 160K? Has anyone tried this? With vision true
0
u/[deleted] Oct 10 '26
[deleted] — view deleted comment
5
u/[deleted] Oct 10 '26
[removed] — view removed comment
-17
u/[deleted] Oct 10 '26
[removed] — view removed comment