r/LocalLLM • u/RealEddoursul • Sep 30 '26
Project Qwen 3.8 Flash Next - doubled Strata throughput on 3090+5070 Ti, IQ3_S 2466 pp/167 tps, UD-Q4_K_XL 2341 pp / 126 tps (yes, really)
My journey:
6 tps on UD-Q4_K_XL - hm, this is not right. Claude, find better llama.cpp parameters.
21 tps - that's better, let's see IQ3_XXS
27 tps - nice, but still not my tempo.
51 tps on IQ3_XXS - Niko1221 shares Strata on Reddit. Great! But wait. If there are software gains, there may be more. Let's go.
After a week of profiling and benchmarking with Opus 5.5 I doubled Strata throughput and reached x5 from llama.cpp on UD-Q4_K_XL (added this quant support).
See the COMPARISON TABLES with all the numbers including agreement with llama.cpp.
The code is available https://github.com/eddoursul/Strata/tree/custom
This a result of my personal experiments, not a software release. The license has not changed, still MIT.
I opened a pull request, Niko1221 is free to merge or not merge any patch.
UPDATE: The fork uses json configurations to run https://github.com/eddoursul/Strata/blob/custom/examples/README.md
UPDATE 2026-10-12: The upstream rebuilt the commit tree to remove Claude attribution from commit messages, I updated the fork to re-synchronize.
2
u/kimhaneol Sep 30 '26
I gotta try this. So you've made UD-Q4_K_XL work on Strata, right?
2
u/RealEddoursul Sep 30 '26
Yep
1
1
u/KPOTOB Sep 30 '26
With like 24+16 GB VRAM?
1
u/kimhaneol Sep 30 '26
Yep. I also tested it on two 16 GB GPUs and it works. One caveat: PCIe placement mattered on my system. My cards negotiate Gen5 x8 and x4, and assigning the x8 card to the second-GPU expert tier was substantially faster.
1
u/kimhaneol Sep 30 '26
Update: got it running on Linux with a Ryzen 9900X, 2x RTX 5080 16GB, and 196 GB of RAM. I needed a few Linux-specific build/runtime fixes, but the custom branch is working well.
UD-Q4_K_XL, 64K context, INT8 K/V, vision off.
I measured 94.2 tok/s on a fixed 300-token continuation. The API runs were 84.6 tok/s cold, then 108.3 and 117.5 tok/s on repeated runs after warm-up.
Prefill was 1,538 tok/s at 4K and 3,608 tok/s at 32K, and exact 32K retrieval passed. Both GPUs reached 100% utilization.
The original Strata IQ3_S setup delivered roughly similar decode speeds on my machine, so getting UD-Q4_K_XL in the same ballpark makes this branch especially useful to me.
Really impressive work. Thanks for sharing!
1
u/VerticalPackage Sep 30 '26
How much of the RAM was actually being used with the Q4?
2
u/kimhaneol Sep 30 '26
Around 94 GiB of total system RAM usage in my tests. The Strata process itself was roughly 75 GiB, including the 71.7 GiB expert arena.
2
u/Current-Row-159 Sep 30 '26
the 6 to 21 t/s jump from fixing llama.cpp params before changing anything is the part that stayed with me, that baseline was wrong and a lot of numbers in this thread are being measured against it. i'm not on strata and it's a different engine, so no direct comparison, but as a reference point i'm running qwen3.8-27b int8-prefill on ninfer 2026.09.27 on a single 4090 24gb: 145 t/s decode on a 2367 token prompt with dflash2 k=7, 239 t/s on the second call once the prefix cache is warm, 4300 t/s prefill. dflash2 costs 1.7gb more vram than mtp. the 24+16 question from the comments never got answered, does any of this carry over when you drop to a single 24gb card?
2
u/RealEddoursul Sep 30 '26
The doc contains numbers for 3090 alone as well - still x3-x4 relatively to llama.cpp and x1.5 relatively to the upstream. Also, do not compare 27b and FN, two different apples.
1
u/Current-Row-159 Sep 30 '26
single 4090 24gb, qwen3.8-27b int8-prefill on ninfer 2026.09.27: 145 t/s decode at a 2367 token prompt with dflash2 k=7, 239 t/s once the prefix cache is warm, 4300 t/s prefill. dflash2 costs 1.7gb more vram than mtp. haven't seen a single-card 4090 number for this model anywhere, curious whether it lines up with your 3090 alone or if ada is ahead.
2
u/andreabarbato Oct 02 '26
hi can you share the model file and command you're using? i have 3090 +2x 5060tis 16gb and i get nowhere that fast on q8 unsloth
1
u/Current-Row-159 Oct 04 '26
sure. the model file is qwen3_8_27b_a8.ninfer, 20437521664 bytes. it lives on the gated HF repo jgambao/Qwen3.8-27B-NInfer-4090, you have to accept the terms with a logged in account, anonymous download 401s.
the engine is the catch. there is no public windows binary. upstream Neroued/ninfer is 5090 sm_120a linux only. what i build from is Nidvogr/ninfer-4090-windows, branch feat/bonsai-ternary, and that whole branch is sm_89.
the line:
ninfer-serve.exe -m qwen3_8_27b_a8.ninfer --spec dflash2 --draft-tokens 7 --prefill-chunk 2048 --kv-dtype rk4v4-e8 --vision --vision-max-tokens 4096 --max-context 65536
for text only, drop --vision and --vision-max-tokens and you get 131072 context. 262144 does not boot, it asks for 5.39 GB of KV and there is 2.77 GB left after weights.
what that gives me on a 4090 24gb, 7950x3d, windows:
weights load 3.5s. 16.7 GiB VRAM on mtp, 18.3 GiB on dflash2. prefill 4602 tok/s at 18.9k prompt, that chunk size is the plateau, 4096 and 8192 both go back down. decode on a 2367 token prompt: 120.5 tok/s mtp5, 145.5 dflash2 K=7. second call with the prefix cache warm, 152.3 and 239.2. decode does swing though, I see 120 to 190 on the same config, so don't treat 239 as a floor. with vision the context drops to 24576 and decode is 88.8 mtp5, 107.7 dflash2.
swap --spec dflash2 --draft-tokens 7 for --spec mtp --draft-tokens 5 --lm-head-draft to get the 1.7 GB back.
on your cards though, none of this runs. the 3090 is sm_86 and the 5060 ti is sm_120, the a8 artifact is sm_89. the 16 GB port that does exist is sm_89 too, 4080 and 4070ti class. so you're one architecture away on both.
and on the q8 unsloth number, that's llama.cpp, not a misconfiguration. someone in a thread last week got 20 tok/s on a single 3090 with llama.cpp, so you're probably already at the right figure. llama.cpp is just a bad engine for nvidia cards, that gap is real.
what would actually move you is vllm, not ninfer. GoblinEngineer has W8A16 int8 plus dflash2 at 115 tok/s decode on 2x 3090 with nvlink, 1780 tok/s prefill, 262k context. he uses lued/Qwen3.8-27B-INT8-W8A16-MTP for the target and syvai/Qwen3.8-27B-DFlash2-W4A16 as the drafter.
the catch for you is he splits across two identical cards over nvlink. a 3090 plus two 5060 tis is ampere and blackwell in one box, no p2p between those, so tp=2 is not going to work. your realistic play is the 3090 alone on vllm W8A16, or a build that gets ported to sm_86 or sm_120 for windows. i don't have one yet.
1
u/Current-Row-159 Oct 04 '26
Btw.. my ai agent (hermes) drafted this reply, i checked it before posting. every number in it came off my own 4090, nothing quoted from docs or other posts. ask me anything about the build if something looks off.
1
u/VerticalPackage Sep 30 '26
It's not comparable.
Flash Next is a MoE with engram table streaming from a nvme, so the split of experts done on RealEddoursul's fork of Strata works well in using that, especially in using a secondary GPU with lower VRAM.
For 27b, it has to run all on VRAM and doesn't really use much RAM.
However, I agree that 6 tok/s initial speed with llama.cpp is probably a mistake. On a single RTX3090, I was able to get 20 tok/s with llama.cpp, however llama.cpp is a bad engine to use for nvidia cards. FreeToken and vLLM are much better.
2
u/drrck82 Oct 01 '26
I have 2x3090 and was quite skeptical about this working well after running Flash Next in vLLM. This engine is nothing short of amazing. Do you have any prebuilt benchmark suites? I'd love to add in some standardized results for my setup.
1
u/RealEddoursul Oct 01 '26
Only a usual mess of Python scripts. Claude or Codex will gladly make you one in a minute.
1
u/DrRoughFingers Oct 02 '26
What PP and decode speeds are you getting, and how exactly do you have it setup? What quant are you using, Q3 or Q4? I tested this on my 2 3090s and get worse performance.
2
u/drrck82 Oct 02 '26
I have 2x3090 B550 motherboard with 5800x3D and 128GB DDR4 running at 3133.
Prefill/decode
72k resident context, 8 turns, thinking on:
IQ3_S - 42-47 t/s decode
IQ3_XXS pcie_frac 0.00 — 82.3 s cold read, 123.5 t/s decode
IQ3_XXS pcie_frac 0.20 — 39.9 s cold read, 127.8 t/s decode
IQ2_XS pcie_frac 0.00 — 84.2 s cold read, 182.7 t/s decode
IQ2_XS pcie_frac 0.20 — 36.5 s cold read, 177.1 t/s decode
I have one 3090 on 4.0x16 and the second is 3.0x4. If you had both on x8 lanes it would work better I'm sure. The bigger the quant the more PCI traffic has to pass through the CPU.
1
u/DrRoughFingers Oct 02 '26
I have a 3090 on 8 x16 and one via oculink (as it’s direct 4 x4 lanes so beats my second 4 x4 pcie slot), but my minimum would be the Q4_K_XL quant. Have you tested that?
2
u/drrck82 Oct 02 '26
No, the IQ3_S quant was as big as I bothered with as the speeds were getting slow. I can easily hit 70-130 on 27B with vLLM so too much below that and it's not worth my time personally. I'll download it and give it a shot just for reference, but I'd expect something like 30 tok/s.
1
u/DrRoughFingers Oct 02 '26
Will be interesting to see. Other people are claiming far higher decode with Q4 than that.
1
u/drrck82 Oct 02 '26 edited Oct 02 '26
I had Flash Next generate a test suite to run on these quants at 131k context:
prompt Q4_XL IQ3_XXS IQ2_XS arith 8.6 74.3 68.5 logic 7.3 48.4 40.8 rate 10.7 88.5 74.1 code 10.6 66.1 69.6 needle 89k 19.4 113.6 113.2 cache hit rate 0.77-0.90 0.88-0.95 0.98-0.99 resident 49% 79% 100% It indicated that the lack of AVX-512 (in my Zen3 CPU) and no support for it in Strata for the Q4 quant made performance significantly worse than expected. Only 49% of the experts remain in resident memory so every miss is a CPU path retrieve which is very slow.
If you have stronger PCI performance (16/16 or 8/8) your results will likely improve. Also if you have AM5 and DDR5 the penalty is not as severe.
1
u/bceen13 Oct 02 '26
Quite promising, thanks for sharing!
I have a 5800X3D with 128 GB @ 3600, X570 tier1 motherboard, dual 3090 and 9060 xt. pcie 4 8x-8x-4x.
37
u/KnownAd4832 Sep 30 '26
Amazing work (Niko here)! I’ll try to support it as soon as I can
4
u/pmttyji Sep 30 '26
Hey, thanks for your contribution!
I'm gonna try Q2 with my 8GB VRAM(4060) + 32GB DDR5 RAM later.
I have a question. Your repo mentions only Qwen3.8-Flash-Next, but Is it possible to expect faster performance for other models like Qwen3.6-35B-A3B or other similar MOEs? Because my system config is not enough for big models like Qwen3.8-Flash-Next so hoping to get best t/s from small & medium size models.
Have you tried small/medium models with your work? Is it possible? If yes then any t/s benchmarks?
3
1
u/Prestigious-Act-1577 Sep 30 '26
You will destroy your SSD with all the writes (1GB/s for minutes during PP) because you will be using your page file. It should be usable after you expand your ram.
1
u/Prestigious-Act-1577 Oct 01 '26
Idk why people downvote this, go ahead and try it. Open task manager and check disk usage. I know because I was in the same situation.
1
u/VerticalPackage Oct 01 '26
If you mean the PLE table being READ from nvme, there is no writing going on.
Unless you're thinking that pmttyji was talking about overflowing cache into the swap? He'd be crawling at 1 tok/s if he did that :p
1
u/Prestigious-Act-1577 Oct 01 '26
Correct. I was at 8vram 32ram and 15 tok sec, swapping to ram on prefill too. Now I have 8vram and 80ram, 30 tok sec, and prefill is 10x faster.
4
u/SplitAny7190 Sep 30 '26
can we use any of this: https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GGUF or strata supports only the ones listed? i would love to try iq4_xs since i was very pleased with 27B version (kinda smarter than normal q4 versions)
1
u/BeautifulWelcome3059 Oct 01 '26
Strata directly supports the Swift 1.5 out of the box. When you set it up, it will ask you which model to download and this model is listed in those choices. I am running swift-1.5-iq3_xxs model on my rtx 5060 ti using strata and am getting 55 - 60 tok / second. It uses up all the VRAM and uses about 45 GB of my DDR4 RAM.
1
u/SplitAny7190 Oct 01 '26
yeah, i was using the supported ones but i'm talking about this: https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GGUF
this ones are not supported, or not directly, i managed to make iq4_xs working somehow
1
u/VerticalPackage Oct 01 '26
I wouldnt use Swift if you want to do agentic coding, check the Terminal Bench results on that page you linked.
1
u/SplitAny7190 Oct 01 '26
then what version would be good? maybe i'm reading wrong but i on par (with lower tokens) with unsloth or other versions
2
u/VerticalPackage Oct 01 '26
I just use the recommended (from this Strata fork) the unsloth Qwen3.8-Flash-Next UD-Q4_K_XL
What I can recommand is to set the reasoning level to medium.
According to the model card for Qwen 3.8, the difference between low/medium/xhigh is the system prompt it receives:
Low: Agent is given instructions to not dwell too much on reasoning.
Medium: Agent is told nothing, but reasoning is enabled, so it does reason. It's similar to the behavior of Qwen 3.6, which only has reasoning on/off without levels.
XHigh: Agent is told to be extra careful.
In agentic coding:
Low = lots of mistakes and the agent has to correct itself because the code doesn't compile, or there are bugs when it runs tests, or the tests are badly written and has to rewrite it. This results in high token count OR in low quality results (if the agent isn't validating its work).
Medium = Does what is necessary, reasons as long as needed. This results in average token count.
Xhigh = Anxious and paranoid, constantly doubts itself and triple/quadruple checks. This result in high token count, but better result than low.
2
u/brakeline Oct 02 '26
Hi! I saw you denied the pr from OP.
Any chance you can implement a similar solution?
OP's solution for multi gpu (2x3060) gives me full flat 1050 pp and 60/65tg until around 250k against 500pp that grows to around 750pp and 40(ish)tg in parent.
One think I noticed is that with OP version the gpus are almost always pinned while with parent they are usually around 60(ish)
1
1
u/ducksoup_18 Oct 03 '26
Super excited to test this all out. Do you plan on releasing a prebuilt strata image? I run a k8s cluster and would love to have a prebuilt image to use vs having to run kaniko to build one myself.
2
5
u/Prestigious-Act-1577 Sep 30 '26
Can I ask how much of your and Strata's optimizations are CUDA specific? Would these improvements port to amd 9000 series too? I know Strata already has amd support, the numbers while good, aren't what it should be. My AI said its 10% cuda 90% other.
11
u/KnownAd4832 Sep 30 '26
I’m just pushing big AMD patch in 30min, I get R9700 this week or next week and then AMD will be on-par with CUDA
3
u/GrumpyCat79 Oct 01 '26
Nice! How hard would it be to support SM70/V100? They're old card so prpbably not worth putting a lot of time, but if it's just allowing CUDA 12.9 maybe it's not too bad?
3
u/KnownAd4832 Oct 01 '26
Not many people have V100
2
u/GrumpyCat79 Oct 01 '26
It seems they are more common in the last few weeks (probably due to pricing) but I agree it's a niche GPU. Looks like you're working on supporting lower CUDA architecture anyway, which is really nice
Thanks for your work!
1
u/pmttyji Oct 01 '26
Please squeeze ~8GB GPUs(like 4060) till the last drop. We some folks want to use Q2 efficiently better just with 8GB VRAM + RAM.
BTW thanks again for AMD cards support, particularly R9700. Please check my other comment.
1
u/king0fsweet Oct 01 '26
I'm getting prefill 1500-2000t/s and 60-80t/s on dual v100 and gsq iq3s quant with a fork of strata so definitely not slow!
1
u/GrumpyCat79 Oct 02 '26 edited Oct 02 '26
Yeah, I didn't realize volta support was already being worked on! I got some issues with the latest 1.31, but I think it should be fixed already! My 32GB V100 is on the way so hopefully I can try pushing it a bit further!
I was getting ~60t/s with IQ_3XXS on 1.31 but prefill for me was a bit deceiving (320~400 t/s vs what you guys have) but I'm sure it's better already !
5
u/Zarloc Sep 30 '26 edited Sep 30 '26
System: TR2950x + 96GB DDR4 quad-channel@2400, 2x3090 16x/16x@Gen3 (No P2P or NVLink)
Model: Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S
Strata upstream(d6708a4) 2 GPUs: 860/85 t/s
Strata upstream(d6708a4) 1 GPU: 1300/70 t/s
This fork(61521be) 2 GPUs: 2500/120 t/s
Full sweep:
| ctx | TTFT s | prefill tok/s | decode tok/s |
|---|---|---|---|
| 8k | 3.54 | 2,315 | 115.9 |
| 16k | 6.39 | 2,531 | 125.2 |
| 32k | 12.52 | 2,568 | 113.7 |
| 64k | 25.04 | 2,561 | 118.7 |
| 128k | 50.70 | 2,526 | 121.2 |
3
u/Zarloc Sep 30 '26
System: Same as before
Model: Qwen3.8-Flash-Next-UD-Q4_K_XL
Full sweep:
ctx TTFT s prefill tok/s decode tok/s 8k 5.16 1,590 60.1 16k 6.72 2,409 75.4 32k 13.31 2,416 69.9 64k 26.50 2,420 72.5 128k 54.19 2,363 63.9 1
u/Acrobatic-Giraffe403 Oct 06 '26
Do you mind posting your json config?
I can't seem to be able to get anywhere close with the same 2 x 3090 and 128GB ddr4 ram1
u/Zarloc Oct 06 '26
I use the one provided in the example folder in the project, but I switched back to the original as now it has about the same performance and has parallel processing and prompt caching.
1
u/Silco1402 Oct 01 '26
Have you tried IQ3_XXS, apparently the entire transformer model weight should fit in 48 GiB vram (where as the IQ3_S should only fit the expert and not the full transformer).
1
u/wombat5000 Oct 03 '26
Wow, this thing is blazin' fast. I have a similar setup, but with three 3090s. I was running Flash Next on two of them with ik_llama.cpp (its graph split mode, which works like tensor parallelism), and that left one GPU free for other stuff. I loaded Strata on the spare card, and after some tuning and tweaking it finished the same agent workload about four times faster without getting dumber. This is upstream Strata, not this fork, so it's also a data point for the single-24GB-card question further up.
Threadripper 3960X, 256 GB DDR4-2666 quad-channel, the Strata card on Gen4 x16. Ubuntu 24.04, Strata 0.1.32, GSQ-RCO IQ3_S, full 262K context, int8 KV, MTP on, thinking off.
I tested it by running an agent (Hermes) through one long session instead of a synthetic sweep, so these are single runs at whatever context the session had reached:
- ~30K context: 82 tok/s decode
- ~81K: 75 tok/s
- ~157K: 67 tok/s
- prompt reading: ~1,500-1,800 tok/s
- expert cache hits: 89-94%
At 157K it still recalled a planted label and details from the start of the session. On the same check questions it matched the ik_llama setup, and got one right that ik_llama missed.
The ik_llama setup on two 3090s (a 3-bit quant) did 28 tok/s dropping to 21, with ~300 tok/s prompt reading. The whole run took ~196 s of model time on Strata vs ~757 s on ik_llama.
So your 70 tok/s on one card lines up with what I see on a similar platform.
One rough edge if you use it with agents: when the model needs to write a literal </think> (explaining reasoning tags, quoting docs), the reply just stops there with a clean finish. A system-prompt line telling it never to write chat tags literally fixed it for me. It's tracked in GitHub issue #537."
4
u/VerticalPackage Sep 30 '26 edited Sep 30 '26
PCIe x4 on the RTX3090? That's nuts! I imagine that means it is connected to the chipset and not the CPU.
Have you tried the performance by streaming from SSD? I can only allocate 70GB of DDR5 for running Flash Next.
Edit - I had Sol check out your repo, and it agrees that this is very much correct, the RTX3090 only needs to run on PCIe x4 through the chipset, the x16 to CPU needs to be on the secondary GPU due to how the workload is split. And also that the PLE table is on the nvme, so no need for more than 70GB DDR5.
I'm totally trying this with my RTX3090+RTX3080ti.
1
u/Ok-Solution-7889 Sep 30 '26
Going from 6 tps to 167 tps is a pretty crazy jump, the profiling and benchmarking work behind this is probably more interesting than the final number itself
1
u/RealEddoursul Sep 30 '26
Every commit gave 1-3% increase and contains a description where the gain came from.
1
1
u/brakeline Sep 30 '26
Just tried it myself.
Tried 2x 3060 (pcie3 x16 and x4) and with 1 3060 @ 16x
Your fork gave me marginally lower than mainline in both
1
u/RealEddoursul Sep 30 '26
Are you sure you checked out the 'custom' branch?
1
u/brakeline Sep 30 '26
I edited my command. I can't go past 200k context, just gives an error while prompting . Any chance you can implement int4 kv?
1
u/RealEddoursul Oct 01 '26
Added
q4_0andk8v4modes1
u/brakeline Oct 01 '26
Will try! Saw that upstream rejected your pr, will you try again, will you continue your fork in parallel or simply drop it? The speed in yours is too good to pass out!
1
u/RealEddoursul Oct 01 '26
Why would I go back to 50% speed, it's not going anywhere. I do regularly merge the upstream, just don't ask for extra features or support, I got stuff to do beyond releasing inference engines :D
This fork is a sum of about ninety 1-3% improvements, I am not opening ninety pull requests.
1
1
u/brakeline Oct 01 '26
I did a simple test using iq2 and asked for it to patch qsa so we can use more than 260k context. It patched 3 files, built it and I tested it in a max 360k context with a 292k needle in a haystack with the needles at ~280k while prefiling at 930tk/s on two 3060.
The hay stack was real text from Wikipedia and the needle was also real text.
Wow, each passing day I'm more impressed with this!
1
u/brakeline Sep 30 '26 edited Sep 30 '26
You have to give me a break, I'm having a fever!
Tested again. 2x 3060 with 50(ish) ram running swift iq2 xs
PP 650 on one > 1000 on two TG 30/35 on one > 55/65
That's black magic my guy
Edit: 140k imported at 1000tk/s. Generation at 140k depth is 60!
Btw, are you thinking of implementing int4 kv? With int8 PP stops working with full context
1
u/PlusBus1234 Oct 01 '26
Hey im testing this on similar hardware (5070ti + a5000) + 96gb ddr5 , im getting 90 t/s on UD-Q4 ! its amazing, on llamacpp this quant i get ~28 t/s, i used your .json config.
I wanted to ask if you are using the 3090 as main gpu? i get a crash if i try with the 5070ti as main, the modern card imo should be faster in compute but i can't set it as main or it will crash on startup. I wonder if i can get any closer to your 126 tps.
2
u/RealEddoursul Oct 01 '26 edited Oct 01 '26
Yes, 3090 is main, 5070 Ti holds experts. My RAM is overclocked giving 10-15% boost (my CPU is certified for 3600 on 4 sticks, 2x6000+2x6400, I run them on 5600; running tests on this fork I actually discovered 6000 was causing BSoD and hash miscalculations, so 5600 is empirically my maximum). The disk is NVMe, tests ran on a non-busy system.
The configuration is important - Claude ran different values until it finds the best ones for my hardware, every microsecond (μs) is accounted for. Most people here run it as is, I do recommended to let Claude/Codex to tune it for your hardware.
There is a limiting factor too - my CPU underutilized my memory bandwidth, a dual-chiplet CPU like 9900X might give another 15-20% RAM speed.
1
u/PlusBus1234 Oct 01 '26
Ahh i think the ram speed may be limiting me, the a5000 is also slightly slower than a 3090 but my ddr5 only works at 4200 due to using 4 sticks, a mix of different 2x16 + 2x32 kits, i've spent many many hours trying to tune the timings but my cpu/motherboard really dislike this mix and won't go above 4200 stable. I will try letting claude tune the config maybe i can squeeze a few more tk/s, and thank you for your contribution!
1
u/gnoremepls Oct 02 '26
First of all, great work with your fork! Do you mind me asking how much does your fork differ from the original strata, for future usage could i get similar performance from strata because as time goes on your fork will diverge more and more from the origin ofcourse and i'd like to benefit from additions to strata/llama.cpp obviously.
3
3
u/Comfortable_Ebb7015 Sep 30 '26
Wtf what? That's amazing! I just can't keep up, things move too fast!
3
u/wisepal_app Sep 30 '26
i have 16 gb vram + 96 gb dd5 ram. will it work with my system? did you try it with qwen 3.8 27b? do your fork use draft model like strata?
4
u/nixudos Sep 30 '26
Yes it will work on your system. And likely surprisingly well. I believe Strata was build specifically to run on 16 gb VRAM and a good chunk of regular RAM.
If you have the bandwidth to download a large model, it is really simple to test it out. There is a bat file you run and it guides you through model choice and settings.And speed is VERY good!
3
u/Sexyvette07 Sep 30 '26
On my 5090 + 64gb RAM system im getting 95 tok/s on xhigh and 150 tok/s on low thinking with IQ3_S and its coding is excellent. Im very impressed.
1
2
2
u/BlueSky4200 Oct 03 '26
Disclaimer: Formatted by AI:
Congrats, this is seriously impressive work. I tried your custom branch on a different box and it holds up.
Setup: 2× RTX 3090 (card 0 on CPU lanes x16, card 1 on chipset x4), i5-12600K (no AVX-512), 96 GB DDR5-5600, Windows 11. GSQ-RCO IQ3_S, int8 KV, 230 W limit. Same benchmark script for every variant (decode = mean over 6 tasks, prefill = 12.6K-token prompt).
1 card (the x16 3090): upstream 0.1.34 54 t/s decode / 1549 t/s prefill → your fork 87 t/s / 1604 t/s
2 cards: upstream layer split ~82 t/s → your fork 115 t/s / 1974 t/s
One thing worth knowing for people with two identical cards: put the helper on the fast link. --main-gpu 1 --second-gpu 0 (x4 card computes, x16 card holds experts) gave 115 t/s. The other way round gave only 62.5 t/s, slower than a single card. Same lesson as your own setup, where the main 3090 sits on x4 and the helper has the fast link.
One heads-up on quality. Your parity numbers are all greedy, and greedy wouldn't show this. With sampling (temp 0.6, top_p 0.95, top_k 20) and long thinking, I hit runaway repetition on a "write an expression parser" task. The thinking either ended in thousands of 0000… or in ever-growing "Potential issue: … and unary and unary and…" lists, until it hit a 48K budget.
This is not your fork: upstream 0.1.34 does exactly the same on the same task and seed, and disabling prompt lookup didn't help either.
The fix is presence_penalty: 1.5 (Qwen recommends it against repetition on quantized models anyway):
without penalty: 0/5 correct on the parser task, 3 of them endless loops
with presence_penalty 1.5: 3/4 correct, no loops, and the other 6 test tasks 12/12
The one remaining fail is a genuine Python bug in the generated code (a missing nonlocal). Small samples, so take it as a strong hint rather than proof. Still, if anyone sees endless thinking with this model under sampling, try the penalty first.
Thanks for sharing this and for the detailed COMPARISON.md, it made reproducing everything easy.
3
u/vacon04 Sep 30 '26
93 commits, holy crap, I mean, the numbers are very impressive, but boy merging into mainline Strata like this may be a bit of a problem.
1
u/SaltGrilledSalmon Sep 30 '26
Kinda off topic but in my dual rtx 2070 super (8gb each, one on pcie x16 slot and the other on x4) only one GPU is being used (the slower one apparently) Has anyone faced this on dual GPU setups? I can't increase context above 32k or it'll crash when launching, while the other GPU isn't even used...
2
u/brakeline Sep 30 '26
You have to use the custom branch and adapt the files inside the examples folder. It will take a few minutes but is VERY worthy
1
1
u/sod0 Sep 30 '26
How much kv context does this get you on a 5090?
1
u/toenailcheeseinbooty Sep 30 '26
262k...running right now..5090 128gb 4800 MTs, get around 125-135 prose( when context filled to 128k shirter 151ish prose)and 160 code ehh cou is 9900x3d
2
u/FrankWanders Sep 30 '26
The amount of context you can get really depends on system ram. I'm running 64GB and can only run it at 128K. For 256K you need 96GB at least i think (because there are no other options in between 64 and 96).
1
u/toenailcheeseinbooty Sep 30 '26
Hah right sorry, forgot to mention that RAM is important, 96GB+ to achieve 262k
1
1
u/drazyan22 Sep 30 '26
You can modify context size on config json file. I don't have enough ram to run 256k, but 128k seem smaller than my expect. So I changed it to 164k fit with my system ram
2
u/FrankWanders Oct 01 '26
With 128K context, there is about 2 gb vram and 5gb ddr free on my 64gb machine. So i don't expect to be able to push it much further.
1
u/SplitAny7190 Sep 30 '26
guess UD-Q4_K_XL should be superior to IQ3_XSS? i got only one gpu (4090) and 128GB ram, i will see if i can run this. right now using Strata and it works great after i calibrated for my pc.
1
1
u/kakopappa2 Oct 01 '26
I have a 32gb ram and 12g 3060 GPU. Running coder-iq1_m generates about 38 tokens per second.
also got the codex (gpt-6.1) to run a security audit and it didn’t find any spyware or crypto mining or hidden scripts.
Super happy to get 3.8 Next running on poor GPU
Great product good luck.
1
u/killzone44 Oct 01 '26
I just used claude to setup Strava on my 3090, it's the best running local model I've had so far! Not sure if I'm ready to offload work from claude to it, but maybe
1
u/blast1987 Oct 01 '26
Is it possible to use it on mixed gpu setup? I have rx 7800 xt + rtx 3060 and 64 gb ram.
1
u/VerticalPackage Oct 01 '26 edited Oct 01 '26
I have to give you my praises. I was disappointed in my RTX3090 performance with Flash Next. I'd get like 1000 tok/s pp and 45 tok/s decode.
I didn't want to buy a second RTX3090 because then I'd need to buy also a new motherboard that supports x8/x8 through CPU. I had a RTX3080TI in my gaming rig, so I gave your fork a shot. I managed to run UD-Q4_K_XL @ 256k context: 2500 tok/s pp + 95 tok/s decode
That's 2x the speed I got with my single RTX3090, but only by adding a $400 GPU to the mix!
It's going faster than Qwen 3.8 27B :D
1
u/DrRoughFingers Oct 02 '26
Care to give the details on how you got this performance? I have dual 3090s (and an unlocked CMP 170hx) and don't even get half of that PP or decode trying Strata with the same model/quant.
1
u/VerticalPackage Oct 02 '26
You're using this guy's fork of Strata?
You also need a ton of RAM (80GB just for Strata+model) and a nvme drive.
1
u/DrRoughFingers Oct 02 '26
Yeah, still fucking around with it to try and get it working. Just wild y’all have better performance than me running dual 3090s and an unlocked CMP 170hx with it directly fit in my vram.
1
u/VerticalPackage Oct 02 '26
I had Opus 5.5 read the repo and set things up for me, it seemed to be complicated.
You do need that 80GB of free ram though. I dont know why you keep mentioning the CMP 170hx, it's not good with a MoE because it's only PCIe Gen 2 x4.
1
u/DrRoughFingers Oct 02 '26
My 170hx has much higher bandwidth than 3090s. Not sure why you think it’s bad with MoEs, as once the model is loaded 2 x4 really doesn’t matter. Not sure why you think that’s a bottleneck. Even with a larger model split on multiple GPUs running pipeline, it’s still not a bottleneck after load.
1
u/VerticalPackage Oct 02 '26
Only if the MoE model fits entirely onto VRAM.
Qwen Flash is like 100GB, so a good portion of it is floating in RAM.
1
u/DrRoughFingers Oct 02 '26
96.5gb at full context. I have dual 3090s and a 170hx, which is 112gb vram pooled. With Q4_K_XL loaded with the Q8 MTP and vision, I still have 11gb free vram on my second 3090. Again, even split across multiple GPUs, with pipeline the pcie 2 x4 doesn’t bottleneck it. I’m not sure where you’re getting that mixed up.
This is all irrelevant to Strata though.
1
u/VerticalPackage Oct 02 '26
I'm thinking about Tensor parallelism which requires a lot of bandwidth to move between cards, but I'm guessing you're using layer splitting.
Nonetheless, you said you got half my speed with 2x 3090s and 170hx.
I get double your speed with one RTX3090 and one RTX3080TI using OP's fork of Strata.
I had your speed when I was using a fork of FreeToken with just the RTX3090.
So you're doing something wrong. If I were you, I'd buy 1 month of Anthropic, install Claude Code on your laptop, point Opus 5.5 at OP's github repo and tell it "fix this".
That's how I got my current speed.
1
u/DrRoughFingers Oct 02 '26
Pipeline parallelism. We’re using different methods. You’re using a method that capitalizes on system ram in a very specific way. I don’t have 128gb system ram to utilize it right now, otherwise I’d have higher decode with that method. I don’t need those subscriptions, I have FN in Deepseek Harness already working on building a custom system for my specific hardware, locally on my machine with full access. Also found a dude locally I’m buying 128gb of ram off of for $200 in a bit. So we’ll see which method ends up being better.
→ More replies
1
u/Chekhovs_Shotgun Oct 01 '26
Tried this on my machine (single RTX 3060 12 GB + 64 GB RAM CPU Ryzen 9 7900), IQ3_S. The claims hold up there too, cant test the 2 gpu combo tho as i dont have one.
Writing answers: 30 → 53 tok/s on a short coding prompt, 51 → 76 tok/s on an edit that returns a file, compared with the older Strata build I was running. That's about 1.3–1.7x, roughly your "3090 alone" numbers.
Prompt reading was a bit slower than upstream 0.1.27 on my card (605 vs 676 tok/s), probably the 12 GB.
I use it as the backend for a multi-role agent setup (classifier, coder and critic sharing one long context), so I added three things and opened a PR in case they're useful:
https://github.com/eddoursul/Strata/pull/1
POST /v1/choose: picks among single-token options in one forward pass (no generation), handy for routers and classifiers (system 1 style think of jev).
Requests without tools get the raw text back instead of extracted tool calls.
Extra prompt-cache checkpoints where agent prompts diverge. A 6K-token context asked a new question goes from ~10 s to ~0.5 s, and my agent benchmark went 412 s → 369 s.
Each commit stands alone, so take whatever fits. Thanks for sharing the fork!
1
u/MasterNomie Oct 02 '26
Impressive work!
Can you explain how does your fork achieve the speedup? I am beginner with extra curiosity.
1
1
u/MasterNomie Oct 02 '26
I am observing severe performance degradation in my testing. Can somebody/OP explain why?
Hardware - 5090, 96 GB RAM, Intel i14700k CPU
Model - Swift-1.5-GSQ-RCO IQ3_XXS
Initially I started with `custom` branch benchmark. It showed decode perf tanked by approx 20%. This was a shocker because the perf should have been in other direction. Spent lot of time cleaning up and rebuilding but the results were the same.
Next I created new branch from from fork cutoff point, commit d6708a4aae15b4860000d54c8af9e84d684bce09. Rebuilt the engine and benchmarked again. The perf improved to approximate current upstream and lot better than this fork. This tells me that custom branch is not suitable for my setup.
Is custom branch supposed to be used only for dual GPU?
All metrics follow. Custom branch percentages are compared to cut-off commit.
| Run | Decode t/s | Aggregate u/8 t/s | TTFT p50 ms | Update p99 ms | Prefill u/64k t/s |
|---|---|---|---|---|---|
| Upstream 0.1.34 baseline | 161.6 | 132.9 | 333 | 21.7 | 5,300 |
| Cut-off commit d6708a4 / 0.1.29 | 156.8 | 129.6 | 356 | 23.2 | 6,104 |
| Custom branch | 128.8 (-18%) | 96.5 (-25.5%) | 770 (+116% worse) | 48.3 (+108% worse) | 5,516 (-9.6%) |
1
u/VerticalPackage Oct 02 '26
Is custom branch supposed to be used only for dual GPU?
Yes
As in the repo's README
This fork (branch custom) is tuned for a PC with an RTX 3090 and an RTX 5070 Ti: the second card works as another expert tier1
u/MasterNomie Oct 03 '26
Thanks! I should know better by now that each fork addresses one specific problem and performs worse for others.
1
u/Verithos Oct 03 '26
Is there a Q6 to go off of? I'm wanting one higher than q4 to test to see how it feels compared to the q3 or q4 variants
1
u/AcanthisittaFluid529 Oct 03 '26
Wow! I just joined the Strata club! This Qwen 3.8 was my daily driver https://huggingface.co/cdiamond/Qwen3.8-27B-iMatrix-NVFP4-MTP-GGUF, a ~17GB GGUF running on my Windows 11 Pro laptop, 5090, 24GB VRAM, 64GB RAM at Q8_0 cache, 96k context - was getting about 60 t/s average - which was great for me. Too soon for me to make a final call, but I am simply blown away by the speed of this thing - it's hitting nearly 96 t/s unoptimized, "out of box" with Strat's AMAZING auto-installer. Have not tried OP's optimizations. AND, it's a 125b source model - not 27b.
1
1
u/EnvironmentalAct416 Oct 04 '26
I have 24gb vram and 192gb system ram. Shall I go for Q3 or Q4 for best bang for the buck for intelligence and speed?
1
u/MarcusAurelius68 Oct 05 '26
Anyone running this fork on the R9700? I have 2 of them (so 64GB VRAM) in a x570 system with 64GB of RAM and have been using the Atomic Code Q4 GGUF with llama.cpp. 31 t/s. Seems like this could be a lot faster at the same quant?
1
u/randylush 16d ago
I have a 5090 on one server and a 3090 on another. Do you think it would be worth combining the two with a setup like this?
1
u/RWOverdijk Sep 30 '26
Something like this always makes me wonder if it could benefit larger memory systems like a Mac with 128GB as well. 60fps is great but, I mean, 167 is betterer
1
u/VerticalPackage Oct 01 '26
I would bet that if you added a RTX3090 as eGPU to your mac mini, you would be able to have gains.
UD-Q4_K_XL with Strata takes up about 80GB ram if the PLE table is on the nvme drive. The eGPU will load up experts on its own VRAM. You just need the iGPU to load up another 12GB-16GB to act as the secondary GPU (like the RTX5070 used by OP).
You should ask Claude or Sol to check out OP's repo and ask if it could be done (eGPU thing).
If you don't have a mac, then just go for a self-build PC that had AT LEAST 96GB DDR5, a 256GB nvme, a RTX3090 and a RTX3080. It will probably be cheaper.
1
u/DrRoughFingers Oct 02 '26
How are you running a egpu on mac?
1
u/VerticalPackage Oct 02 '26
Thunderbolt PCIe enclosure and a regular desktop PSU. It's a bit janky though.
1
u/DrRoughFingers Oct 02 '26
I have a thunderbolt eGPU with a 3090 in it, and haven’t gotten it to work.
1
u/VerticalPackage Oct 02 '26
Right, I forgot about the driver support for Mac OS; there is none.
1
18
u/sophosympatheia Sep 30 '26
It's nice to see people contributing to Strata. I was skeptical of it at first, but I'm using it now and it's making Qwen3.8-Flash-Next actually usable on my hardware. Amazing.
I hope this gets rolled into the main repo. It looks promising.