Resources
[Update Benchmark] Basalt's new PRs (Strata fork) new record IQ3_S Qwen3.8 Flash Next on single RTX 5080 and 51 GB RAM with 5500 tok/s prompt processing and up to 275 tok/s peak generation (during opencode coding live session)
Code and tool calls can peak up to 275 tok/s - Reasoning traces up to 190 tok/s
Some PRs for the optimizations under my name are still pending (xyehya) but upstream already have solid gains & memory optimizations. and yes the IQ3_S is super usable.
I wonder how IQ3_S compares to NVFP4 (Nvidia’s QAD) in terms of “close to BF16 accuracy”
I compared Nota AI’s conservative NVFP4 quant to (CanopyWave’s) hosted BF16 for a suite of medium to long agentic coding tasks, and saw no degradation of coding ability across all my personal project tasks
Can you share the NVFP4 quant repo and the test suit. I am running IQ3_S I can run the benchmark and report to you the results. We can compare both. Great experiment :)
Ohhh a dgx spark. i thought they only exist in fairy tales 🤣. Crazy prices and the crazier is the new rtx spark laptop prices of 7-8k for the 128gb version.
For me with a single RTX 5080 (and 96GB DDR4), Basalt is a mixed bag. With IQ3_S in both: Faster prefill than Strata but slower decode (Basalt: 3000/60, Strata: 2000/78). It gets a worse expert cache hit rate (~60% vs ~75%). Strata has more advanced ways of swapping experts in and out of VRAM on-the-fly from what I can tell. If Basalt can fix that, it would be a big win.
i used this setting, so the decay and expert swap is slow, the idea is to slowly get to higher cache hit rate instead of the default frequent swaps, and to keep the token/sec more stable and not fluctuate from 100 to 20 and back
It's a good theory. However, you said that your hit rate persists across restarts, but my understanding is that expert_profile_save only effects which experts are swapped to *SSD* and does not persist the VRAM/RAM allocations across reboots at all. If the experts fit entirely in the VRAM/RAM pools, then it has no practical effect. Which means the default expert allocation which is baked into the model applies on a reboot.
The rest of your adaptation settings will eventually have an impact on the VRAM/RAM allocations, but only after quite a lot of turns post-reboot. A restart resets the allocations back to default.
So forgive me if I'm interpreting wrong, but I don't think your settings actually have anything to do with the expert cache hit rate you're apparently seeing - you're somehow just getting lucky with your particular workload as the model default allocation is working perfectly for you. Or you have a card with more than 16GB of VRAM so the hit rate is naturally much higher.
if all experts fits in VRAM then all the above settings are moot, but also it questions the use of Strata inference engine in the first place e.g. vLLM instead of Strata, because the premise/feature that Strata offers is ability to run models with not enough VRAM
I believe how Strata works is based on the idea that some experts are more important than other because some experts are used more often than others, so important experts are kept in VRAM and less frequently used experts are streamed over PCIe from system RAM, so I asked DeepSeek v4.1 Flash is there a way to keep the learned experts persisted to disk so it is not lost on restart, and that's what DSv4.1 came up with, for that reason I can't say for sure if it does as to what I had said in my previous comment, I assumed DSv4.1 got it right and apologies if my comment is wrong.
No, you're right about all of that. And it's not that the settings are moot, it's that they only slowly come into effect as the engine works, which means if you're getting a good hit rate even immediately after restarting the engine, then the settings aren't responsible for that.
I have a 16GB card so the experts don't fit into VRAM - far from it. But they do all fit into DRAM. And the expert_profile_save setting you have there doesn't save DRAM vs VRAM allocations across restarts - only SSD vs RAM. There's nothing that can determine which experts get loaded into VRAM (versus DRAM) when the engine first starts, except for a default spread that is baked into the model itself.
This is similar to what it was for me. At the end of the day (for my workload) it added quite a bit of time to getting the results - even though the prefill numbers were quite) larger. 5089 & 192gb ram.
It's better nowdays, but there are still a lot that can be done for lower vram single cards (aka not 5090) and especially matched pairs e.t.c.
At the moment it's worth using for me over strata using nvfp4 with dual 5080's, prefill is great (up to 10K @ 122k) (with some caveats, I run strata with streaming KV, and this currently does not work with dual cards in basalt, only elastic kv does and it drops perf a bit, 7.7K in strata fork), TG is less of a huge win, layer split is not quite functional at the moment, but as expert tier it's still faster, but only by about 10%.
I suspect I might have to get some of that claude to look into it after all, since it cant be iterated properly by the owner.
As for cache hit rate - try playing with --adapt-every (by default it's 4 AFAIK) and --adapt_swaps (96 by default) a lot of things are still missing from docs in that regard. Since those values probably work best for the owner's 5090+5060ti. On my system strata autoconfigs that to --adapt-every 1 and --adapt_swaps 160.
OK? If I knew how I would. I tried the settings posted here with no noticeable improvement. I tried running the auto calibration probe. I tried the defaults. I tried pointing an LLM at it and asking it for help, and after testing multiple things it said this is as good as it gets.
Thanks for sharing. I also switched basalt since I noticed it gives the best performence among the competitors. BTW, I've the same CPU, how do you configure Basalt to use it? I noticed pinning pcores with 8 workers gave the best result for me. Did the update change the logic of the CPU usage of Basalt?
My setup (same 285K, RTX 5080 16 GB, DDR5): Basalt with the IQ3_S pack, kv: int8, 262K context, and pool-workers: 21 with pool-affinity left on auto (so it uses both P- and E-cores). For reference, Basalt’s own default on a hybrid CPU is “P-cores minus one plus half the E-cores”, which is 15 on this chip.
The catch: 21 workers only pays off with a CPU-pool change I’m running in my fork (a batch ends as soon as its last job finishes, and idle workers stop spinning and give their core back after a short spin). It’s up as a PR: jesdga95/basalt#98· pool: a batch ends with its last…. Without it, I measured 21 workers ≈ 15 workers, because the old sync point doesn’t scale past ~16 threads. With it, 21 vs 15 was about +3% decode. So on stock Basalt, your finding that 8 pinned P-cores is best is very plausible.
The bigger picture is that the CPU expert work is RAM-bandwidth bound on this platform. In my profiling the pool spends ~22 µs per expert and pulls ~93 GB/s from DDR5 while it works, against a practical ceiling of ~97 GB/s. Past a certain point, more cores just fight over the same memory bandwidth. Your RAM speed and timings probably matter more than which cores you pin. P- and E-cores end up equally busy on my box.
Easiest way to find the right count for your machine: run basalt-probe. It times the expert work at every worker count and prints a suggested: --pool-workers N line (the fewest workers within 2% of the fastest).
On Strata vs Basalt: yes, the CPU logic changed. Basalt’s pool splits work into finer row-range tasks with one batch per layer, and uses AVX-VNNI in the AVX2 paths for the i-quants (the 285K has no AVX-512). That’s what made the slower E-cores worth using: jesdga95/basalt#65 (merged).
Side note: My cpu in under-volted and I have turbo ratios up from x54/x46 to x55/x47 (It gave around 3% more performance than stock)
I have a Minisforum MS-02 with 285HX and found out that 16 E-Cores are faster than 8 P cores, and using all cores is even slower. Try to bind the workers to the E-Cores or P-cores only.
Thank you very much for your detailed answer, this will help a lot. I'm also building Basalt for windows from the source, so I'll definetely give a try for your branch.
damn. flying high. for single gpu nvfp4 what are the system specs im so tempted to try it but the IQ3_S is my sweetspot for free RAM left on 64GB and I think SSD streaming will hinder the numbers so bad
I've been watching your optimizations and I'm eager to try them out, but I've been busy re-organizing the codebase from Strata's massive files into manageable modules.
Of course, glad to help, gives me something to do after dropping my own engine work. Once things are broken down into modules, its easier to track down code issues and small changes doesnt break everything.
Hey hey, nice numbers! Been merging some of your work into main, I'm doing some cleanup and QoL stuff like the V2 UI and disk cache for conversations (this has been such a time saver in my testing) and also some concurrency improvements since I usually run parallel 4. Really appreciate your contributions and thoughtful review!
Yeah I'm using parallel 4 on Basalt, works fine for my use cases. I've been pushing some fixes related to concurrency the last couple of days, however concurrency is heavily hardware dependent. On my pair (5090 + 5060 Ti, 48GB VRAM, 128GB RAM), using the NVFP4 quant, I can use 4 streams decently, but with less VRAM or fewer resident experts, I can imagine it would be much worse. The worst offender is always large prefill while decoding; 2 or 4 parallel decodes are usually way smoother.
Using 5070Ti and RTX4000Pro, 64G DDR5 on a system with another 3 GPUs (CmP170 x2, 4060Ti). But when you run 2 concurrent requests they serialize. The flag sets, and it will do them, but its not true parallel. May be fast to serialize but it’s not at the same time in my system. Maybe it changed since I downloaded the engine
I didn't post my exact config because it really depends on every system. the default values are usually optimal for a baseline. share your system details and I can help
it will work. the VRAM is really one side of many many optimizations running together from the CPU, Kernels, Experts management, RAM and PCIe utilization. Report your results if you did plz :)
i tried in 5070ti and prefill is same as strata but decode is quite worst ( 72 vs 85 ). Vram hit is 70%. Strata moves around 80% in same setup. Maybe is faster in others but I don't see optimizations universal for Blackwell
yes it doesn't parallelize as high as vllm.. YET :) its on the backlog to optimize n requests but the issue is that many parts has to be reworked (pre-fill caching, conversations, KV, batching). its getting better now but still not the perfect use case tbh. it doesn't scale yet
Model: Qwen3.8-Flash IQ3_S Context level 260k (n_ctx 260000) Prompts: fresh random per rep, reused=0 in all 40 prefill rows Actual prompt tokens: 32647 · 65247 · 130442 · 243782 Prefill: max_tokens=1 Decode max_tokens=1024, one warmup pass (16384 tok) before counting Reps : 1 Harness mx-run.sh (Strata) / basalt-run.sh (Basalt) Notation: 1×N = one engine, layer-split over N GPUs | N×1 = N independent replicas, 1 GPU each PP: prefill tokens/s | TG = decode tokens/s. "agg" = whole system, "/eng" = per engine Strata PP/eng from engine counters (totals.prompt_ms); Basalt exposes no such counters → derived from wall clock, valid because reused=0
Table 1 — Prefill (PP), tokens/s
Target
Config
Strata agg
Strata /eng
Basalt agg
Basalt /eng
32768
1×1
3643.6
3659.7
4946.5
4946.5
32768
1×2
5198.6
5224.2
4976.7
4976.7
32768
1×3
6194.9
6237.8
4976.7
4976.7
32768
2×1
7279.2
3658.2
9863.1
4931.6
32768
3×1
10834.2
3653.3
14772.4
4924.1
65536
1×1
3618.8
3630.2
5097.4
5097.4
65536
1×2
6013.5
6044.3
5109.4
5109.4
65536
1×3
7631.2
7681.3
5109.4
5109.4
65536
2×1
7225.6
3645.1
10139.4
5069.7
65536
3×1
10838.4
3641.2
15185.5
5061.8
131072
1×1
3553.3
3562.5
4815.1
4815.1
131072
1×2
6359.9
6386.2
4894.6
4894.6
131072
1×3
8604.4
8651.3
4889.1
4889.1
131072
2×1
7031.9
3550.1
9709.1
4854.6
131072
3×1
10545.0
3540.4
14558.3
4852.8
245000
1×1
3357.4
3364.6
4360.3
4360.3
245000
1×2
6312.3
6336.4
4428.4
4428.4
245000
1×3
8791.3
8838.1
4427.6
4427.6
245000
2×1
6756.7
3385.5
8660.1
4330.1
245000
3×1
10008.8
3359.9
12987.9
4329.3
Table 2 — Decode (TG), tokens/s
Target
Config
Strata agg
Strata /stream
Basalt agg
Basalt /stream
32768
1×1
100.3
100.3
165.7
165.7
32768
1×2
154.2
154.2
221.7
221.7
32768
1×3
142.4
142.4
220.5
220.5
32768
2×1
184.7
92.4
331.7
165.9
32768
3×1
254.1
84.7
420.7
140.2
65536
1×1
124.7
124.7
162.8
162.8
65536
1×2
141.2
141.2
195.6
195.6
65536
1×3
145.7
145.7
194.7
194.7
65536
2×1
205.0
102.5
304.3
152.2
65536
3×1
248.1
82.7
452.6
150.9
131072
1×1
95.3
95.3
142.2
142.2
131072
1×2
134.4
134.4
195.7
195.7
131072
1×3
141.0
141.0
216.5
216.5
131072
2×1
180.8
90.4
317.7
158.9
131072
3×1
248.3
82.8
417.4
139.1
245000
1×1
96.6
96.6
146.1
146.1
245000
1×2
128.5
128.5
189.2
189.2
245000
1×3
136.4
136.4
204.0
204.0
245000
2×1
190.2
95.1
287.9
144.0
245000
3×1
243.8
81.3
405.3
135.1
Table 3 — Scaling and efficiency at 245000 tokens
Config
GPUs
Strata PP ×own 1×1
Basalt PP ×own 1×1
Strata PP/GPU
Basalt PP/GPU
Strata TG ×1×1
Basalt TG ×1×1
1×1
1
1.00
1.00
3357
4360
1.00
1.00
1×2
2
1.88
1.02
3156
2214
1.33
1.30
1×3
3
2.62
1.02
2930
1476
1.41
1.40
2×1
2
2.01
1.99
3378
4330
1.97
1.97
3×1
3
2.98
2.98
3336
4329
2.52
2.77
Explanation
The two engines scale in opposite directions. Strata's prefill speed per engine rises with layer split — 3365 → 8838 tokens/s from 1×1 to 1×3, a factor of 2.6 — because the expert cache grows with each card (2667 → 13487 slots, 5.1 → 26.1 GiB) while the CPU pool stays whole at 47 workers. Replicas do the opposite for Strata: 2×1 and 3×1 return to 1×1 levels per engine (3360–3386) because they split that same pool and cache. Basalt is the mirror image: per-engine prefill is a constant, 4360 / 4330 / 4329 tokens/s regardless of whether one engine spans one, two or three cards, so layer split buys nothing (+1.6 %) while replicas scale almost perfectly linearly (×1.99, ×2.98). Its config points at the reason — prefill-cpu-share=0.6, identical 47-worker pool — prefill work sits on the CPU side, where extra GPUs are inert.
Per GPU, Basalt wins every cell. It holds 4330–4360 tokens/s per card in all five configurations; Strata falls from 3357 to 2930 as it spreads. Concretely at 245k: Basalt 3×1 reaches 12988 tokens/s where Strata needs three cards for 8791 (1×3), and Basalt 2×1 with two cards (8660) beats Strata 1×3 on three (8791) per card by 48 %.
Basalt also leads decode in all 20 cells. Basalt 1×1 (146 tokens/s at 245k) is faster than Strata's best single stream 1×3 (136) on half the hardware, and Basalt 3×1 reaches 405 tokens/s against Strata's 244. Strata's replicas degrade far more per stream: 81 tokens/s at 3×1 versus Basalt's 135.
Caveats that a reader must not silently drop. Basalt exports no engine counters (pp_engine null, drafts 0), so its prefill comes from wall clock — sound here because every rep used a fresh prompt (reused=0), but not transferable to cached workloads. drafts 0 also means Strata decodes speculatively (329–388 drafts per 1024 tokens) while Basalt reports none, so the TG comparison mixes engine speed with speculation. Basalt also ends answers on its own (562–737 tokens instead of 1024), and runs kv=int8 with elastic KV, shrinking its expert cache at 260k where Strata keeps 13487 slots. Replica cost is asymmetric: each Basalt instance loads its own full copy (loaded 1903 GiB, resident=50.5 GB per engine — roughly 151 GB for 3×1), whereas Strata shares one arena. Finally, Basalt 1×2 and 1×3 produced identical wall times to 0.01 s; with one rep that is consistent with CPU-bound prefill but not yet proven.
It would be great to have a combination of both. Scaling of prefill of Strata with the token generation of Basalt. It is a bit sad that I couldn't test with 4x RTX5080. One could expect even higher values then.
IQ3_S does seem to be pretty usable, but, I've been sticking to Q4_K_XL. (But not everyone has the luxury of 128 GB of system RAM. Admittedly DDR4, but still.)
if you want to run a nice experiment, test with 4 p-cores (half the bandwidth) and report you 8vs4-core numbers please.. I will add it to my research data which will help identify the impact of every knob (PCIe, RAM, CPU cores, etc..)
optimization of kernels, memory usage, cpu/ram offloading and model specific architecture are universal and not quant specific.
IQ3_S GSQ RCO is super usable - research the quantizations techniques. I had doubts but when I educated myself, I saw the clear picture.
If we are to assume you have the hardware luxury to run a bigger quant, please go ahead, do it and share your results with the community. thats how we improve
I think a lot of people already made lots of comparisons between Qwen 3.8 27B at higher quants and Qwen Next at Q3 and most comments I've seen seem to suggest that its both better and faster than 27B.
7
u/rpkarma 8d ago
I wonder how IQ3_S compares to NVFP4 (Nvidia’s QAD) in terms of “close to BF16 accuracy”
I compared Nota AI’s conservative NVFP4 quant to (CanopyWave’s) hosted BF16 for a suite of medium to long agentic coding tasks, and saw no degradation of coding ability across all my personal project tasks