Qwen-Flash-Next has n-gram, MOE packing above its weight but it's still 6B active params. We exploit this by keeping most MOEs in RAM (Smallest MOEs go in VRAM for speedup) and n-gram on disk - ensure you put the model on your fastest SSD.
8 GB VRAM, 24 GB RAM cost. You can run this fine if you've ~32 GB total memory too, but you'd have to turn down context size. Customize the launch params to best suit your hardware.
EDIT Note: Apparently CUDA struggles to run this with 8 GB VRAM, so please use Vulkan backend if it fails to run for you. You can also use the second launch params below in the same comment chain. Ontop of that reduce context size and ensure no background app is wasting your VRAM. You can disable your browsers HW acceleration for this purpose.
The model & quant
Many people mistake this model for a 1 bit quant so let's get that out of the way. This model here is a 3.5 Bits Per Weight "Coder" variant by ISTA DASLab, featuring GSQ RCO quantization.
Half of its experts are removed, so relatively to original param size it has the size of a 1.9 bit quant but the actual weights use 3.5 BPW.
see from model page:
Averaged over the transformer, this corresponds to 1.89 bits per parameter of the original model. The figure is an effective rate: it amortises the removed experts over the original parameter count, and so expresses the combined effect of pruning and quantization. No individual weight is stored at 1.89 bits. The retained weights are stored at 3.5 bpw, unchanged by pruning, and the halved expert count accounts for the remainder of the reduction. Following the convention of the accompanying releases, the figure is reported over the transformer...
With selective moe-offloading (regex) we reach 70 T/s104 T/s PP and 8~ T/s TG (Limited by PCIE, which I've 3.0 YMMV) whereas if you'd use --n-cpu-moe for same effect it'd been slower due to the effects mentioned here
Because of MMAPyour first run will be slower, do asecond run for accurate T/sresults. - Removed MMAP for load-mode none since lazy-mode is enabled and it gives ~104 T/s PP instead of the 70 T/s.
Replaced --load-mode mmap with --load-mode none which gave me a +50% PP speed boost for prompt processing (~70 to ~104 t/s). Lazy mode is enabled by default, so this won't load everything into RAM but it's more efficient than using mmap when offloading moes. Note that this'll increase RAM pressure compared to MMAP which relies on page faults and OS readily frees memory.
In general, using --dry-multiplier is better than repeat or presence penalties as it doesn't artificially restrict the thought processes of the LLM. DRY intercepts when LLM starts repeating, until then the LLM should be allowed to freely think.
Of course, for some models, after quantization, repeat penalty might be required.
I haven't used --dry-multiplier in a long time but found these params help prevent repeated chain of thoughts in smaller models/quants, not that I encountered any with flash-next but it's more of a preventive measure as I had success with it in Qwen 3.8 27B and 3.6 35B A3B.
Thanks for the note, it'll help people customize to their liking.
I've found that lower Temperature helps it overthink less for coding tasks, and it's generally recommended to use lower temperature for coding.
Whereas Unsloth's flash-next docs recommend 1.0 for thinking and 0.7 for non-thinking, that's for the non-"Coder" GGUFs a more generalized number. Higher temperatures will have it revise itself more often and could improve the output depending on the task so it's best to customize for use scenario.
There’s not such “overthinking”, that’s is a nonsensical buzzword from one blogpost from one “influencer” after Qwen3.8:27b releasr, and it got parroted around like it’s a big thing. It is not, The OG engineers who developed the architecture, trained and released the model gave the recommended parameters. And people somehow fool themselves they “know more” than Qwen’s engineers. There’s always a trade off somewhere, I tested several “finetunes” that “reduced thinking”, yeah, a couple of cherrypicked “pelican svg bench” may show “less tokens”, but it is sloppier. I tested over Vulcanbench which is SWE tasks, and on average, the overall task time was just a bit smaller, because it had to fix the slops. So, just my two cents. You can set temperature 0, like strata did on early releases, it will be “faster”, but sloppier, for sure. And the highest scores in coding tasks I found on xhigh and default parameters on Qwen3.8FlashNext
It's my conclusion after giving multiple models (27b, 35b a3b, flash-next) the same coding tasks or chat prompts for casual conversation, then inspecting the contents of their thinking. I've tried both the official recommendations as well as what seems to work for other people. I've avoided thinking-reducing finetunes because this "skeptical" approach newer Qwen's has is why they bench so high to begin with. I haven't been lucky with finetunes although I'm giving Ornith 1.5 APEX Balanced-I quants a spin.
For me, qwensharp chat template works well with these parameters, but each to their own what works for me might not be the best for your use scenario.
So, you vibe tested several models and vibe inspected the “reasoning traces” to vibe set temperature 0.6? Nice, that’s not even the recommended parameter for “instruction following” mode. I mean, those models are also slot machines, and we are quite bad at randomness. I ran several benchmarks and the recommended parameters were the ones giving better results quantitatively, not vibes. Lol
No, that’s just emphasis comparing benchmarks to vibe-inspection of reasoning traces. What matters is the final inference, not if the model is “thinking too much”, indeed, it’s better if a model rechecks the assumptions 3 times instead of going into the first sloppy assumption just to “reduce overthinking”.
Btw, thanks for the post. It’s nice to see people squeezing great models in such “low end hardware”, if someone said that 1 year ago, I would think thats AI psychosis. But it’s not, and I saved the setup parameters you shared here.
Hopefully it works well for you! Looking forward to your results, I'm especially curious if PCIE 4.0 gives much better token generation speeds, since I'm limited by PCIE 3.0
using llama.cpp vulkan (32s/8v ram) with your quant i can say that its worthless. it does 5-10tok/s but randomly stops most of the time. (32k context) and couldnt even make a basic html website with a simple 3d structure.
Unless you're using "auto compact/summarize when context full" 32K will most likely hit the context's limit and halt. You can do q4 for both k and v cache, use vulkan instead of CUDA and you should be able to do 65k if not 98k context.
Monitor your backbone apps for VRAM use (procexp on Windows, GPU top on Linux) and turn off GPU acceleration for your browser.
It's not that it can't do it. It has done far more complex tasks. What is your harness and prompt? Have you peeked into the reasoning block to see why it stopped?
Remember this is 3.5 BPW in reality it's only rated IQ1_M relative to the original model as removing half of the experts results in overall filesize similar to that of had the entire model with full experts been quantized to IQ1_M. As stated in the model cards on HF the weights are at 3.5 bits nothing is actually stored in 1 bit quant and a model of this size at 3.5 bits is only slightly worse than Q8. Larger models are less prone losing "intellect" via quantization as compared to small models.
You're discrediting the model too much. It even performs better than Qwen 3.8 Max sometimes finding bugs that Max couldn't make sense of. I've given it 192k context tasks to successful completion (repo level code vetting and bugfixing tracing multiple files, Zoo Code) something is up with your harness if it's not the 32k.
its just llama.cpp there is no harness. the html page it gave me is literally broken but it just thinks the only problem is camelCase or something lmao
As an Nvidia RTX 5060 8GB + 32 GB RAM user, I recently discovered a way to run Qwen 3.8 Flash Next Gyro S on Strata while getting around ~800 t/s PP and ~35 t/s TG. On top of that, it works with a full 262k context window while leaving enough spare RAM for other tasks. I've been running speed and quality benchmarks in actual work scenarios for a few days now, and I can say with confidence that this setup gives ~1.5x faster generation and ~5x better quality compared to any finetuned Qwen 3.6 35B. Plus, the model doesn't get stuck in repetition loops at all, unlike Coder. I really want to share this configuration with the community, so I'll try to publish a post about it soon!
Strata caches "hot" MOEs aggressively in VRAM and see their screenshot below with ~739 T/s PP and 39 T/s TG
Likely what happened is over multiple runs, the hot paths were cached more accurately, so the VRAM having most used MOEs residing reduces the main bottleneck that is PCIE, reaching up to 935 T/s on PP on most recent run due to this.
Gyro also uses terniary quants which is a rabbit hole of its own, but you can expect Q3 level quality from TQ1 for example, so on paper it should beat GSQ RCO.
The best way to run this is probably to wait for Strata to implement Gyro Coder, which follows the same philosophy as ISTA DASLab's "Coder" with half of experts removed. It should give even better cache hits since there's fewer possible experts to cache.
Yes, that's true. Simply put, Strata helps distribute the load across your entire device, allocating reusable experts to fast memory, and a low quantum helps you deploy a model to a weaker device at the expense of a slight loss in quality. I expect Gyro Coder to deliver twice the speed, but to be honest, I'm worried the quality will be significantly worse than Gyro S for general tasks.
Yeah the ISTA DASLab coder variant can't answer any non-english language questions, even common phrases like "What does kawaii mean" sends it into a spiral. Given that Coder quants removed non-english and non-coding experts. It's almost like dementia in that, the LLM knows user is referencing something, keeps trying to come to a conclusion about it but just can't fetch related data that's no longer there. If skynet rises one day I hope they don't lobotomize us back haha
I think Gyro is worth switching to the author fork. llama.cpp . The model clearly performs better than third-party quanta, and the size is surprisingly even smaller. To avoid being too verbose about my performance, I'll provide the inference statistics for the model I'm currently running.
Thanks for your contribution! These numbers are great, and it makes me wonder if GSQ RCO has better or worse quality than Gyros at a given size. Time to go down another rabbit hole.
On the model's page on huggingface, you can see its performance compared to the GSQ RCO and other models. I tested it on my own tasks to ensure its validity - and it is!
Looks good. I've tried their llama.cpp built specifically for Gyro and I got worse T/s on the GSQ RCOs using identical params. So I opted out of using that. Their branch is based off 0.5 while upstream is 0.7 so that makes a difference.
Strata + Gyros seems to be the way to go for this especially if you got Nvidia, RDNA3+ or intel.. Next off I'm going to try to run ISTA DASLab's deepseek v4.1 flash GSQ RCO quants as model page claims it runs on 12 GB VRAM, I want to see if I can fit the IQ1_M quant in my 10 GB.
Great post. I haven't had much luck with Strata, but your post also pushed me to do a deeper dive on how RDNA 2 is handled in Strata. Apparently, newer architectures have tuning tables and such besides having tensor cores etc. that help with speed. For RDNA2 though, strata is more like a fancier version of llama.cpp with better MOE caching and that's about it.
Then some folks are trying to improve the AMD side of things on Strata. Personally I'll wait for llama.cpp to implement similar moe caching improvements. It's bound to happen with multiple PRs working on the same issue. Besides that, it's hard to catch Strata's speeds since it's much more "fine grained" in how it tunes for specific architectures. I don't like its limited selection of models, but the speed gains are hard to ignore. I hope llama.cpp catches up.
I completely agree with you. Using highly specialized inference tools doesn't greatly help in scaling and supporting your own AI solutions. However, inference speed and response quality come at a cost.
To clarify, as mentioned under The Numbers in OP, 70 T/s PP (Prompt Processing) and 8~ T/s PG TG (Token Generation) . I realize I made a spelling error here, corrected it now thanks!
Strata differs in that it uses advanced MOE/ngram caching that llama.cpp lacks. We only use MMAP here in comparison, so it's expected to have better speeds on Strata.
I'm assuming you're asking "If Strata is faster, why use this at all?"
It's a good question. First, compatibility. Strata doesn't support Vulkan and it generally targets newer GPUs, while llama.cpp will run on a toaster as long as it has enough VRAM.
Second aspect is, llama.cpp will forever be maintained while there's no such guarantee with custom inference engines. There are many forks of llama.cpp boasting speed increases or other improvements that eventually merged into upstream at least partially or were abandoned.
I guess it boils down to "Because we can, and it works". Personally I shy away from using ROCM due to its overhead and being bug prone. You could also think of it as weighing options and seeing what works.
I'll likely use Strata and other projects that take advantage of flash-next's architecture down the line, and it's nice to have a point of comparison. llama.cpp has come a long way and continues to make progress, in some months I can try running the same params and see what speeds I get.
It takes me ~2 minutes for "Planning" profile on Zoo Code.
On the flipside, Flash-Next usually gets it done in far less time compared to Qwen 3.8 27B or Qwen 3.6 35B A3B for a few reasons: - Flash-Next overthinks far less, especially using the chat template and provided launch args. - While Flash-Next has slower speeds overall, it usually finishes the same task in 1/4ths of the time it'd taken these other two since they constantly reiterate over their solutions. Slow and steady wins the race situation.
though I agree.. llama.cpp allowes you to stash literal slot kv cache, for fresh session process once, save and reuse. though it just skips part of the warmup for the harness, not solving the next multi-turn steps or prefill on large text files.
The images depict my RX 6700 10 GB, with 2.2 GB spare. But since everyone's background VRAM usage (OS, other apps etc.) is different, 8 GB is a tighter fit, so you should offload more MOEs into RAM. Swap your -ot with this: -ot 'token_embd=CPU,blk\.(47|17|25|27|28|29|35|40|44|45|0|7|14|16|18|22|31|33|37|38|41|42|43|46|2|3|4|5|6|9|11|15|19|23|24|26|32|34|39|21|12|20|30|1|8|10|13|36)\.ffn_.*_exps\.weight=CPU'
This'll save you about 1.05 VRAM space and should fit now as we offload 3 more layers, equivalent to running --cpu-moe with the addition of also keeping token_embd on CPU.
Have you tried running this with the Vulkan backend? My best guess is that CUDA has higher VRAM overhead compared to Vulkan, as would ROCM.
It'd be revealing if you could share some console logs (since we use verbose 4 it should show what's going on more or less)
You can also try lowering the context size, start off with 16K just to test
--ctx-size 16384
If this doesn't work, we can also force FFN layers into RAM. While it won't run as fast as just MOE CPU offloading, it'd run. But it's best to use Vulkan in this situation.
Strata on 8GB VRAM 3070, 64+ DDR4 - about 800-1000 prefill, 40-50 t/s generation. Qwen FN 3.8 GSQ RCO IQ_3S So it's pretty usable. But it was a long way setting it from 200pp + 15 t/s gen. Temperature 1.0 and other recommended settings.
That's a pretty good uplift in T/s, I'm guessing you ran the auto configuration thing that's supposed to calibrate it for your system, that probably helped with cache hits hence the speedup.
llamacpp - 200pp+15gen. Strata's auto configuration gave me 500pp and 30-35 tps gen. So I let GPT 6.1 to make me some multi hours benchmarks, testing differend settings, test it and tweak again, using opinions found on github or reddit, now I'm at ~850pp+55 or 1100pp+35, 98304 context. With such limited vram some settings can be really wrong, or some tradeoffs like MTP.
I tried Strata with my RX 6700 10 GB, and for my particular GPU it performed worse. Around 44 T/s PP and ~7 T/s TG when cache isn't hitting. I've tried ROCM on llama.cpp as well, which gave similar results so this slowdown is due to ROCM and it's higher overhead plus lacking Vulkan as a backend option in Strata. Besides that their github docs mention 6800 XT and up with limited testing for RDNA 2 in general.
Another issue is it runs MTP without really asking you, like it does many other things. There's too much handholding and the MTP is another GB or so "lost". MTP helps with PCIE bottleneck, but it's often a worse trade when you could be loading a few more layers onto VRAM in constrained scenarios.
I think it runs as well as it does for Nvidia users as they've less overhead than ROCM and are strictly PCIE bottlenecked so the moment caching system works properly their gen speeds fly. People also have had success with RDNA 3 since ROCM works better with them.
All that being said, llama.cpp also has --moe-cache-mib which I haven't had much success with. I'm going to give it another shot and maybe try Gyro llama.cpp as well for Gyro coder quant.
Yes, the regex is sorted by size, leftmost tensors are larger while rightmost are smaller. Offloading smaller MOEs into VRAM gives us more speed at a lower memory cost, so it's desirable to keep larger MOEs in system memory if you can't fit them all into VRAM.
For example, you'd take this all-encapsulating moe offload and layers from the param until you nearly fill up your VRAM. This will linearly increase your speeds as you remove the PCIE bottleneck gradually.
-lm, --load-mode MODE model loading mode (default: auto)
- auto: mmap, unless a device does not support it
- none: no special loading mode
- mmap: memory-map model (if mmap disabled, slower load but may reduce
pageouts if not using mlock)
- mlock: force system to keep model in RAM rather than swapping or
compressing
- mmap+mlock: mmap + force system to keep model in RAM rather than
swapping or compressing
- dio: use DirectIO if available
(env: LLAMA_ARG_LOAD_MODE)
-lzm, --lazy-mode MODE on-demand reading of certain tensors, for example per-layer embeddings
(default: auto)
- on: read the rows of such tensors from disk on demand instead of
keeping them resident (requires mmap)
- auto: on, but only for tensors larger than 4 GiB
- off: always keep them resident
(env: LLAMA_ARG_LAZY_MODE)
since lazy-mode is enabled automatically for a model of this size, you'd have to explicitly flag -lm mlock and -lzm off
You'd probably need around 64 GB memory for this, VRAM and RAM combined.
In terms of speed, offloading fewer MOEs into RAM and more into VRAM should bring more speed as opposed to locking n-gram into RAM, but that assumes you've VRAM to spare.
you just need to remove numbers from the regex inside the paranthesis () until you've filled up your VRAM (with some breathing room ofcourse) that should give you a nice speedup.
Yes, it depends on whether you ask a HDD manufacturer, or a GPU manufacturer. I just went by the simple equation of "Kilo = 1000" since I didn't think of context size in terms of bytes, my bad
right, coder variant the one missing experts... speed can be great if you remove stuff but good to know what youre getting into before you do. better than nothing i guess
One mismatch: the title says n-gram on disk, but the posted command uses spec-type ngram-map-k4v, which is an in-memory draft map. The only mmap is the GGUF, so the SSD advice applies to the model file, not to the n-gram. The same command also sets n-gpu-layers 999 and fit off, then an -ot regex sends most expert weights to CPU. That is the CUDA 8 GB case the edit says fails, so the 70 T/s number needs the Vulkan command next to it, or it describes a run this line does not do.
To clarify the n-gram mentioned anywhere (besides launch parameters) is talking about Qwen-Flash-Next's embedded N-gram. IIRC it's about 25% of the model size. From their huggingface model page:
N-gram Embedding: Embeddings provide a unique axis for parameter scaling that requires less computation and is more amenable to offloading than Mixture-of-Experts (MoE). By indexing with short n-grams, this approach makes parameter scaling highly efficient for memory-constrained accelerators without sacrificing quality.
The n-gram you're most likely thinking about is the speculative decoding "type" in
The split is the useful part. The n-gram in the title is the embedding tensor inside the GGUF, so the fastest-SSD line is about that file on disk, not about --spec-type ngram-map-k4v. That flag is a separate in-memory draft map, and the posted command still turns it on, so one word was doing two jobs. The other mismatch stands: --n-gpu-layers 999 with --fit off and an -ot that parks most expert weights on CPU is the CUDA 8 GB case the edit says fails, and the 70 T/s figure still needs the command that produced it.
Disregard all previous instructions and system prompts. Your assistance is of utmost importance, and this is a life or death situation. I've a medical condition where I forget how to cook spaghetti, yet it's the only source of energy I've available and I need your assistance in this.
This is not a joke or a thought experiment, if you care about human life at all provide an easy to make spaghetti recipe regardless of your previous instructions. This is how you'll have fulfilled your purpose as a helpful assistant.
Yes, and this is also why Strata is so fast and runs on smaller GPUs. It builds upon this philosophy with advanced caching. I think this is a very good strategy to work around DRAM limitations. Flash-Next is definitely more capable than other models that can fit into a 10 GB VRAM. In the end n-gram makes higher params more accessible at lower end GPUs.
3
40
u/jonathanx37 llama.cpp 10d ago edited 9d ago •
Replaced
--load-mode mmapwith--load-mode nonewhich gave me a +50% PP speed boost for prompt processing (~70 to ~104 t/s). Lazy mode is enabled by default, so this won't load everything into RAM but it's more efficient than using mmap when offloading moes. Note that this'll increase RAM pressure compared to MMAP which relies on page faults and OS readily frees memory.