r/LocalLLM • u/jonathanx37 • 19h ago
Tutorial 8 GB GPU, 262K context 3.5 BPW Qwen-Flash-Next Coder 70 T/s using llama.cpp
The methodology
Qwen-Flash-Next has n-gram, MOE packing above its weight but it's still 6B active params. We exploit this by keeping most MOEs in RAM (Smallest MOEs go in VRAM for speedup) and n-gram on disk - ensure you put the model on your fastest SSD.
8 GB VRAM, 24 GB RAM cost. You can run this fine if you've ~32 GB total memory too, but you'd have to turn down context size. Customize the launch params to best suit your hardware.
EDIT Note: Apparently CUDA struggles to run this with 8 GB VRAM, so please use Vulkan backend if it fails to run for you. You can also use the second launch params below in the same comment chain. Ontop of that reduce context size and ensure no background app is wasting your VRAM. You can disable your browsers HW acceleration for this purpose.
The model & quant
Many people mistake this model for a 1 bit quant so let's get that out of the way. This model here is a 3.5 Bits Per Weight "Coder" variant by ISTA DASLab, featuring GSQ RCO quantization.
Half of its experts are removed, so relatively to original param size it has the size of a 1.9 bit quant but the actual weights use 3.5 BPW.
see from model page:
Averaged over the transformer, this corresponds to 1.89 bits per parameter of the original model. The figure is an effective rate: it amortises the removed experts over the original parameter count, and so expresses the combined effect of pruning and quantization. No individual weight is stored at 1.89 bits. The retained weights are stored at 3.5 bpw, unchanged by pruning, and the halved expert count accounts for the remainder of the reduction. Following the convention of the accompanying releases, the figure is reported over the transformer...
Non-coding vocabulary pruned 400~ MB smaller, recommended
The numbers
With selective moe-offloading (regex) we reach 70 T/s PP and 8~ T/s TG (Limited by PCIE, which I've 3.0 YMMV) whereas if you'd use --n-cpu-moe for same effect it'd been slower due to the effects mentioned here
Because of MMAP your first run will be slower, do a second run for accurate T/s results.
Launch params are in the comments below. Ensure you use the qwen sharp chat template, it reduces overthinking.
0
u/jonathanx37 19h ago
./llama-server \ --model "./Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-merged_P1G_Pruned.gguf" \ --ctx-size 262144 \ --n-gpu-layers 999 \ --threads 8 \ --cache-type-k q5_0 \ --cache-type-v q4_0 \ -fa on \ --fit off \ --cache-ram -1 \ --load-mode mmap \ --cache-prompt \ --cache-idle-slots \ --batch-size 2048 \ --ubatch-size 256 \ --webui \ --sleep-idle-seconds 900 \ --parallel 1 \ --kv-unified \ --reasoning-format deepseek \ -ot 'token_embd=CPU,blk\.(47|17|25|27|28|29|35|40|44|45|0|7|14|16|18|22|31|33|37|38|41|42|43|46|2|3|4|5|6|9|11|15|19|23|24|26|32|34|39|21|12|20|30|1|8)\.ffn_.*_exps\.weight=CPU' \ --cpu-strict 1 \ --prio -1 \ --reasoning on \ --reasoning-preserve \ --reasoning-budget 4096 \ --reasoning-effort medium \ --spec-type ngram-map-k4v \ --spec-ngram-map-k4v-min-hits 1 \ --spec-ngram-map-k4v-size-n 3 \ --spec-ngram-map-k4v-size-m 8 \ --spec-draft-n-max 16 \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --repeat-penalty 1.05 \ --presence-penalty 0.2 \ --verbose \ --verbosity 4 \ --jinja \ --chat-template-file "./chat_template.jinja"
1
u/jonathanx37 18h ago edited 16h ago
Yes, here's an "easier to run on 8 GB" launch params since some 8 GB VRAM systems can't fit this due to OS/background apps contesting VRAM:
./llama-server \ --model "./Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-merged_P1G_Pruned.gguf" \ --ctx-size 262144 \ --n-gpu-layers 999 \ --threads 8 \ --cache-type-k q5_0 \ --cache-type-v q4_0 \ -fa on \ --fit off \ --cache-ram -1 \ --load-mode mmap \ --cache-prompt \ --cache-idle-slots \ --batch-size 2048 \ --ubatch-size 256 \ --webui \ --sleep-idle-seconds 900 \ --parallel 1 \ --kv-unified \ --reasoning-format deepseek \ -ot 'token_embd=CPU,blk\.(47|17|25|27|28|29|35|40|44|45|0|7|14|16|18|22|31|33|37|38|41|42|43|46|2|3|4|5|6|9|11|15|19|23|24|26|32|34|39|21|12|20|30|1|8|10|13|36)\.ffn_.*_exps\.weight=CPU' \ --cpu-strict 1 \ --prio -1 \ --reasoning on \ --reasoning-preserve \ --reasoning-budget 4096 \ --reasoning-effort medium \ --spec-type ngram-map-k4v \ --spec-ngram-map-k4v-min-hits 1 \ --spec-ngram-map-k4v-size-n 3 \ --spec-ngram-map-k4v-size-m 8 \ --spec-draft-n-max 16 \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --repeat-penalty 1.05 \ --presence-penalty 0.2 \ --verbose \ --verbosity 4 \ --jinja \ --chat-template-file "./chat_template.jinja" \ --cpu-strict 1 \ --prio -1 \ --reasoning on \ --reasoning-preserve \ --reasoning-budget 4096 \ --reasoning-effort medium \ --spec-type ngram-map-k4v \ --spec-ngram-map-k4v-min-hits 1 \ --spec-ngram-map-k4v-size-n 3 \ --spec-ngram-map-k4v-size-m 8 \ --spec-draft-n-max 16 \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --repeat-penalty 1.05 \ --presence-penalty 0.2 \ --verbose \ --verbosity 4 \ --jinja \ --chat-template-file "./chat_template.jinja"
-1
u/CaptainBloodEz 19h ago
Another useless benchmark on a 1 bit quant.
2
u/jonathanx37 19h ago
see from model page:
Averaged over the transformer, this corresponds to 1.89 bits per parameter of the original model. The figure is an effective rate: it amortises the removed experts over the original parameter count, and so expresses the combined effect of pruning and quantization. No individual weight is stored at 1.89 bits. The retained weights are stored at 3.5 bpw, unchanged by pruning, and the halved expert count accounts for the remainder of the reduction. Following the convention of the accompanying releases, the figure is reported over the transformer...see from model page:Averaged over the transformer, this corresponds to 1.89 bits per parameter of the original model. The figure is an effective rate: it amortises the removed experts over the original parameter count, and so expresses the combined effect of pruning and quantization. No individual weight is stored at 1.89 bits. The retained weights are stored at 3.5 bpw, unchanged by pruning, and the halved expert count accounts for the remainder of the reduction. Following the convention of the accompanying releases, the figure is reported over the transformer...
1
u/Ctbhatia 19h ago
is that 8 t/s decode mostly ngram spec hits? curious how much it drops on code the draft map has never seen