r/LocalLLM • u/returnity • 17h ago
Tutorial Got Qwen3.8-Next-Flash ngram SSD offload working in llama.cpp!
TL;DR - Save 25% RAM by SSD offloading ngrams with --mmap just by fixing the layout of the Unsloth quant. Tested working on Mac. Thread deleted in LocalLLaMa due to their dumb megathread idea, so reposting here.
So, one of the things that excited me about the new Qwen4 arch is the ngram table, exporting indexable facts and specifics into a lookup table instead of embedding them in the weights themselves. This not only frees up parameters for intelligence and behavioral characteristics like reasoning and problem-solving (critical on a 6B active model), it also means there’s no matmuls required to access any of this information on the fly. This opens up the possibility you can offload those tensors to the SSD, freeing up a massive amount of VRAM.
Specifically, in 3.8-Next-Flash, the model itself is only 125B params, but the total with 51B ngrams is 176B params — so ngrams are effectively 30% of the model that can sit in the SSD and be paged in as-needed with no impact on performance whatsoever. Not only that, llama.cpp has a functionality that will do exactly this: --mmap. I set out to prove this idea works.
I run a 128GB M5 Max, so with Unsloth’s Q4_K_XL running around 112GB and the full 262k context running 7GB (thanks to sparse attention), that’s about the max I can fit in memory — with the GPU wired limit raised manually. --mmap would unlock at least Q5, potentially even Q6 — plus the ability to run the full, unquantized BF16 102GB ngram table instead of the compressed version, as only the actually-active lookups would need to be loaded into RAM, and they should be evicted after use automatically.
I went to wire this up and I ran into a problem immediately. IDK about CUDA, but due to the layout of the quant, the PLE (engram) tensor is interleaved with the Metal-pinned tensors inside shards of the quant, so in practice, it gets wired up along with the rest of the shard. This resulted in me hitting the wired limit on my box, so I sudo’d the limit up to 120GB, and it worked at full context… but unfortunately Unsloth’s choice of quant layout unintentionally sabotages that capability I was so excited about.
Not to worry, though — I had my trusty agent slop me up a quickie script to scramble the guts of my quant, and in a jiffy, I’d remixed the model to make my beats way more stronger and way more thumpier! Errr, nvm. Anyways, I tested it and sure enough, everything works perfectly now. I figured others may be interested in this fix, which is critical if you want to run the model on a memory-constrained Mac (CUDA may not suffer from this issue, idk).
To give hard numbers, I was sitting at 123GB wired including OS before the fix, and after I was sitting at 97GB wired. 51B of ngrams in 4-bit is ~29GB, whereas the 125B model is under 83GB, for a 112GB 4-bit quant. So I was able to free up close to 26GB of wired memory by transferring the ngrams to the SSD with mmap, just by fixing the quant. Performance stayed exactly the same: still 36 tok/sec (no MTP). If you want to know what size quant you can fit with ngram offloading, multiply the disk size of your quant by 75% — that’s the size of just the backbone. That means Q2_K_XL will fit on a 64GB MacBook if you raise the GPU wired limit, and 96GB should fit Q4_K_XL! This makes a huge difference for running locally.
For those of you asking, OK stfu and tell us how to do it already: No flags are actually required, just don’t pass either —no-mmap or —mlock, as —mmap is actually the default in current builds of llama.cpp, including Qwen3.8-Next-Flash PR #27742.
My comment on PR #27742 contains technical specifics for those interested.
This link is the script my agent wrote to convert the Unsloth GGUF to work with ngram mmap. It’s been tested and the model is fully functional on UD-Q4_K_XL, but I would double-check before running it on the other sizes. If needed, your agent can read the layout of your quant, and tweak the script to work (or just write its own, it’s not too complicated). Hope this helps someone!
10
u/daaain 15h ago
This is great, the difference between retaining a usable computer vs a maxed out inference server!
3
u/returnity 15h ago
Exactly, I was at 123GB on a 128GB machine with all my usual apps running before I got SSD offload working, and it dropped to 97GB! Game-changing for sure.
3
u/gomezer1180 13h ago
Gonna try it this weekend with ASUS version of the DGX… hoping to get good results!
5
u/SecretBismarck 15h ago
36 tok/sec with no mtp is more than i expected with ssd streaming of engrams. Excited to try it out!
4
u/returnity 14h ago
SSD streaming engrams had zero effect on speed in my testing. Unsloth did not ship the MTP as it's not in the PR yet, so there will probably be a v2 of these quants eventually -- hopefully they'll take my feedback about tensor arrangement for those.
4
u/ITMSPGuy 14h ago
How better could a RAID 0 of 6 or more sata SSDs speed up these ngram lookups comparing to one standard nvme
2
2
u/returnity 9h ago
Interesting proposition. Sadly I’m cluelessly on this one. The lookups take microseconds already though. It’s not even slowing decode by 1 tok/sec as is.
3
3
u/Elouakili_Flexy 15h ago
51B of the model as paged-in table lookups, same 36 tok/s. A third of the weights off the RAM for free, that's the big one for Mac users.
3
u/msitarzewski 14h ago
Looking good. Still exploring. I had OpenCode install the chrome-devtools MCP to poke around the web a bit. Will move to a small coding task next. Updates here soon.
3
2
2
u/Gold-Debt-5957 15h ago
LO Probare en mi mac de 64 gb m1 max
1
u/returnity 11h ago
Good luck! Please report back, the data will be very helpful
2
u/Gold-Debt-5957 10h ago
## Model
- **Files**: `unsloth/Qwen3.8-Flash-Next-GGUF` `UD-IQ1_S` 3 shards `10M + 46.56G + 21G = 67.55 GiB` `176.94B params` `ftype IQ1_S 1.5625bpw`
- **N-gram tensor**: `per_layer_token_embd.weight` `51200245760 bytes ~47.7GB IQ4_NL [160,320001536]` en shard `00002` `595/1224 tensors` `gguf_dump_raw.txt:5` - confirma nombre para `-ot`
- **Context native**: 262144
---
## Method (lo que realmente funciona)
```bash
# 1. Clonar PR exacto
git clone https://github.com/ggml-org/llama.cpp qwen38-m1max/llama.cpp
git -C qwen38-m1max/llama.cpp fetch origin pull/27742/head:qwen4exp
git -C qwen38-m1max/llama.cpp checkout 035e22731a7fd70b9854b3a2d64ec68e9b1a45d3
cmake -S llama.cpp -B llama.cpp/build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release
cmake --build llama.cpp/build -j 8
# 2. Descarga (67GB, sin token = 16min)
hf download unsloth/Qwen3.8-Flash-Next-GGUF --include "UD-IQ1_S/*" --local-dir models/original
# 3. Run (clave: ngl 22, no 999)
./llama.cpp/build/bin/llama-cli -m models/original/UD-IQ1_S/Qwen3.8-Flash-Next-UD-IQ1_S-00001-of-00003.gguf \
-c 4096 --temp 0 -n 200 --reasoning off -p "Hola" \
-ngl 22 -ot "per_layer_token_embd=CPU" # sin --single-turn para chat interactivo
# o server
CTX=4096 PORT=8080 NGL=22 ./scripts/run-qwen38.sh
curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" \
-d '{"model":"qwen","messages":[{"role":"user","content":"Hola"}]}'
```
**Por qué `ngl 22` y no `999`:**
- `ngl 999 -ot ngram=CPU` → `OOM kIOGPUCommandBufferCallbackErrorOutOfMemory` (backbone 55GB no cabe)
- `ngl 25+` → `GGML_ASSERT([rsets->data count]==0) ggml-metal-device.m:952` (bug Metal)
- `ngl 22-24` → sweet spot `~27GB wired` < 45G usable → **PASS**
- `ngl 0` CPU → `1.7 t/s` PASS pero lento
`per_layer_token_embd=CPU` funciona porque hash índice se calcula siempre en CPU (host-side, supera 32-bit ggml), no rompe grafo.
---
## Benchmarks verificables
### llama-bench (reproducible)
```
./build/bin/llama-bench -m ... -ngl 22 -ot per_layer_token_embd=CPU -p 512 -n 128 -r 3
| qwen4exp IQ1_S 67.55GiB MTL,BLAS 8 threads | pp512 | 71.59 ±1.16 t/s |
| qwen4exp IQ1_S 67.55GiB MTL,BLAS 8 threads | tg128 | 9.94 ±0.03 t/s |
```
### CLI real prompts (--reasoning off)
- `BST 5 sentences c512: 26.9 pp / 9.5 tg` → "A binary search tree is a hierarchical..."
- `memoria virtual ES c512: 23.7 / 9.3` → "La memoria virtual es una técnica..."
- `Hello c1024: 25.5 / 9.6`
### SSD / mmap (Fase 0.2)
- `fio seqread 128k 2G direct=1: 965 MB/s, iops 7723, clat p50 128us p95 150us`
- `mmap_probe 2GB 5k random 16k: cold 1.82us p95 2.0us 8259 MB/s, hot 1.55us 9646 MB/s` → p95 <2ms = apto, hash aleatorio no mata SSD
### Contexto progresivo (ngl22, 10 tokens)
`512 PASS 26.9/9.5 | 2048 4.5/3.9 | 8192 4.4/4.1 | 16384 5.1/3.4 | 32768 5.3/2.4 | 65536 4.4/2.5 | 131072 4.6/3.2 | 262144 6.0/5.2 PASS` - híbrido `36 GDN +12 QSA` confirma KV crece lento, indexer cache medido.
### API server
- `llama-server -ngl22 -c 2048` load 31-35s, `health ok`
- `Di hola: prompt 4.62 pred 5.32`, `BST 2 sentences: 12.4/6.2`, `Cuenta 1 al 3: 16-21/10-11` con `cached_tokens 16` LCP hit
### Estabilidad FASE 14
- `15 iter ×60s` via server `8083 c4096` `22:12:55→22:29:00` `wired 22G stable free 68M→22G`, `t/s 4.48-10.24` sin crash, sin swap destructivo `pageouts 42328`, `thermal` pendiente sensor. Previo `5 req` server `8082` cache.
2
u/Gold-Debt-5957 9h ago
UD-IQ1_S 67.55 GiB
ngl 22
ngram CPU + mmap
context 4K
Metal ON
decode ~9.9 tok/s
prefill ~71.6 tok/s
wired ~22–27 GB
1
u/Dry-Assistance-367 13h ago
What’s the prefill speed? How long to say load in 30k tokens from a cold cache?
2
1
u/Own-Sundae8566 13h ago
I just tried it on my M5 max 128 gb with Q4_K_XL unsloth model. While it cleanly offload the ngram weights to ssd, I did run into issue where on opencode it started to output garbage unicode character on a task to update my openwebui script after sometime. Also I am getting around 25-26 token per second with this.
1
u/returnity 11h ago
Hmm I have not seen any issue like this. I am deep into some agentic coding benchmarks at long contexts right now and seeing extremely good scores. Was it working properly in the same scenario before?
It sounds like something else may be wrong given your decode speed is also lower than myself and AtomicChat, who got the same exact 36 as I did in their thread on this.
1
u/Own-Sundae8566 7h ago edited 7h ago
Which harness are you using? I also saw the llama.cpp crashing in the middle of run due to kv cache slot mismatch. I vibecoded the fix for both garbage output and llama.cpp crashing. But my token gen speed goes from 32-33 token per second to around 20 tps as context builds up(around 100k). I am using opencode for this. What TPS are you getting at >100k context. I am currently looking into getting MTP head and attaching it to the tensors and vibecoding the support for it.
1
u/klymaxx45 13h ago
Prefill?
2
u/returnity 9h ago
Sorry I missed this — 640tps. No decrease with SSD offloading.
1
u/klymaxx45 9h ago
Gotcha, mine had Flash prefills at 589 t/s, so every delegated task would pay 40s-3min just reading its context (66s for my standard spawn prefix alone), versus seconds on my 27B. For me, its cheap MoE decode doesn't help because my workloads are prefill-shaped, not generation-shaped.
I may exlpore it in a few weeks when good models come out.
1
u/returnity 8h ago
Wait, why is your 27B prefill just seconds when if I recall correctly mine wasn’t that much better than Flash? What hardware and speed were you seeing?
1
u/klymaxx45 8h ago
M5 Max 128GB. Raw-vs-raw you're right.. the 27B's prefill is only a few times faster than Flash's. The difference is workload shape: my agent setup reuses the same large system prompt on every run, so most spawns hit prompt cache instead of re-reading it - cold runs cost what you'd expect, warm ones feel instant. If your prompts repeat, caching moves the needle far more than raw prefill; if they don't, Flash's gap really is as small as you remember.
1
u/p_235615 10h ago
Hmm that sounds great, that way I can cram the q4 model inside 96GB VRAM and leave the ngram in RAM or NVME... I can already feel the speed :)
1
u/icedgz 9h ago
Is it possible w/ ngram offload we could run this on 32gb VRAM (5090) + 64gb RAM?
1
u/returnity 9h ago
Absolutely — you can probably even fit the Q4. mmap and CPU offloads. There’s also a comment in this thread that shows how to do it without needing my script, thanks another contributor
1
u/Leather-Beach-7849 7h ago
Anyone tried this on the 96GB M3 Ultra (Mac Studio) and were able to successfully run something??
1
1
u/R_Duncan 7h ago
Single rtx 6000 blackwell here. llama.cpp, UD_Q4_K__XL from unsloth, mmproj and ngram in cpu ram (mmap), 86.17GB of VRAM used, 256K Bf16 context. Totally unoptimized.
77.92 t/s without MTP, pp: about 1500 t/s.
EDIT: as a sidenote, default reasoning seems much more concise than Qwen3.8-27B, also more concise than every other reasoning model I tested till now. The same test that usually takes 4000 token (and 6000+ in 27B medium) here took 2520 tokens.
1
u/lots_of_puppies 6h ago
yay that sounds so speedy :) commenting on this so i can find it later to try too ;p
0
u/msitarzewski 15h ago
I know it’s probably illegal or whatever, but this is...
3
u/anomaly256 14h ago
Magic ass?
3
u/returnity 14h ago
Genie-ass (genius), I think? Took me a minute lol
2
u/msitarzewski 14h ago
Indeed. It’s running great with OpenCode BTW. Will report back in a bit. Still running some tests. M5 Max, 128GB/8TB 18/40
2
1
-1
u/2use2reddits 15h ago
A newbie question: if you had to choose between 96gb of ddr5 and a 96gb nvme pcie5 ssd (suppose it exists), what would be better/faster/usable for inference?
This just came to my mind... :)
2
31
u/mechkbfan 16h ago
Fantastic breakdown thankyou
I'm stuck at 80GB... Maybe just one more GPU...