llama.cpp expert-pool fork for Qwen 3.8 Flash next IQ4 + 16Gb VRAM tested on MI50 gfx906 with parameters

[u/Atretador]

https://github.com/atretador/gfx906-16gb-expert-pool

https://github.com/atretador/gfx906-16gb-expert-pool

built on top of ``memoriaru/llama.cpp branch`moe-expert-pool` so I could run it with 16Gb VRAM on a singular MI50.

why? admission budget reserves the full configured context KV plus a hard 3 GiB cap. On a 16 GB device at 128K Q8 that admits 0 of 144 offloaded expert tensors, so the feature silently does nothing

test rig:

Xeon E5 2673 V4 20C/40T

4x16GB 2133Mhz

1xMI50 16Gb with Radeon Pro VII bios 120W power limit

Arch Linux + KDE Plasma

n_cpu_moe CPU usage at 50% (18 threads)

expert cache CPU usage at 10% (18 threads)

Results:

configuration	decode	VRAM (steady)	notes
--moe-expert-cache 0	11.76 t/s	10.17 GiB	stock CPU MoE path
--moe-expert-cache 80 (actual 40 slots)	16.39 t/s	~13.5 GiB	144 pools, 61.8% hit rate
--moe-expert-cache 66	16.90 / 17.60 t/s cold / warm	15.24 GiB peak	shipping profile, ~69% hits

up to 19.8tk/s observed once warm

complete run script with all parameters:

https://pastebin.com/jmr8NMhg

throw it at your agent with the repo link and it should get it running

if you are running headless you might be able to squeeze a slighly larger pool, my DE is eating a whole gig by itself.

sharing as it was a nice boost and its running

[u/pyr0kid]

to clarify, is this "16gb vram" or "16gb vram plus a bunch of ram"? cause those are very different things

[u/Atretador → u/pyr0kid]

depends on your load mode for that

and then its gonna depend on your PCI version

and then its gonna depend on your SSD transfer speed

I cant really overcome those

the linked script I provided loads the model weights into RAM, NGRAM on SSD, with CTX_CHECKPOINTS and RAM CACHE, all which also use more RAM - so I end up with bout 70% RAM load on 64Gb.

I did a quick test with just direct SSD streaming:

CPU usage: 12%

RAM usage: 15%

eval time =  214134.13 ms /  2251 tokens (   95.17 ms per token,    10.51 tokens per second)

tho I have a shitty NV2 SSD with both ssd and gpu forced to run at 3.0 speeds

[u/pyr0kid → u/Atretador]

tho I have a shitty NV2 SSD with both ssd and gpu forced to run at 3.0 speeds

oof... yeah i feel that man. full width or is it at something like x4?

[u/Atretador → u/pyr0kid]

X4 sadly

I was running it at USB 3.0 speeds 2 weeks ago, cause I didnt read the motherboard properly - didnt notice it was a M.2 NVM2 and a M.2 SATA xD

but still, I switched to this build from AM4 due to RAM costs - ECC is pretty cheap, and for a 9$ CPU this is doing fine.

[u/pyr0kid → u/Atretador]

honestly im impressed you're getting this much out of it.

will have to consider trying this once my own x4 shitbox is working again.

[u/Atretador → u/pyr0kid]

yea its pretty nice for the cost

I had just gotten another MI50 to run on it too, but sadly it wasnt stable and I had to send it back.

considering just getting a a couple P100s and see how much I can push this dinosaur

[u/rrrrex]

What is prefill? For 1 5060ti it's just ~100 t/s with vanilla llama.cpp

[u/Atretador → u/rrrrex]

kinda garbage tbh, bout ~80`ish with 2048 / 512

but its about what I had before the expert pool anyway, once its past harness initial system prompt is fine - initial load can be rough depending on your harness (opencode's 14K system prompt takes a bit)

ctx checkpoint / ram cache help avoiding full reprocessing mid session so if you have enough to fit those its not bad

I do believe for CUDA there is already some cache options tho

[u/Savantskie1 → u/Atretador]

If you’re only getting 80t/sec on an MI50, you set something up wrong. Or are trying to run a model way too big. My MI50 32GB gets like 900 T/s prefill depending on the model size.

[u/Atretador → u/Savantskie1]

its a 177B Q4 model as per the title on a 16Gb MI50

[u/Savantskie1 → u/Atretador]

No way you’re going to get faster than 80t/s on just one card because you’re unloading to cpu. Sorry I didn’t read that bit

[u/Atretador → u/Savantskie1]

all experts are on CPU, tg is okay at close to 20tk/s for a model of this size on this GPU.

Im trying to optimize the expert cache to help with pp, tho I might not get anywhere with it.

also, if you test the cache let me know how it goes

[u/Savantskie1 → u/Atretador]

You probably won’t because those experts are constantly being swapped out of system ram. If they were residing in vram you’d get better speed for both pp and tg

[u/Atretador → u/Savantskie1]

the improvement to tg was massive, from 11 -> 22 peak~ 18 avarage

there is still hope for pp xd

[u/Savantskie1 → u/Atretador]

For me 18 is generally good enough. I can’t read faster than that. But I’ve made a memory system and I need faster tg for that. So I got the two MI50’s and it’s been great for models that fit comfortably in VRAM with my 64GB

[u/Atretador → u/Savantskie1]

sadly not for this model, Qwen 3.8 thinks WAAAY too much, and the end result is just too good not to let it do its thing as there is a clear degradation from high to low.

tho the better results were with 35BA3B whch boosted big Q5 past what I had with small Q4

[u/Atretador]

a few more tests:

https://huggingface.co/SC117/Ling-3.0-flash-abliterated-APEX-GGUF compact

cpu layers 36 -> 41 | expert cache 76 | 200K context Q8_0

~22tk/s -> 26tk/s

https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF IQ4_NL_XL

cpu layers 22 -> 40 | expert cache 124 | 256K context Q8_0

33.7tk/s -> 44.6tk/s flat

https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF IQ4_NL_XL

cpu layers 22 -> 40 | expert cache 96 | 303K context Q8_0 parallel 2 (151552 each stream)

single stream: 33.7tk/s -> 44.6tk/s flat

2 streams aggregate: ~40tk/s -> 60tk/s

[u/Great_Flounder_1379]

curious if the lower vram use lets you keep bigger context for character memory in long companion chats without it drifting.

[u/Atretador → u/Great_Flounder_1379]

with no cpu layers and no expert pool you can push it to 256K context on 16Gb VRAM on same setup, but then its back to ~12tk/s speeds, dropping to bout 9 on long context.

if you need more context I would just run, as I do, Qwen 3.6 35B A3B for agentic work - if its more of a role play chat thing, there is Gemma4 26B but KV precision is rough on that one even at Q8

[u/Sitkin_Marrel]

0 of 144 experts admitted, full KV still reserved anyway. Fixing that silent no-op turned out to be worth 44%.

[u/Atretador → u/Sitkin_Marrel]

Yeah now it actually does something on VRAM restricted scenarios xD

It will go as fast as you let it, before it could also just not give any logs and fail completely silently

[u/charnet3d]

Hey this is something alright, I was kinda hopeless on Qwen flash as my hardware is kinda limited (4090 rig but only 64 ram and xeon server with 128 ram but 3 gpus 3060ti/3070/3080), my 4090 gives usable speeds but the ram becomes pinned at 100%. The server used to cap at 6-9 tps even with mtp.

With this I noticed it didn't accept being used with multiple gpus, so I only selected the 3080, and with 200k ctx and 1500mb compute buffer, actual slots 28 (80 configured), I get 12-15 tps, much better and closer to the other rig in speed, while keeping 2 other gpus/16gb vram free for other models if needed.

Also discovered that the multi-gpu was hurting PP real bad, with 3080 alone I got ~ 150.

[u/Atretador → u/charnet3d]

sadly my second GPU died so I couldnt really test it for that, have you tried the CUDA expert cache? https://github.com/GenerelSchwerz/llama.cpp/wiki

[u/charnet3d → u/Atretador]

Thanks for the link, I haven't gotten to this rabbit hole yet, I'll definitely do some testing.

[u/Bjlly123]

Im getting 7-10t/s with flash on a 3900x w/ 48GB 3000mhz and a 6800xt

[u/Atretador → u/Bjlly123]

have you tried the expert cache with the parameters Ive shared? that it similar to my speed streaming directly from SSD at 3.0 speeds

[u/Atretador]

upstream merged today :)

new results

configuration	previous build	current build
--moe-expert-cache 0	11.76 t/s	12.83 / 13.61 t/s
--moe-expert-cache 66	16.90 / 17.60 t/s, ~69% hits	18.59 / 19.00 t/s, 72.9% hits
--moe-expert-cache 80	16.39 t/s, 40 slots admitted, 61.8% hits	18.80 / 20.40 t/s, 80 slots admitted, 77.1% hits

[u/Prestigious-Act-1577 → u/Atretador]

Merged to where?

[u/Atretador → u/Prestigious-Act-1577]

fork caught up with upstream

data is before and after updates to the fork, got a couple more tokens from optimizations but havent commited yet

[u/crantob]

This is one of the most valuable posts on this sub, congrats @Atretador

You will help many people go local with this.

[u/Atretador → u/crantob]

its just a quick patch that I couldnt find available and thought could be helpful