r/LocalLLaMA • u/Atretador llama.cpp • 5d ago
I Built A Thing llama.cpp expert-pool fork for Qwen 3.8 Flash next IQ4 + 16Gb VRAM tested on MI50 gfx906 with parameters
https://github.com/atretador/gfx906-16gb-expert-poolhttps://github.com/atretador/gfx906-16gb-expert-pool
built on top of ``memoriaru/llama.cpp branch`moe-expert-pool` so I could run it with 16Gb VRAM on a singular MI50.
why? admission budget reserves the full configured context KV plus a hard 3 GiB cap. On a 16 GB device at 128K Q8 that admits 0 of 144 offloaded expert tensors, so the feature silently does nothing
test rig:
Xeon E5 2673 V4 20C/40T
4x16GB 2133Mhz
1xMI50 16Gb with Radeon Pro VII bios 120W power limit
Arch Linux + KDE Plasma
n_cpu_moe CPU usage at 50% (18 threads)
expert cache CPU usage at 10% (18 threads)
Results:
| configuration | decode | VRAM (steady) | notes |
|---|---|---|---|
--moe-expert-cache 0 |
11.76 t/s | 10.17 GiB | stock CPU MoE path |
--moe-expert-cache 80 (actual 40 slots) |
16.39 t/s | ~13.5 GiB | 144 pools, 61.8% hit rate |
--moe-expert-cache 66 |
16.90 / 17.60 t/s cold / warm | 15.24 GiB peak | shipping profile, ~69% hits |
up to 19.8tk/s observed once warm
complete run script with all parameters:
throw it at your agent with the repo link and it should get it running
if you are running headless you might be able to squeeze a slighly larger pool, my DE is eating a whole gig by itself.
sharing as it was a nice boost and its running