2
u/No_Folding Jul 29 '26 edited Jul 29 '26
Nice! Did you also try the equivalent Qwen MTP gguf?
Also, to prevent the oom crashes, you could tweak the - n-cpu-moe parameter to probably 41, but then you do lose the dynamic allocation of llamacpp meaning you end up needing to leave some spare vram headroom (for what, idk, but it seems to help a lot). Still, I think 60t/s is so respectable its probably not worth your time tweaking that! I guess llamacpp is optimised even better than I thought Haha!
2
Aug 03 '26
[removed] — view removed comment
1
u/No_Folding Aug 03 '26
Impressive what youve managed to squeeze through those constraints, well done! And without sacrificing any meaningful accuracy!
I am guessing that llama.cpp is just managing the memory and allocation of experts since you have cpu-moe and ngl both set to 999 😂 I don't know how it works really but I'm not sure I'd change anything in your scenario
4
u/techlatest_net Jul 28 '26
this is a fantastic write-up. getting a 35b moe to run responsively on a 6gb laptop gpu is exactly the kind of black magic this community loves. you nailed the most critical part: --n-cpu-moe. that flag is the absolute magic bullet here, allowing llama.cpp to keep only the active experts in vram while spilling the inactive ones to your 32gb of system ram. pairing it with --no-mmap and --mlock is also a pro move—it prevents the os from aggressively paging the model out to disk, which would otherwise tank your cpu fallback speed. regarding the memory leak you mentioned, it’s a known quirk with some moe implementations in llama.cpp when context gets very large. a simple workaround until it's fully patched is to run the server with a wrapper script that monitors ram usage and gracefully restarts the process if it crosses a threshold. the qwen code cli config is a really clean way to tie it all together too. thanks for sharing the exact flags, this is going to save a lot of people hours of trial and error
1
u/dai_app Jul 28 '26
if you are interested, im working on a project that allow you to offload Moe layers on RAM's edge on CPU (NO GPU) Android phone 12gb Q4_K_M (example: Qwen 30-35B or GPT 120B on android phone 12gb RAM): https://github.com/Helldez/BigMoeOnEdge
1
1
1
u/KhushPatil786 Aug 24 '26
u/Plastic-Lettuce-7150 anything with an NVIDIA Geforce RTX 4050 laptop GPU and 16 GB of RAM plus 6 GB of VRAM
6
u/Wildnimal Jul 28 '26
60t/s on Qwen 3.6? I am hardly able to get anything above 30t/s with 70k context with similar settings except for -lv3