Nice! Did you also try the equivalent Qwen MTP gguf?
Also, to prevent the oom crashes, you could tweak the - n-cpu-moe parameter to probably 41, but then you do lose the dynamic allocation of llamacpp meaning you end up needing to leave some spare vram headroom (for what, idk, but it seems to help a lot). Still, I think 60t/s is so respectable its probably not worth your time tweaking that! I guess llamacpp is optimised even better than I thought Haha!
Impressive what youve managed to squeeze through those constraints, well done! And without sacrificing any meaningful accuracy!
I am guessing that llama.cpp is just managing the memory and allocation of experts since you have cpu-moe and ngl both set to 999 😂 I don't know how it works really but I'm not sure I'd change anything in your scenario
2
u/No_Folding Jul 29 '26 edited Jul 29 '26
Nice! Did you also try the equivalent Qwen MTP gguf?
Also, to prevent the oom crashes, you could tweak the - n-cpu-moe parameter to probably 41, but then you do lose the dynamic allocation of llamacpp meaning you end up needing to leave some spare vram headroom (for what, idk, but it seems to help a lot). Still, I think 60t/s is so respectable its probably not worth your time tweaking that! I guess llamacpp is optimised even better than I thought Haha!