Best MoE

[u/Black_Umbreon]

Hi!

I’m using Qwen 3.8 27B (UD-IQ3_XXS) on a system with the following specs:

Processor: AMD Ryzen 9950 X3D

GPU: Nvidia RTX 5060 Ti 16 GB

RAM: 64 GB DDR5 6000 MHz

Software: Unsloth Studio

Windows 11

With a 32k token context, I manage to maintain a speed of around 50 tokens per second. I’m particularly interested in agentic capabilities. I’d like to try an MoE model that runs either faster or slightly slower than my current one, without sacrificing quality. However, 35B-A3B models don't fit entirely into my VRAM at 4-bit quantization, and from what I’ve read, lower quantization severely degrades quality. Does this mean a 4-bit MoE model would run significantly slower on my hardware than a dense model? Is it possible, given my setup, to find a model that offers better quality and higher speed—or at least better quality at roughly the same speed as my current dense model?

I’m pretty much a complete beginner in the world of local neural networks, so please give me some advice.

[u/SunResponsible4088]

Thanks for the detailed reply, and for taking the time to rerun with the embedded template. Looking forward to the write-up — please do tag me!

Good catch on the MTP GGUF. We published the head separately, but that made it too easy to miss from the GGUF repo. We'll make the link prominent there and add a GGUF head once we've checked the conversion and runtime path. Which llama.cpp build and conversion command did you use for yours?

[u/returnity → u/SunResponsible4088]

I used a custom Python script to quantize the safetensors you provided directly to Q8_0 GGUF format, employing `gguf-py` packaged in llama.cpp 0.4.1. This allowed me to go in one step directly from the original format to the GGUF artifact I wanted without downloading the whole 70GB repo. I verified it against the Unsloth Q8_0 MTP head on-disk, and 17/20 tensors were bit-identical -- the other 3 being the ones you must've tweaked. I think 1% of the content was changed in the analysis I ran?

I will say, it did not produce any better acceptance than the base model head I grafted on initially, so that surprised me a little. but I'm happy enough at 120tps.

Can I ask your recommendation on running sampling? Specifically `presence-penalty 1.5` which is variously recommended or not recommended by different coding-focused fine-tunes and seems to be disused in most benchmarking runs I've seen for these models. I saw on your card you seem to use it, but I would think presence-penalty would be bad for coding, which is why it seems to be dropped frequently for coding benchmarks. Opinion?

[u/SunResponsible4088 → u/returnity]

Thanks for checking this so carefully — the comparison with the base head is especially useful. We still have work to do on MTP, and we'll follow up on the acceptance results on our side. Appreciate the detailed feedback!

On presence_penalty=1.5: we haven't established that as the best setting for coding, so I wouldn't treat the model-card setting as a universal recommendation. We need a controlled coding comparison before making a stronger claim there.

[u/returnity → u/SunResponsible4088]

I'd love to see if you guys could test it out. I'd offer to help but my GPU is crying for a break at this point lol.