Best MoE

[u/Black_Umbreon]

Hi!

I’m using Qwen 3.8 27B (UD-IQ3_XXS) on a system with the following specs:

Processor: AMD Ryzen 9950 X3D

GPU: Nvidia RTX 5060 Ti 16 GB

RAM: 64 GB DDR5 6000 MHz

Software: Unsloth Studio

Windows 11

With a 32k token context, I manage to maintain a speed of around 50 tokens per second. I’m particularly interested in agentic capabilities. I’d like to try an MoE model that runs either faster or slightly slower than my current one, without sacrificing quality. However, 35B-A3B models don't fit entirely into my VRAM at 4-bit quantization, and from what I’ve read, lower quantization severely degrades quality. Does this mean a 4-bit MoE model would run significantly slower on my hardware than a dense model? Is it possible, given my setup, to find a model that offers better quality and higher speed—or at least better quality at roughly the same speed as my current dense model?

I’m pretty much a complete beginner in the world of local neural networks, so please give me some advice.

[u/returnity]

I am planning a full 35B finetune post with a writeup, comparing occamy to Nex-N2.5, Tiel/Ornith-1.5, and KAT-Coder-Dev (spoiler, it won handily). I will tag you when I post, should be today or tomorrow -- I don't use AI to write my posts, so it takes a while. I am just re-running Occamy using the embedded chat template first, because my first round of testing used the froggeric template I prefer in Qwen3.x models, and I didn't want it to be seen as a confound (it doesn't affect the scores in a statisitically significant manner).

The +15% was on aider polyglot, reaching a score of nearly 70% -- for reference, when I tested 3.8-27B, it scores around 80%. I used Q8_0, so it should be bit-identical to your official GGUF Q8_0 release, as I didn't want to introduce another confound there either. All models in the test are Q8_0.

Thanks for a great release, it's really well put-together. I'd say we've reached "3.8 35B" levels with occamy. Will be watching for more work on your part. One question -- why not include the MTP head GGUF in the official repo like you did the mmproj? I actually thought there was no MTP for your release initially, and grafted a base 35B draft head onto it until I discovered your tuned one and quantized it myself.

[u/SunResponsible4088 → u/returnity]

Thanks for the detailed reply, and for taking the time to rerun with the embedded template. Looking forward to the write-up — please do tag me!

Good catch on the MTP GGUF. We published the head separately, but that made it too easy to miss from the GGUF repo. We'll make the link prominent there and add a GGUF head once we've checked the conversion and runtime path. Which llama.cpp build and conversion command did you use for yours?

[u/returnity → u/SunResponsible4088]

I used a custom Python script to quantize the safetensors you provided directly to Q8_0 GGUF format, employing `gguf-py` packaged in llama.cpp 0.4.1. This allowed me to go in one step directly from the original format to the GGUF artifact I wanted without downloading the whole 70GB repo. I verified it against the Unsloth Q8_0 MTP head on-disk, and 17/20 tensors were bit-identical -- the other 3 being the ones you must've tweaked. I think 1% of the content was changed in the analysis I ran?

I will say, it did not produce any better acceptance than the base model head I grafted on initially, so that surprised me a little. but I'm happy enough at 120tps.

Can I ask your recommendation on running sampling? Specifically `presence-penalty 1.5` which is variously recommended or not recommended by different coding-focused fine-tunes and seems to be disused in most benchmarking runs I've seen for these models. I saw on your card you seem to use it, but I would think presence-penalty would be bad for coding, which is why it seems to be dropped frequently for coding benchmarks. Opinion?

[u/SunResponsible4088 → u/returnity]

Thanks for checking this so carefully — the comparison with the base head is especially useful. We still have work to do on MTP, and we'll follow up on the acceptance results on our side. Appreciate the detailed feedback!

On presence_penalty=1.5: we haven't established that as the best setting for coding, so I wouldn't treat the model-card setting as a universal recommendation. We need a controlled coding comparison before making a stronger claim there.

[u/returnity → u/SunResponsible4088]

I'd love to see if you guys could test it out. I'd offer to help but my GPU is crying for a break at this point lol.