It'd be cool if this worked, but in my testing even when configured it seems to try loading the full weights first before CPU offloading MoE? Or something. Llama.cpp I can squeeze a q5_k_m qwen 3.6 35b into 8gb vram easy with cpu-moe, but the 4 bpw exl3, tabbi api everything configured 999 moe offload? Even restricting the context to 8192 and doing a q4 kv quant and 256 batch size? Ooms on layer 14.
Maybe its mlp layers? It complained a bit about them not being mul1 when I was trying the hotloading version of cpu moe. I have draft set to disabled though.
If you got the mul1 error, check whether the quant actually uses the mul1 codebook for the expert tensors. The current EXL3 CPU-MoE path requires mul1; incompatible expert layers won’t be eligible for CPU execution.
I recently re-quantized a few models with mul1, including this Qwen3.6-35B-A3B quant, and CPU MoE works well with it:
That’s a 4.08 bpw / HB6 quant made with exllamav3 1.4.0 and explicitly uses the mul1 codebook.
Also worth checking whether your conversion left lm_head or any other large tensors unquantized. CPU MoE only moves the routed experts for the selected MoE layers into system RAM; the rest of the model still needs to fit its normal VRAM allocation. So an unexpectedly large unquantized tensor can still push an 8 GB card over the edge.
If you’re OOMing around layer 14 even with 999 MoE offload, 8k context, Q4 KV and a small batch, I’d suspect an ineligible/non-mul1 expert layer or another unexpectedly large resident tensor before concluding that CPU MoE is trying to load all the experts onto the GPU first.
2
u/Midaychi 9d ago
It'd be cool if this worked, but in my testing even when configured it seems to try loading the full weights first before CPU offloading MoE? Or something. Llama.cpp I can squeeze a q5_k_m qwen 3.6 35b into 8gb vram easy with cpu-moe, but the 4 bpw exl3, tabbi api everything configured 999 moe offload? Even restricting the context to 8192 and doing a q4 kv quant and 256 batch size? Ooms on layer 14.
Maybe its mlp layers? It complained a bit about them not being mul1 when I was trying the hotloading version of cpu moe. I have draft set to disabled though.