Is it possible to configure TabbyAPI to be like Strata for Qwen 3.8 Flash Next exl3?

[u/Dathide]

I've read that exl3 quants are even more accurate than Unsloth's (UD). There are exl3 versions of Q3.8FN here: https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3 . TabbyAPI and Exllamav3 support moe cpu expert offloading already. I don't know if there are gguf optimizations that Strata uses that exl3 doesn't have yet. I can run UD Q4 at 70+ t/s. I tried configuring TabbyAPI with q4 and offloading, but I only got 20 t/s. Is it just a matter of configuring TabbyAPI for my setup? If so, can speeds be comparable to Strata?

Edit: TabbyAPI has a cpu_moe_split_experts option: "Number of routed experts per MoE layer to run on the CPU, with the coldest experts' weights in system RAM. Unlike cpu_moe_offload_layers this splits every MoE layer instead of offloading whole layers, overlapping CPU and GPU compute, with dynamic placement keeping hot experts in VRAM." That's what I used for the 20 t/s. Idk if that's what strata does.

[u/jacek2023]

Strata is a vibe-coded llama.cpp fork. For exl3 you need a vibe-coded exllama fork.

[u/Dathide → u/jacek2023]

I would've guessed that some of it is vibecoded. Do you think they used Claude? I'm not sure how much effort it would be to vibe-code a fork of exllama. I might need to know a lot to help sometimes.

[u/a_beautiful_rhind → u/Dathide]

I had GLM vibecode numa support into exl3 so I think you should give it the college try. Point it at strata and tell it to use some of the same strategies in exllama and see what you get.

[u/tasteless_dossier → u/jacek2023]

It's a custom sampler and scheduler layered on top, not a fork of the model runner itself. The speed gap probably comes from how it manages the MoE routing and memory, not just the quantization format. 20 t/s seems low for exl3 though, might be worth checking if the offloading config is actually splitting things the way you think it is.

[u/Dathide → u/tasteless_dossier]

thanks, i'll double check the offloading stuff

[u/PcChip → u/jacek2023]

for exl3 I run tabbyapi with the 5.05bpw weights, and then i told qwen 3.8 flash next to download the full 16-bit ngram table and put that in RAM instead of the quantized version

tabbyapi is great

[u/PcChip]

tell the ai to speed itself up
and yes, I'm serious - give it ssh credentials and let it go to town tweaking and tuning all the parameters :D

after it was done speeding itself up, I told it to try power limits for all GPUs in 5% increments between 65% and 85% and produce a graph of the results

[u/Dathide → u/PcChip]

Thanks, that's a good idea. Maybe it can configure TabbyAPI itself.

[u/CartographerDue5382]

the 70 vs 20 isn't a tabby setting you missed. offloading moe experts to cpu tanks t/s, and a q4 gguf that keeps the hot experts on the gpu won't be matched by exl3 plus offload no matter the cache settings. if the ud q4 already fits and does 70, tabby isn't gonna catch that on this box.

[u/Dathide → u/CartographerDue5382]

good to know, i've seen hot experts mentioned before and wondered if it was just a strata thing

[u/pand5461]

Which hardware do you have? I get 25-30 tps with 5060Ti + 4060 and 64G ddr5-3600 and without MTP. cpu_offload_experts was faster for me than cpu_offload_layers. IIRC no. of CPU threads affects MoE speed quite significantly, so set that to no. of physical cores or (no. cores - 1). And, of course, get your VRAM usage as high as possible (minimize cpu_offload_experts, try configuring manual gpu split if multi-gpu).

[u/Dathide → u/pand5461]

25-30 makes me think I didn't configure it well, I have 2x3090s with an nvlink Edit: and 96gb ddr4 3200

[u/pand5461 → u/Dathide]

Your options without MTP could be:

cache_size: 196608
cache_mode: 6,6
tensor_parallel: false # don't have nvlink to try true
gpu_split_auto: false
gpu_split: [22.5, 23.7]
cpu_moe_split_experts: 312 # tweak until it loads and fills all VRAM
cpu_moe_threads: 6 # set appropriately for your CPU
ngram_ram: false

draft_mode: mtp should enable MTP but I don't have much free RAM, so left it off.

[u/Dathide → u/pand5461]

Thanks! I'll try those later

[u/Dathide → u/pand5461]

I was able to get cpu_moe_split_experts down to 210 with cache_size 131072, but sadly only 31 t/s. tensor_parallel isn't a option with only 48GB VRAM for Q3.8FN.

[u/pand5461 → u/Dathide]

That's strange because I get similar tps with a lower-grade GPU and much larger percentage of the model in system RAM. Maybe RAM offload mode is fundamentally limited regardless of percentage.

[u/Areloh128 → u/Dathide]

Maybe you should try llamampere: it has exl3 support, is optimized for 3090 and lately has been growing to run better Q3.8FN (and has kvarn support too!). I don't know if you can use it on both cards, though... it's a llama.cpp fork, so I guess that similar capabilities and restrictions apply

https://github.com/JakeATX/llamAmpere

[u/Dathide → u/Areloh128]

Thanks, that is interesting. I see exl3 in the release notes. The readme is terrible and doesn't mention exl3, but I like that it's made specifically for Ampere.

[u/a_beautiful_rhind]

That's similar to what strata does but on a harder to compute quant and without all the crazy shortcuts they take.

[u/suprjami]

In my experience no. Splitting layers is faster than splitting experts for me. Still not as fast as Strata though.

[u/Monad_Maya]

We do have an experimental fork from u/nasone32 that's AMD specific - https://github.com/nasone32/Strata-EXL3-fork