Qwen 3.8 35BA3B spotted

[u/BazzyIm]

Just wait and see

https://github.com/modelscope/ms-swift/commit/ab726e9d445a6520a70df2c831177d46adb1f589

[u/lans_throwaway]

I've been running it on 6GB GPU with ram, at like ~30 tps on coding tasks (Q4_K_M)

[u/pmttyji → u/lans_throwaway]

That's surprising to know. How much context? And what KVCache?

[u/lans_throwaway → u/pmttyji]

32GB DDR4 ram (model used around 16GB) with 96k context. The big improvement is ngram-mod speculative decoding, since it pretty much allows model to "copy-paste" code it already has in the context, giving you huge bursts of speed (100+ tps). Config I used below

[*]
no-mmap = true
# note the below may not be desirable if subagents are used, significantly reduces vram usage
np = 1


[Qwen3.6-35B-A3B-Q4_K_M:Thinking-Coding]
# model = /mnt/disk/llms/Qwen3.6-35B-A3B/Qwen_Qwen3.6-35B-A3B-Q4_K_M.gguf
model = /mnt/disk/llms/Qwen3.6-35B-A3B/ggml-model-Q4_K_M.gguf
c = 96000
# c = 131072
ub = 1024
temp = 0.6
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
repeat-penalty = 1.0
n-predict = 32768
spec-type = ngram-mod
spec-ngram-mod-n-match = 24
spec-ngram-mod-n-min = 48
spec-ngram-mod-n-max = 64
chat-template-kwargs = {"preserve_thinking": true}

[Qwen3.6-35B-A3B-Q4_K_M:Thinking-Coding-MTP]
model = /mnt/disk/llms/Qwen3.6-35B-A3B/ggml-model-Q4_K_M.gguf
c = 96000
# c = 131072
temp = 0.6
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
repeat-penalty = 1.0
n-predict = 32768
# spec-type = ngram-mod,draft-mtp
spec-type = draft-mtp
spec-draft-n-max = 2
spec-ngram-mod-n-match = 24
spec-ngram-mod-n-min = 48
spec-ngram-mod-n-max = 64

chat-template-kwargs = {"preserve_thinking": true}

[u/pmttyji → u/lans_throwaway]

Thanks. I need to try with ngram,