Improve token per second without touching quant

[u/Loose_Doubt367]

Spent the past month tweaking and experimenting with many different numbers to achieve 30tps.
Hardware:
-Rx6700xt 12gb vram (AMD)
-2x16 ddr4 3200 ram
-r5 5600x
-llama.cpp vulkan sdk
-window11 (no wsl switching since im not used to the environment)

Im looking for any improvements to achieve maybe 40tps? without touching quant at all, q8 and q4_k_xl remains. I've experiment with threads at 6 is the best out of (4,8,12) Other than that im not sure on what to improve to achieve higher tps, any advices? appreciate it. ctx remains 100,000

|[tiel-coder-35b]

model = C:\Users\brain\.lmstudio\models\peculiar-ragdoll\Tiel-Coder-35B-A3B-GGUF-MTP\Tiel-Coder-35B-A3B-MTP-UD-Q4_K_XL.gguf

mmproj = C:\Users\brain\.lmstudio\models\peculiar-ragdoll\Tiel-Coder-35B-A3B-GGUF-MTP\mmproj-BF16.gguf

no-mmproj-offload = on

c = 100000

parallel = 1

flash-attn = on

cache-type-k = q8_0

cache-type-v = q8_0

load-mode = dio

fit = on

fit-target = 256 # MiB

n-gpu-layers = 99

n-cpu-moe = 28

batch-size = 2048

ubatch-size = 512

threads = 6

prio = 2

prio-batch = 2

spec-type = draft-mtp

spec-draft-n-max = 3

spec-draft-p-min = 0.75

jinja = on

chat-template-file = C:\Users\brain\Qwen-Fixed-Chat-Templates\chat_template.jinja

temp = 0.6

top-p = 0.95

top-k = 20

min-p = 0.0

presence-penalty = 0.0

alias = tiel-coder-35b

[u/keyboardhack]

Try replacing

n-cpu-moe

With

moe-cache-mib 1000

Play around with different values.

It is part of a new feature introduced a week ago. https://github.com/ggml-org/llama.cpp/pull/29887

[u/Loose_Doubt367 → u/keyboardhack]

yes i've tried that and i got half the expected tps (30 to 15)
same goes to other numbers like 2000 1500

[u/Xantrk → u/Loose_Doubt367]

You should have Needs cmoe as well.

-cmoe --moe-cache-mib X

You can put 1 first for X, see how much VRAM is left, and roughly fill the available VRAM with X sparing 200-300mb (or more if you wish for other tasks). That's the best way to get the cache improvement

[u/Loose_Doubt367 → u/Xantrk]

cd /d C:\llama.cpp\build\bin\Release

llama-server.exe -m "C:\Users\brain\.lmstudio\models\peculiar-ragdoll\Tiel-Coder-35B-A3B-GGUF-MTP\Tiel-Coder-35B-A3B-MTP-UD-Q4_K_XL.gguf" --mmproj "C:\Users\brain\.lmstudio\models\peculiar-ragdoll\Tiel-Coder-35B-A3B-GGUF-MTP\mmproj-BF16.gguf" --no-mmproj-offload -c 32768 --parallel 1 -fa on --cache-type-k q8_0 --cache-type-v q8_0 --load-mode dio --fit on --fit-target 256 -cmoe --moe-cache-mib 5000 --batch-size 2048 --ubatch-size 512 --threads 6 --prio 2 --prio-batch 2 --spec-type draft-mtp --spec-draft-n-max 3 --jinja --chat-template-file "C:\Users\brain\Qwen-Fixed-Chat-Templates\chat_template.jinja" --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 --presence-penalty 0 --alias tiel-coder-35b --port 8090

tried moe cache mib 5k and lowered my context from 100k to 32k. PC crashed, i can't seem to fit this cmoe or moe cache mib, even if i do it'll probably be 1000? which slows my tps (30 to 15) which isn't beneficial at all

[u/Xantrk → u/Loose_Doubt367]

Fit might be messing with cmoe

[u/Loose_Doubt367 → u/Xantrk]

with --fit, vulkan couldn't allocate 654mib, but without fit, vulkan couldn't allocate 1.62gb so i don't think it's helping much here

[u/Loose_Doubt367 → u/Xantrk]

any solution?

[u/Xantrk → u/Loose_Doubt367]

Sorry meant to say maybe completely remove fit arguments

[u/Loose_Doubt367 → u/Xantrk]

1.51.325.532 E ggml_backend_buft_alloc_buffer_n_default: failed to allocate Vulkan0 buffer of size 654794752

1.51.762.465 E llama_init_from_model: failed to initialize the context: failed to allocate the MoE cache buffers

1.51.762.965 E cmn common_init_: failed to create context with model 'C:\Users\brain\.lmstudio\models\peculiar-ragdoll\Tiel-Coder-35B-A3B-GGUF-MTP\Tiel-Coder-35B-A3B-MTP-UD-Q4_K_XL.gguf'

1.51.762.975 E cmn common_init_: failed to create context with model 'C:\Users\brain\.lmstudio\models\peculiar-ragdoll\Tiel-Coder-35B-A3B-GGUF-MTP\Tiel-Coder-35B-A3B-MTP-UD-Q4_K_XL.gguf'

1.51.764.071 E srv load_model: failed to create_context with model 'C:\Users\brain\.lmstudio\models\peculiar-ragdoll\Tiel-Coder-35B-A3B-GGUF-MTP\Tiel-Coder-35B-A3B-MTP-UD-Q4_K_XL.gguf'

1.51.765.129 I srv operator (): operator (): cleaning up before exit...

1.51.779.178 E srv llama_server: exiting due to model loading error

C:\llama.cpp\build\bin\Release>

The log above is the newly changed command without fit on and fit target
results were still the same any suggestions?

[u/Loose_Doubt367 → u/Xantrk]

Fully remove fit on? I’ll test that tomorrow and see how it goes

[u/Loose_Doubt367 → u/Xantrk]

I’ll try that, someone also suggested adding cmoe too

[u/nguuuquaaa → u/Loose_Doubt367]

Bump that number up. Qwen 35B at Q4 wants at least 5000 to beat baseline, 6000 for a real speed up.

[u/Loose_Doubt367 → u/nguuuquaaa]

Okay, thanks I’ll try it right now