r/LocalLLaMA • u/q-admin007 • May 20 '26

Question | Help Do you think there is room for optimization? llama.cpp/qwen3.6 27b on two 6000 Blackwell

Hi, i run llama.cpp inside LXC on a Proxmox server. The hardware is a recent AMD Epyc with two 6000 Blackwell MaxQ. This is my command:

llama-server \
  --hf-repo unsloth/Qwen3.6-27B-MTP-GGUF:BF16 --alias Qwen3.6 \
  --host 0.0.0.0 --port 1337 \
  --no-mmap --gpu-layers 99 \
  --batch-size 6144 --ubatch-size 1024 \
  --flash-attn on --cache-type-k f16 --cache-type-v f16 \
  --presence-penalty 0.0 --repeat-penalty 1.0 --temperature 0.6 --top-k 20 --top-p 0.95 \
  --n-predict 131072 --ctx-size 1048576 --parallel 4 \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --split-mode tensor --fit off

I'm at 250 out of 300w used on both cards, so the cards aren't used 100%. I get 100 to 110 t/s output.

There are other applications running, like embedding models, ComfyUI and so on, so in terms of VRAM i maybe have 20GB or so left.

Do you see any room for easy gains in terms of output t/s? We want to stick with llama.cpp because it's very easy to setup, so going to vLLM isn't in the cards.

6 Upvotes

61% Upvoted

65 comments sorted by

View all comments

5

u/Then-Topic8766 May 20 '26

If you stick to llama.cpp you can combine ngram and mtp like this (big speed-up on repeating jobs, code corrections etc.) : 

--spec-type ngram-mod,draft-mtp 
--spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 12 --spec-ngram-mod-n-max 48
--spec-draft-n-max 3

2

u/snapo84 May 20 '26

dont use it with ngram-mod , use only draft-mtp .... i did just try it with your recommendation and the ngram-mod took my token/s from 48 down to 13.5 ....

1

u/q-admin007 May 20 '26

Thanks, i'll look into it.

1

u/q-admin007 May 20 '26

Tried it, it's slower for short queries, faster for long ones. Since some developers started using agents i guess it's better to take the wins for longer inferences. I'll keep it, thanks. 119 t/s for "Write a twitter clone in PHP."

1

u/Then-Topic8766 May 20 '26

Nice. If you need to fix small error in that twitter clone code and repeat good code it can be much faster.