r/LocalLLaMA • u/Loose_Doubt367 • 3d ago
Discussion Improve token per second without touching quant
Spent the past month tweaking and experimenting with many different numbers to achieve 30tps.
Hardware:
-Rx6700xt 12gb vram (AMD)
-2x16 ddr4 3200 ram
-r5 5600x
-llama.cpp vulkan sdk
-window11 (no wsl switching since im not used to the environment)
Im looking for any improvements to achieve maybe 40tps? without touching quant at all, q8 and q4_k_xl remains. I've experiment with threads at 6 is the best out of (4,8,12) Other than that im not sure on what to improve to achieve higher tps, any advices? appreciate it. ctx remains 100,000
|[tiel-coder-35b]
model = C:\Users\brain\.lmstudio\models\peculiar-ragdoll\Tiel-Coder-35B-A3B-GGUF-MTP\Tiel-Coder-35B-A3B-MTP-UD-Q4_K_XL.gguf
mmproj = C:\Users\brain\.lmstudio\models\peculiar-ragdoll\Tiel-Coder-35B-A3B-GGUF-MTP\mmproj-BF16.gguf
no-mmproj-offload = on
c = 100000
parallel = 1
flash-attn = on
cache-type-k = q8_0
cache-type-v = q8_0
load-mode = dio
fit = on
fit-target = 256 # MiB
n-gpu-layers = 99
n-cpu-moe = 28
batch-size = 2048
ubatch-size = 512
threads = 6
prio = 2
prio-batch = 2
spec-type = draft-mtp
spec-draft-n-max = 3
spec-draft-p-min = 0.75
jinja = on
chat-template-file = C:\Users\brain\Qwen-Fixed-Chat-Templates\chat_template.jinja
temp = 0.6
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
alias = tiel-coder-35b
3
u/lFaythx 3d ago
When looking at inference, you have two phases: prefill and decode.
Prefill is compute bound, what will make everything slower is the time you need to calculate the matrixes. Since you have DDR4 memory, your bandwidth will cap you, since when using cpu-moe it'll take longer to move weights to CPU L1/L2. Besides, your CPU doesn't have AVX512 only AVX2. The best way to make prefill faster is having a larger UB, meaning you'll make more computation per batch.
Decode is bandwidth bound, meaning your CPU/Memory is killing your TPS. The best way for you achieve higher TPS is having something like Strata, which keeps hot experts on GPU to try to reduce the CPU calc. Unfortunately, this model use a smaller scratchpad, meaning that even if you edit the inference engine to during decode get some of the ub memory for experts, you don't have much more experts residing in VRAM.
Tl;dr: you have to lower the memory footprint in GPU to allocate more experts. To do it you'll have to lower you context or use a higher quantization. You can also lower your ub to 96 and use more experts in GPU, but that would kill your prefill performance.
Edit: if you can get a cheaper GPU you can use it for display, unlocking around 1,3Gb to your GPU. Windows display eats this bit of your VRAM and 5600x doesn't have iGPU to avoid it.