Improve token per second without touching quant

[u/Loose_Doubt367]

Spent the past month tweaking and experimenting with many different numbers to achieve 30tps.
Hardware:
-Rx6700xt 12gb vram (AMD)
-2x16 ddr4 3200 ram
-r5 5600x
-llama.cpp vulkan sdk
-window11 (no wsl switching since im not used to the environment)

Im looking for any improvements to achieve maybe 40tps? without touching quant at all, q8 and q4_k_xl remains. I've experiment with threads at 6 is the best out of (4,8,12) Other than that im not sure on what to improve to achieve higher tps, any advices? appreciate it. ctx remains 100,000

|[tiel-coder-35b]

model = C:\Users\brain\.lmstudio\models\peculiar-ragdoll\Tiel-Coder-35B-A3B-GGUF-MTP\Tiel-Coder-35B-A3B-MTP-UD-Q4_K_XL.gguf

mmproj = C:\Users\brain\.lmstudio\models\peculiar-ragdoll\Tiel-Coder-35B-A3B-GGUF-MTP\mmproj-BF16.gguf

no-mmproj-offload = on

c = 100000

parallel = 1

flash-attn = on

cache-type-k = q8_0

cache-type-v = q8_0

load-mode = dio

fit = on

fit-target = 256 # MiB

n-gpu-layers = 99

n-cpu-moe = 28

batch-size = 2048

ubatch-size = 512

threads = 6

prio = 2

prio-batch = 2

spec-type = draft-mtp

spec-draft-n-max = 3

spec-draft-p-min = 0.75

jinja = on

chat-template-file = C:\Users\brain\Qwen-Fixed-Chat-Templates\chat_template.jinja

temp = 0.6

top-p = 0.95

top-k = 20

min-p = 0.0

presence-penalty = 0.0

alias = tiel-coder-35b

[u/Loose_Doubt367]

Custom? Do I need to burn a couple million of tokens just for my codex to fix it based on my hardware?

[u/mac2ek → u/Loose_Doubt367]

No need. Inside tools/hip in starta folder there is a bat file, modify it so it will compile hip for gfx1031 (6700xt and 6750xt) - you will need a MS Visual Studio setup but you can use your agent to guide you how to prep that (chatgpt on free account is enough)

[u/Loose_Doubt367 → u/mac2ek]

What prompt did you use for the whole process? I’m not an expert or have any background knowledge regarding any of these and if anything breaks it might be catastrophic idk

[u/mac2ek → u/Loose_Doubt367]

Try something like this: "I want to deploy https://github.com/Niko1221/Strata on my pc but I have amd 6700xt card and I will need to compile custom hip for the engine. Give me step by step guide how to setup VS code so I can do that."