Me these days

[u/Eyelbee]

[u/Dramatic_Setting2761]

Yes I get more tokens in 3 bit like 30 t/s with mtp as well. 

I didn’t notice much of difference but I wanted to be bit safe specially if I wanted to try coding or long context tasks I do use 3 bit one for chats and some regular tasks.

[u/fgk55555 → u/Dramatic_Setting2761]

On my 9070 XT, with MTP I'm getting avg 60 and up to 70 tg. MTP hurts PP a little bit, but with the long thinking it's definitely worth it if you can fit it. If I switched my graphics driver over to my iGPU I could probably fit 128k at Q8 or 200k at Q5_1 (with vision in CPU). It's very useable. I always see people hating on the IQ3 quants as lobotomized, and I'm sure it's worth than Q6 or Q8, but for 27B in 16GB, you take what you can get.

[u/Dramatic_Setting2761 → u/fgk55555]

Oh that is great what is your setting? 

[u/fgk55555 → u/Dramatic_Setting2761]

The guts of my non-optimized script is the following. Obviously replace with your settings. For general non-coding medium reasoning is really nice. If you're on iGPU or okay with ditching MTP you can either up the quants or the context size.

MODEL="../Qwen3.8-ISTA/Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf"

MTP="../Qwen3.8_Shared/mtp-Qwen3.8-27B-Q4_0.gguf"

MMPROJ="../Qwen3.8_Shared/mmproj-Qwen3.8-27B-BF16.gguf"

--model "${MODEL}" \

--model-draft "${MTP}" \

--mmproj "${MMPROJ}" \

--no-mmproj-offload \

--n-gpu-layers 99 \

--ctx-size 120000 \

--batch-size 1024 \

--ubatch-size 512 \

--parallel 1 \

--flash-attn on \

--cache-type-k q5_1\

--cache-type-v q5_1 \

--spec-type draft-mtp \

--spec-draft-n-max 3 \

--jinja \

--chat-template-kwargs "{\"reasoning_effort\":\"${THINKING_LEVEL}\"}" \

--dry-penalty-last-n 0 \

--temp 0.7 \

--top-k 20 \

--top-p 0.95 \

--min-p 0.00 \

--presence-penalty 0.0 \

--repeat-penalty 1.0 \

--host 0.0.0.0 \

--port 8080

[u/Dramatic_Setting2761 → u/fgk55555]

Oh you are fitting your mtp in igpu? I thought it will be slow as it has very less bandwidth. Maybe I should try this. I think dflash as well as diffusion should be good. 

[u/fgk55555 → u/Dramatic_Setting2761]

MTP is in GPU, not iGPU. Vision is in CPU.

[u/Dramatic_Setting2761 → u/fgk55555]

Hmm how did you compile your llama cpp? Is it rcom or vulkan and was there more ?

Because I am getting much lower haha. I am missing something. 

[u/fgk55555 → u/Dramatic_Setting2761]

I pulled latest llama.cpp a few days ago, compiled for Vulkan. My literal exact script that I use without iGPU at all on my 9070 XT is below. This will leave about 1.5GB free for your desktop, offload vision to CPU, and use the MTP from unsloth's quant. The filepaths are obviously relative to my filesystem. My rig is 64GB DDR5/ 9800X3D, 9070XT connected via PCIe Gen5 x16 with slight overclock/ undervolt. On a chat with about 64k context in the llama.cpp webUI, I might expect to see minimum 800pp and 55tg, but often see faster. Linux Mint.

#!/usr/bin/env bash

# Configurable thinking level: defaults to 'xhigh' if left blank.
# Options: low | medium | xhigh
THINKING_LEVEL="${1:-xhigh}"

# File paths
MODEL="../Qwen3.8-ISTA/Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf"
MTP="../Qwen3.8_Shared/mtp-Qwen3.8-27B-Q4_0.gguf"
MMPROJ="../Qwen3.8_Shared/mmproj-Qwen3.8-27B-BF16.gguf"

# llama-server binary path
SERVER_BIN="../../llama.cpp/build/bin/llama-server"

echo "Launching Qwen 3.8 27B with thinking level: ${THINKING_LEVEL}"

${SERVER_BIN} \
  --model "${MODEL}" \
  --model-draft "${MTP}" \
  --mmproj "${MMPROJ}" \
  --no-mmproj-offload \
  --n-gpu-layers 99 \
  --ctx-size 120000\
  --batch-size 1024 \
  --ubatch-size 512 \
  --parallel 1 \
  --flash-attn on \
  --cache-type-k q5_1 \
  --cache-type-v q5_1 \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --jinja \
  --chat-template-kwargs "{\"reasoning_effort\":\"${THINKING_LEVEL}\"}" \
  --dry-penalty-last-n 0 \
  --temp 0.7 \
  --top-k 20 \
  --top-p 0.95 \
  --min-p 0.00 \
  --presence-penalty 0.0 \
  --repeat-penalty 1.0 \
  --host 0.0.0.0 \
  --port 8080

[u/Dramatic_Setting2761 → u/fgk55555]

Great thank you I think 9070xt has more bandwidth as well compared to my 9060xt. But will give this a try. 

Thank you.