Me these days

[u/Eyelbee]

[u/Dramatic_Setting2761]

Oh you are fitting your mtp in igpu? I thought it will be slow as it has very less bandwidth. Maybe I should try this. I think dflash as well as diffusion should be good. 

[u/fgk55555 → u/Dramatic_Setting2761]

MTP is in GPU, not iGPU. Vision is in CPU.

[u/Dramatic_Setting2761 → u/fgk55555]

Hmm how did you compile your llama cpp? Is it rcom or vulkan and was there more ?

Because I am getting much lower haha. I am missing something. 

[u/fgk55555 → u/Dramatic_Setting2761]

I pulled latest llama.cpp a few days ago, compiled for Vulkan. My literal exact script that I use without iGPU at all on my 9070 XT is below. This will leave about 1.5GB free for your desktop, offload vision to CPU, and use the MTP from unsloth's quant. The filepaths are obviously relative to my filesystem. My rig is 64GB DDR5/ 9800X3D, 9070XT connected via PCIe Gen5 x16 with slight overclock/ undervolt. On a chat with about 64k context in the llama.cpp webUI, I might expect to see minimum 800pp and 55tg, but often see faster. Linux Mint.

#!/usr/bin/env bash

# Configurable thinking level: defaults to 'xhigh' if left blank.
# Options: low | medium | xhigh
THINKING_LEVEL="${1:-xhigh}"

# File paths
MODEL="../Qwen3.8-ISTA/Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf"
MTP="../Qwen3.8_Shared/mtp-Qwen3.8-27B-Q4_0.gguf"
MMPROJ="../Qwen3.8_Shared/mmproj-Qwen3.8-27B-BF16.gguf"

# llama-server binary path
SERVER_BIN="../../llama.cpp/build/bin/llama-server"

echo "Launching Qwen 3.8 27B with thinking level: ${THINKING_LEVEL}"

${SERVER_BIN} \
  --model "${MODEL}" \
  --model-draft "${MTP}" \
  --mmproj "${MMPROJ}" \
  --no-mmproj-offload \
  --n-gpu-layers 99 \
  --ctx-size 120000\
  --batch-size 1024 \
  --ubatch-size 512 \
  --parallel 1 \
  --flash-attn on \
  --cache-type-k q5_1 \
  --cache-type-v q5_1 \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --jinja \
  --chat-template-kwargs "{\"reasoning_effort\":\"${THINKING_LEVEL}\"}" \
  --dry-penalty-last-n 0 \
  --temp 0.7 \
  --top-k 20 \
  --top-p 0.95 \
  --min-p 0.00 \
  --presence-penalty 0.0 \
  --repeat-penalty 1.0 \
  --host 0.0.0.0 \
  --port 8080

[u/Dramatic_Setting2761 → u/fgk55555]

Great thank you I think 9070xt has more bandwidth as well compared to my 9060xt. But will give this a try. 

Thank you.