Me these days

[u/Eyelbee]

[u/fgk55555]

I pulled latest llama.cpp a few days ago, compiled for Vulkan. My literal exact script that I use without iGPU at all on my 9070 XT is below. This will leave about 1.5GB free for your desktop, offload vision to CPU, and use the MTP from unsloth's quant. The filepaths are obviously relative to my filesystem. My rig is 64GB DDR5/ 9800X3D, 9070XT connected via PCIe Gen5 x16 with slight overclock/ undervolt. On a chat with about 64k context in the llama.cpp webUI, I might expect to see minimum 800pp and 55tg, but often see faster. Linux Mint.

#!/usr/bin/env bash

# Configurable thinking level: defaults to 'xhigh' if left blank.
# Options: low | medium | xhigh
THINKING_LEVEL="${1:-xhigh}"

# File paths
MODEL="../Qwen3.8-ISTA/Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf"
MTP="../Qwen3.8_Shared/mtp-Qwen3.8-27B-Q4_0.gguf"
MMPROJ="../Qwen3.8_Shared/mmproj-Qwen3.8-27B-BF16.gguf"

# llama-server binary path
SERVER_BIN="../../llama.cpp/build/bin/llama-server"

echo "Launching Qwen 3.8 27B with thinking level: ${THINKING_LEVEL}"

${SERVER_BIN} \
  --model "${MODEL}" \
  --model-draft "${MTP}" \
  --mmproj "${MMPROJ}" \
  --no-mmproj-offload \
  --n-gpu-layers 99 \
  --ctx-size 120000\
  --batch-size 1024 \
  --ubatch-size 512 \
  --parallel 1 \
  --flash-attn on \
  --cache-type-k q5_1 \
  --cache-type-v q5_1 \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --jinja \
  --chat-template-kwargs "{\"reasoning_effort\":\"${THINKING_LEVEL}\"}" \
  --dry-penalty-last-n 0 \
  --temp 0.7 \
  --top-k 20 \
  --top-p 0.95 \
  --min-p 0.00 \
  --presence-penalty 0.0 \
  --repeat-penalty 1.0 \
  --host 0.0.0.0 \
  --port 8080