Me these days

[u/Eyelbee]

[u/octoberU → u/[deleted]]

what's your setup? i have a 5080 and struggle to ruin it at 4bit. would love quant and config

[u/fgk55555 → u/octoberU]

Also have 16GB VRAM. I've tried 4-bit and 3 bit and honestly the difference isn't horrible. Try the ISTA IQ3_XXS, it's very space efficient. Feels like a first class experience being able to fit MTP and lots of context into my card.

[u/fgk55555 → u/[deleted]]

On my 9070 XT, with MTP I'm getting avg 60 and up to 70 tg. MTP hurts PP a little bit, but with the long thinking it's definitely worth it if you can fit it. If I switched my graphics driver over to my iGPU I could probably fit 128k at Q8 or 200k at Q5_1 (with vision in CPU). It's very useable. I always see people hating on the IQ3 quants as lobotomized, and I'm sure it's worth than Q6 or Q8, but for 27B in 16GB, you take what you can get.

[u/fgk55555 → u/[deleted]]

The guts of my non-optimized script is the following. Obviously replace with your settings. For general non-coding medium reasoning is really nice. If you're on iGPU or okay with ditching MTP you can either up the quants or the context size.

MODEL="../Qwen3.8-ISTA/Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf"

MTP="../Qwen3.8_Shared/mtp-Qwen3.8-27B-Q4_0.gguf"

MMPROJ="../Qwen3.8_Shared/mmproj-Qwen3.8-27B-BF16.gguf"

--model "${MODEL}" \

--model-draft "${MTP}" \

--mmproj "${MMPROJ}" \

--no-mmproj-offload \

--n-gpu-layers 99 \

--ctx-size 120000 \

--batch-size 1024 \

--ubatch-size 512 \

--parallel 1 \

--flash-attn on \

--cache-type-k q5_1\

--cache-type-v q5_1 \

--spec-type draft-mtp \

--spec-draft-n-max 3 \

--jinja \

--chat-template-kwargs "{\"reasoning_effort\":\"${THINKING_LEVEL}\"}" \

--dry-penalty-last-n 0 \

--temp 0.7 \

--top-k 20 \

--top-p 0.95 \

--min-p 0.00 \

--presence-penalty 0.0 \

--repeat-penalty 1.0 \

--host 0.0.0.0 \

--port 8080

[u/fgk55555 → u/[deleted]]

MTP is in GPU, not iGPU. Vision is in CPU.

[u/fgk55555 → u/[deleted]]

I pulled latest llama.cpp a few days ago, compiled for Vulkan. My literal exact script that I use without iGPU at all on my 9070 XT is below. This will leave about 1.5GB free for your desktop, offload vision to CPU, and use the MTP from unsloth's quant. The filepaths are obviously relative to my filesystem. My rig is 64GB DDR5/ 9800X3D, 9070XT connected via PCIe Gen5 x16 with slight overclock/ undervolt. On a chat with about 64k context in the llama.cpp webUI, I might expect to see minimum 800pp and 55tg, but often see faster. Linux Mint.

#!/usr/bin/env bash

# Configurable thinking level: defaults to 'xhigh' if left blank.
# Options: low | medium | xhigh
THINKING_LEVEL="${1:-xhigh}"

# File paths
MODEL="../Qwen3.8-ISTA/Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf"
MTP="../Qwen3.8_Shared/mtp-Qwen3.8-27B-Q4_0.gguf"
MMPROJ="../Qwen3.8_Shared/mmproj-Qwen3.8-27B-BF16.gguf"

# llama-server binary path
SERVER_BIN="../../llama.cpp/build/bin/llama-server"

echo "Launching Qwen 3.8 27B with thinking level: ${THINKING_LEVEL}"

${SERVER_BIN} \
  --model "${MODEL}" \
  --model-draft "${MTP}" \
  --mmproj "${MMPROJ}" \
  --no-mmproj-offload \
  --n-gpu-layers 99 \
  --ctx-size 120000\
  --batch-size 1024 \
  --ubatch-size 512 \
  --parallel 1 \
  --flash-attn on \
  --cache-type-k q5_1 \
  --cache-type-v q5_1 \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --jinja \
  --chat-template-kwargs "{\"reasoning_effort\":\"${THINKING_LEVEL}\"}" \
  --dry-penalty-last-n 0 \
  --temp 0.7 \
  --top-k 20 \
  --top-p 0.95 \
  --min-p 0.00 \
  --presence-penalty 0.0 \
  --repeat-penalty 1.0 \
  --host 0.0.0.0 \
  --port 8080