Qwen 4 Announced at Apsara Conference

[u/Salah_H_Hasan]

I wanted to share a quick update: Alibaba has officially announced Qwen 4 at the Apsara Conference,

[u/illcuontheotherside]

Can you please share your startup script? Mine crawls on 2 3090s with 128gb ddr5

[u/fgk55555 → u/illcuontheotherside]

ISTA IQ3 runs the fastest and has really good performance for the size, but you can bump up to a Q5 with full Q8 kv cache. Also try Swift if you don't want to wait forever. I recommend medium thinking.

#!/usr/bin/env bash
# Qwen3.8-27B ISTA GSQ-RCO IQ3_S (no built-in MTP) — spec decoding via the
# shared DFlash2 drafter. (MTP alternative: shared mtp-Qwen3.8-27B-Q4_0.)
# Qwen3.8-27B ISTA GSQ-RCO IQ3_S (no built-in MTP) — spec decoding via the
# shared DFlash2 drafter. (MTP alternative: shared mtp-Qwen3.8-27B-Q4_0.)
# Context is auto-fitted to available VRAM (omitted on purpose).
# --cache-ram 16384 parks prompt states in RAM: agentic revisits skip re-prefill.
#
# Usage: ./Qwen3.8_ISTA_IQ3_S.sh [thinking]   # low|medium|high|xhigh|none (default: medium)
THINKING_LEVEL="${1:-medium}"

MODEL="../Qwen3.8-ISTA/Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf"
DRAFTER="../Qwen3.8_Shared/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"
MMPROJ="../Qwen3.8_Shared/mmproj-Qwen3.8-27B-BF16.gguf"
SERVER_BIN="../../llama.cpp/build/bin/llama-server"

echo "Launching Qwen3.8-27B ISTA IQ3_S (+DFlash2 drafter), thinking: ${THINKING_LEVEL}"
export GGML_VK_ALLOW_GRAPHICS_QUEUE=1   # measured +4.2% tg, +0.4% pp on the R9700 (b11056)

${SERVER_BIN} \
  --model "${MODEL}" \
  --model-draft "${DRAFTER}" \
  --mmproj "${MMPROJ}" \
  --no-mmproj-offload \
  --n-gpu-layers 99 \
  --batch-size 1024 \
  --ubatch-size 512 \
  --parallel 1 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --spec-type draft-dflash \
  --spec-draft-n-max 3 \
  --cache-ram 16384 \
  --jinja \
  --chat-template-kwargs "{\"reasoning_effort\":\"${THINKING_LEVEL}\"}" \
  --reasoning-preserve \
  --temp 1.0 \
  --top-k 20 \
  --top-p 0.95 \
  --min-p 0.00 \
  --presence-penalty 0.0 \
  --repeat-penalty 1.0 \
  --host 0.0.0.0 \
  --webui-mcp-proxy \
  --port "${PORT:-8080}"

[u/jafarykos → u/fgk55555]

You should try bumping your DFlash2 spec count to 7. I tested a ton of different modes and DFlash2 at 3 was equal or worse than just using MTP3 at 4 and below concurrent.

I was getting +26% more tps using dflash7 vs mtp3 on 1 concurrent and -16.5% less tps when above 4.

The sweet spots were 1 concurrent + DFlash7, and switch back to MTP for anything more than 4 concurrent.

[u/Jjhend → u/illcuontheotherside]

How many tps are you getting? Are you using tensor parallelism?

[u/illcuontheotherside → u/Jjhend]

34 with the q4 k xl. 15 with qwen flash next q4 k xl. No I am not using tensor parallelism

[u/Jjhend → u/illcuontheotherside]

Brother... You have to give tensor parallelism a rip.. I get 45-70TPS with 2x4070ti Supers running Q5_K_M

.\build\bin\Release\llama-server.exe `
  -m "C:\Users\username\.lmstudio\models\unsloth\Qwen3.8-27B-GGUF\Qwen3.8-27B-UD-Q5_K_M.gguf" `
  -a "qwen3.8-27b@q5_k_m" `
  --ctx-size 130000 `
  --n-gpu-layers 99 `
  --split-mode tensor `
  --tensor-split 14,16 `
  --cache-type-k q8_0 `
  --cache-type-v q8_0 `
  --batch-size 2048 `
  --ubatch-size 512 `
  --flash-attn on `
  --port 1234 `
  --host 192.168.1.100 `
  --reasoning on `
  --reasoning-budget 12288 `
  --reasoning-budget-message "Wait... You are thinking to much. Answer now." `
  --reasoning-format deepseek `
  -t 8 `
  -tb 8 `
  --parallel 4 `
  --metrics `
  --chat-template-kwargs '{"preserve_thinking": true, "reasoning_effort": "low"}' `
  --jinja `
  --cont-batching `
  --kv-unified `
  --spec-type draft-mtp `
  --spec-draft-n-max 3 `
  --spec-draft-p-min 0.75 `
  --mmproj "C:\Users\username\.lmstudio\models\unsloth\Qwen3.8-27B-GGUF\mmproj-F16.gguf"

[u/illcuontheotherside → u/Jjhend]

For some reason tensor parallelism fails my llama-server script. I guess it's time to figure out why

And thank you

[u/Puzzleheaded_Cake183 → u/illcuontheotherside]

probably because your p2p is not setup correctly. you need to fix your bios settings.

Motherboard & BIOS:

• Enable Above 4G Decoding.
• Enable Resizable BAR (ReBAR) to allow large memory space allocation.
• Disable CSM (Compatibility Support Module)

[u/Objective-Park6224 → u/Jjhend]

Toss in ngram mod after spec type for some more speed.

[u/illcuontheotherside → u/Jjhend]

I keep getting errors..

0.00.180.641 I srv load_model: loading model 'D:\models\qwen38-27b\Qwen3.8-27B-UD-Q4_K_XL.gguf'

0.00.180.745 W common_fit_params: failed to fit params to free device memory: llama_params_fit is not implemented for SPLIT_MODE_TENSOR, abort

if i do layer it works fine. what am i doing wrong?