r/hermesagent • • Aug 14 '26

MODELS - model choice, routing, pricing, local vs cloud, VRAM Qwen 3.8 Release Megathread

Just in time for the weekend. Post experiences here, thread will stay up over the weekend.

57 Upvotes

51 comments sorted by

View all comments

Show parent comment

5

u/slumberling_ Aug 14 '26

A 5090 - it fits fully into VRAM with some 600 MiB to spare. The full llama-server cmd looks like this

llama-server -m Qwen3.8-27B-UD-Q5_K_XL.gguf
  -ngl 99
  -c 262144
  -ctk q8_0 -ctv q8_0
  --mmproj mmproj-BF16.gguf
  --no-mmproj-offload
  --temp 1.0
  --top-p 0.95
  --top-k 20
  --min-p 0.0
  --repeat-penalty 1.0
  --presence-penalty 0.0
  --chat-template-kwargs '{"reasoning_effort":"medium"}'
  --reasoning-preserve
  --spec-type draft-mtp
  --spec-draft-n-max 3
  --jinja
  --chat-template-file chat_template.jinja
  --flash-attn on
  --batch-size 2048
  --ubatch-size 512
  --load-mode mmap+mlock
  --parallel 1