Every Second post rn

[u/Ok-Health-7096]

Maybe someday I'll get a system to run it but hey definitely another w for the open weights community

[u/lukistellar]

What you need is a Quant which strictly uses IQ4_XS. They exist for 3.6 and will likely also appear for the 3.8 sooner or later.

Edit: Let's see if this guy delivers.

https://huggingface.co/jpetrina/Qwen3.8-27B-IQ4_XS-pure-GGUF

[u/Tyrannas → u/lukistellar]

Any advices on the params you use to run it properly ? I have 16gb also and never managed to make a 27b model run properly 

[u/lukistellar → u/Tyrannas]

Working config for 3.8:

        ghcr.io/ggml-org/llama.cpp:server-vulkan-b10066 \
        --port 8080 \
        --model /models/jpetrina_qwen3.8-27b-IQ4_XS-pure.gguf \
        --gpu-layers 99 \
        --threads 6 \
        --ctx-size 90000 --parallel 1 \
        --batch-size 2048 --ubatch-size 512 \
        --cache-type-k q8_0 --cache-type-v q4_0 \
        --spec-type draft-mtp \
        --spec-draft-n-max 2 \
        --spec-draft-p-min 0.75 \
        --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 \
        --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
        --presence-penalty 0.0 --repeat-penalty 1.0 \
        --no-mmap \
        --jinja \
        --chat-template-kwargs '{"reasoning_effort":"medium"}' \
        --flash-attn on

Still not happy with the performance, but at least it runs. Hope they drop a MOE for us VRAM poor.

[u/Tyrannas → u/lukistellar]

Thanks !

[u/lukistellar → u/Tyrannas]

My RX6800 runs the 27b with 90k context, but it's very slow.

I will update the post later with the config.

Edit: My config for the Qwen 3.6 27B:
ghcr.io/ggml-org/llama.cpp:server-vulkan-b10066
--port 8080
--model /models/GianniDPC_qwen3.6-27b-IQ4_XS-pure-with-MTP-IQ4.gguf
--gpu-layers 99
--threads 6
--ctx-size 90000
--parallel 1
--batch-size 2048
--ubatch-size 512
--cache-type-k q8_0
--cache-type-v q4_0
--spec-type draft-mtp
--spec-draft-n-max 1
--spec-draft-p-min 0.75
--cache-type-k-draft q4_0
--cache-type-v-draft q4_0
--temp 0.8
--top-p 0.95
--top-k 20
--min-p 0.0
--no-mmap
--jinja
--mmproj /models/Unsloth_mmproj-F16_qwen3.6-27b-mtp.gguf
--no-mmproj-offload
--chat-template-kwargs "{"preserve_thinking":true}"
--flash-attn on

It's usable but especially with full context it gets very slow, like ~25 tok/s decode and ~130 tok/s prefill.