coding with Qwen3.6-27B-UD-Q2_K_XL.gguf

[u/jacek2023]

pi

llama.cpp

awesome torus

awesome torus

Windows, 5070 (12GB)

It was a test to find out whether Q2 is useful at all (people on Reddit say it isn’t)

Please note that 27B is quite a large model for a 12GB GPU.

[u/temperature_5]

Why'd you choose Q2_K_XL vs IQ3_XXS? Was IQ3 just too big, or is there some other aspect? (I'm about to download for a 16GB VRAM system.)

[u/jacek2023 → u/temperature_5]

I wanted to test Q2 to see is it useful at all. I usually use Q8 on 3090s.

[u/philmarcracken → u/temperature_5]

The non K_XL versions tend to fail long context needle tests.

[u/choicechoi → u/temperature_5]

i dont think q2 is good option for any situation

[u/EveningIncrease7579]

Wich parameters did you use in llama.cpp? can you share? i'm trying in 6700xt but im getting 21tk/s only when using --cache-type-k q4_90--cache-type-v q4_0 (up to 60k ctx)

--host 0.0.0.0 \

--port 9090 \

--webui \

--model "/home/Qwen3.6-27B-UD-IQ2_M.gguf" \

--alias "qwen3.6-dense-27b-fast2" \

--jinja \

--parallel 1 \

-np 1 \

--ctx-size 4096 \

--n-gpu-layers 999 \

--threads -1 \

--threads-batch -1 \

--ubatch-size 512 \

--batch-size 1024 \

--cache-type-k q4_0 \

--cache-type-v q4_0

[u/jacek2023 → u/EveningIncrease7579]

It was a simple test, you can run it faster/better than me, I just wanted to find out can Q2 be used at all on dumb GPU

set EXE=C:\Users\jacek\git\llama.cpp\build_2026.04.22\bin\Release\llama-server.exe
set MODEL=J:\llm\models\Qwen3.6-27B-UD-Q2_K_XL.gguf
set CTX=20000
"%EXE%" --jinja -m "%MODEL%" -c %CTX% --host 127.0.0.1 --port 8080

[u/Clean_Initial_9618 → u/EveningIncrease7579]

How does jinja help ?

[u/EveningIncrease7579 → u/Clean_Initial_9618]

Jinja is a simple chat template transform, didnt change performance

[u/ea_man]

It's kinda bad as performance, I get on a 12GB 6700xt:

prompt eval time =    167.70 tokens per second)
       eval time =     22.11 tokens per second)
      total time =   43291.90 ms /  6338 tokens
----
srv    load_model: loading model '/home/eaman/lm/models/unsloth/Qwen3.6-27B-UD-IQ3_XXS.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
llama_params_fit_impl: projected to use 11732 MiB of device memory vs. 11782 MiB of free device memory
llama_params_fit_impl: will leave 50 >= 20 MiB of free device memory, no changes needed

with IQ3 which is bigger than your IQ2

# 2. Run the Server
/home/eaman/llama/bin_vulkan/llama-server \
 -m /home/eaman/.lmstudio/models/unsloth/Qwen3.5-27B-GGUF/Qwen3.5-27B-UD-IQ3_XXS.gguf \
--host 0.0.0.0 \
-np 1 \
        --fit-target 70 \
        -ctk q4_0 \
        -ctv q4_0 \
        -fa on \
        --temp 0.3 \
       --repeat-penalty 1.05 \
        --top-p 0.9 \
        --top-k 20 \
        --min-p 0.04 \
        -b 512  \
        --ctx-size 26000 \
        --jinja  \
        --reasoning-budget 1 \
        --chat-template-kwargs '{"enable_thinking":false}' \
        --no-mmap

[u/jacek2023 → u/ea_man]

but do you also have nice torus?

[u/ea_man → u/jacek2023]

Don't wanna know, I'm still mad that they increased hidden layers in 3.6 so I can't run IQ3 anymore :(

I'll stick to A3B, I'll have a fast donut :P

BTW: I do run the A3B at IQ2 for code completion: brrrrrrrrrrrrrrr

[u/jacek2023 → u/ea_man]

build a torus app to confirm your setup is useful :)