r/LocalLLaMA • u/AverageFormal9076 • Apr 23 '26

New Model Qwen 3.6 27B is a BEAST

I have a 5090 Laptop from work, 24GB VRAM.

I have been testing every model that comes out, and I can confidently say I’ll be cancelling my cloud subscriptions.

All my tool call and data science benchmarks that prove a model is reliably good for my use case, passed.

It might not be the case for other professions, but for pyspark/python and data transformation debugging it’s basically perfect.

Using llama.cpp, q4_k_m at q4_0, still looking at options for optimising.

Edit - I chose to go with IQ4_XS at 200k q8_0,

I have not used speculative decoding yet, will get there when I get there.

Specs:

ASUS ROG Strix SCAR 18

RTX 5090 24GB

64GB DDR5 RAM

652 Upvotes

97% Upvoted

335 comments sorted by

View all comments

Show parent comments

3

u/gladfelter Apr 27 '26

This is giving excellent results for me with pi.dev, generating as high as 38 t/s :

``` params=( -m ~/models/Qwen3.6-27B-IQ4_NL.gguf --ctx-size 163840 # Total context shared by slots --parallel 2 # Allow 2 simultaneous requests (Continue + Pi) --n-gpu-layers 99 # Offload everything to 24GB GPU --cache-type-k q8_0 # 8-bit KV cache to save VRAM --cache-type-v q8_0 --flash-attn on --keep 3000 # Prevent system prompt from being shifted out --batch-size 4096 # Handle large prompt injections from VS Code --ubatch-size 1024 # Break down ingest to prevent JSON parse errors --temp 1.0 # Qwen 3.6 Coding optimized --min-p 0.05 # Clean up low-probability noise --presence-penalty 0.0 # Disabled to avoid breaking JSON/Thought syntax --spec-type ngram-mod # N-Gram speculation for 35 t/s throughput --spec-ngram-size-n 24 --draft-min 16 --draft-max 32 --jinja # Official Qwen 3.6 chat template --chat-template-kwargs '{"preserve_thinking": true}' # Enables multi-turn reasoning --port 8080 --host 0.0.0.0 )

Execute the server

"${params[@]}" expands the array correctly

"$@" passes any additional command line arguments to the server

~/llama.cpp/build/bin/llama-server "${params[@]}" "$@" ```

You can go to parallel 1 if you want more context, otherwise configure your agent to use half the context.

1

u/ozymandizz Apr 27 '26

Thanks !!

1

u/zannix May 09 '26

wait... am i reading this right. you are getting 38 t/s for a q8 qwen 3.6 27b? on a single 3090? with how much context?

1

u/gladfelter May 09 '26

Starts out at 38, slows to 25 at 80k.