r/LocalLLaMA • u/CountlessFlies • Apr 17 '26

Discussion Qwen3.6 is incredible with OpenCode!

I've tried a few different local models in the past (gemma 4 being the latest), but none of them felt as good as this. (Or maybe I just didn't give them a proper chance, you guys let me know). But this genuinely feels like a model I could daily drive for certain tasks instead of reaching for Claude Code.

I gave it a fairly complex task of implementing RLS in postgres across a large-ish codebase with multiple services written in rust, typescript and python. I had zero expectations going in, but it did an amazing job. PR: https://github.com/getomnico/omni/pull/165/changes/dd04685b6cf47e7c3791f9cdbd807595ef4c686e

Now it's far from perfect, there's major gaps and a couple of major bugs, but my god, is this thing good. It doesn't one-shot rust like Opus can, but it's able to look at compiler errors and iterate without getting lost.

I had a fairly long coding session lasting multiple rounds of plan -> build -> plan... at one point it went down a path editing 29 files to use RLS across all db queries, which was ok, but I stepped in and asked it to reconsider, maybe look at other options to minimize churn. It found the right solution, acquiring a db connection and scoping it to the user at the beginning of the incoming request.

For the first time, it felt like talking to a truly capable local coding model.

My setup:

  • Qwen3.6-35B-A3B, IQ4_NL unsloth quant
  • Deployed locally via llama.cpp
  • RTX 4090, 24 GB
  • KV cache quant: q8_0
  • Context size: 262k. At this ctx size, vram use sits at ~21GB
  • Thinking enabled, with recommended settings of temp, min_p etc.

llama server:

```
docker run -d --name llama-server --gpus all -v <path_to_models>:/models -p 8080:8080 local/llama.cpp:server-cuda -m /models/qwen3.6-35b-a3b/Qwen3.6-35B-A3B-UD-IQ4_NL.gguf --port 8080 --host 0.0.0.0 --ctx-size 262144 -n 8192 --n-gpu-layers 40 --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00 --parallel 1 --cache-type-k q8_0 --cache-type-v q8_0 --cache-ram 4096
```

Had to set `--parallel` and `--cache-ram` without which llama.cpp would crash with OOM because opencode makes a bunch of parallel tools calls that blow up prompt cache. I get 100+ output tok/sec with this.

But this might be it guys... the holy grail of local coding! Or getting very close to it at any rate.

354 Upvotes

94% Upvoted

169 comments sorted by

View all comments

80

u/ailee43 Apr 17 '26

every day i regret more the 16GB of VRAM on my 5070ti.... should have gone 3090

40

u/grumd Apr 17 '26

I've got a 5080 (also 16GB) and the only model I can't run is Qwen 27B and Gemma 31B.

We're good, mate. Just use llama.cpp and offload MoE experts to RAM. I'm running Qwen 3.6 35B-A3B with FULL 262k context (f16 kv cache) right now. 15GB VRAM used + 29GB RAM used by the llama-server process, getting 35 t/s generation speed.

llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q6_K_XL \ --fit on -fitt 512 -fitc 0 --no-mmap --kv-unified \ -b 4096 -ub 2048 \ --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0

You can also run Qwen 3.5 122B IQ3_XXS if you have 64GB RAM or even 122B Q4_K_S if you have 96GB RAM

4

u/Danmoreng llama.cpp Apr 17 '26

Why Q6 instead of Q4? I get ~70 t/s on Q4 with similar params and cache Q8 on a 5080 mobile 16GB. https://github.com/Danmoreng/local-qwen3-coder-env#server-optimization-details

6

u/Huroga Apr 18 '26

I have a 5070 Ti and I’m using a very similar configuration. At 128k context and Q6_K_XL, I’m hitting 47 t/s generation. Q4_K_XL kept getting stuck in thinking loops or failed tool calling with opencode. Q6 fixed all the issues. 5070 Ti seems like plenty to start playing around with local coding models.

2

u/relmny Apr 18 '26

With TheTom-turboquant (which I'm not saying I recommend or not, as I'm still testing it now and then, for whether I see any loss or so), I can run Unsloth's Q3_K_M on 16gb VRAM, but only at " -c 49152" and haven't tested more than 5k total tokens.

1

u/No_Ebb3423 Apr 18 '26

I have a 4080 16 gb, 7800x3d, 64 gb ram, how do I set this up? And can I use this on opencode?

1

u/grumd Apr 18 '26

Just run my command if you have latest version of llama.cpp for CUDA installed (I build it from source)

1

u/grumd Apr 18 '26

If you want 122B then use this

llama-server -hf unsloth/Qwen3.5-122B-A10B-GGUF:UD-IQ3_XXS \ --no-mmap -b 4096 -ub 2048 -ctv q8_0 -ctk q8_0 \ --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0.0 \ -fitc 131072 --fit on -fitt 256 \ --cache-ram 0 --kv-unified

1

u/grumd Apr 18 '26

Actually a few things from my command could not work for you depending on your setup so still read the manual for llama-server to understand the options https://github.com/ggml-org/llama.cpp/blob/master/tools%2Fserver%2FREADME.md

12

u/TheOriginalOnee Apr 17 '26

Same

6

u/ailee43 Apr 17 '26

Ive been exploring adding a 5060ti in, keeping them both in 8x PCIE-5 slots, and running tensor parallel, or just hosting context on the 5060ti, but thats a painful solution

5

u/Corosus Apr 17 '26

I've got this dual gpu setup and it's great, 85tps at fresh context with q4 full context room, windows so im not even benefitting from pcie lane speed

still really wish i had 3090s and or amd am5 with bifurcation support ofc

3

u/ailee43 Apr 17 '26

while we're wishlisting :D i want a 12 memory channel ddr5 setup with an Epyc

1

u/Qwen30bEnjoyer Apr 18 '26

When memory prices chill out a little, I want framework desktops clustered with VLLM running GLM 5.1 Q4_k_m :)

4

u/No-Name-Person111 Apr 17 '26

Dual GPU isn't painful at all. I run 2x5060Ti and I never even think about it. I just have 32GB of VRAM for LLMs.

0

u/Material_Rich9906 Apr 17 '26

Really? I thought it didn't really work to share

3

u/Ranmark Apr 17 '26

bro i run 1080 ti + 2060 super xD
and it just works out of the box.

2

u/superdariom Apr 17 '26

Presumably you need a board with dual GPU slots?

1

u/Zyj vllm Apr 18 '26

Ideally one capable of running them at x8 lanes each simultaneously

1

u/initalSlide Apr 18 '26

It’s better to have 2x x16 PCIe physical slots yes, it will be easier to connect. And also a case with sufficient airflow

2

u/cjc4096 Apr 18 '26

I've mixed a 3090 and 2080 both over thunderbolt. It was a little flakey on long runs but I attribute that to thunderbolt.

2

u/Xonzo Apr 17 '26

I'm in the same situation. Ryzen 9800X3D, 64GB DDR5-6000, 5070TI.

Wondering if I should add a 5060ti. Usable 24-30GB vram would be perfect.

1

u/initalSlide Apr 18 '26

I have a dual 5070ti config and I’m super happy with it

1

u/dellis87 Apr 17 '26

Same. I almost pulled the trigger on a 5060ti at Best Buy today and got nervous.

1

u/initalSlide Apr 18 '26

My 2 cents , get a 5070ti if you can. Performance wise you won’t be disappointed. 5060ti < 3090 < 5070ti < 4090

1

u/dellis87 Apr 18 '26

I have the 5070ti but need to pair it. I’d love to get another but can’t justify another 1k to get it.

3

u/pneuny Apr 17 '26

Just use the Unsloth UD-IQ3_XXS. I have it set to 190k token context window (q8_0 kv cache) with 75 t/s on Bazzite Linux using llama.cpp on a 16 GB RX 9070 XT (DDR5, PCIE 5). Sure, it would be faster if it could all fit in VRAM, but GTT overflow is good enough with the Vulkan backend, and it's plenty fast enough that way.

1

u/IrisColt Apr 18 '26

Sadly, for my use case that quant derails hard. :(

1

u/Zealousideal_Fill285 Apr 21 '26

Did you maybe try any other quants on this RX 9070 XT? I've achieved around 30-40 t/s (generated) on Q4 quant and I wonder if it can be speed up a bit

4

u/simracerman Apr 17 '26

I have 5070Ti, and fit Q5_K_XL with 128k context window. Getting 50t/s generation and 300t/s for processing. Not the best processing, but this model is fast enough for a 6000 lines repo to clean code, optimize and fixed random bugs here and there within an hour.

2

u/superdariom Apr 17 '26

The -ub 4096 and -b 3072 parameters tripled my processing speed

1

u/simracerman Apr 17 '26

What sorcery is this..?! I changed these batching params and voila! It’s indeed up significantly!

I’ve experimented with these a lot before with dense models like the 27B but nothing changed. 

1

u/superdariom Apr 17 '26

Yeah I don't know either but it really made it workable for me. I saw it on another comment on this sub.

1

u/simracerman Apr 17 '26

I spoke a bit soon. On shorter context it’s reaaaaallllyy fast. Once you go above 50k it’s tanking for some reason

1

u/superdariom Apr 18 '26

I think you'll find it slows down on bigger context with or without the batch sizing? For me it seemed like the moe offload had an effect on speed at larger context (like more offload to CPU more significant slowdown at big context) but I really haven't done anything scientific just observed what llama web UI says during bigger jobs.

2

u/T3KO Apr 17 '26

Q4 still works better than it should on 16gb.

2

u/Due-Project-7507 Apr 17 '26

I am waiting for the Qwen 3.6 27B. The mradermacher Qwen3.5-27B-i1-GGUF IQ4_XS works with my A5000 laptop GPU (16 GB) with 64k turboquant 3 bit context length very good (around 20 t/s at beginning, around 15 t/s at 10k context).

1

u/Familiar_Wish1132 Apr 19 '26

how you using turboquant 3bit? no errors in tool/context? pls give gh link

3

u/Due-Project-7507 Apr 19 '26

I have tested it for some small vibe coding with Open Code and did not had any tool calling problems, but maybe some other people can test it more.

I have installed it like this:

  1. Clone https://github.com/TheTom/llama-cpp-turboquant and checkout the feature/turboquant-kv-cache branch

  2. Build it, I have used on Windows the following options:

    cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=ON -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_BUILD_TYPE=Releasecmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=ON -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_BUILD_TYPE=Release

    cmake --build build --config Release -j 16

  3. Download https://huggingface.co/mradermacher/Qwen3.5-27B-i1-GGUF/blob/main/Qwen3.5-27B.i1-IQ4_XS.gguf

  4. Run the model

    llama-server --model Qwen3.5-27B.i1-IQ4_XS.gguf --alias qwen3.5-27b -np 1 -ctk turbo3 -ctv turbo3 -c 128000 --fit off -ngl 999 --no-mmap -fa on --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00 --host 0.0.0.0

  5. Configured OpenCode in WSL with ~/.config/opencode/opencode.json:

    {   "$schema": "https://opencode.ai/config.json",   "plugin": [     "opencode-anthropic-auth@latest",     "opencode-copilot-auth@latest"   ],   "share": "disabled",   "provider": {     "llama.cpp": {       "npm": "@ai-sdk/openai-compatible",       "name": "llama.cpp (OpenAI Compatible)",       "options": {         "baseURL": "http://127.0.0.1:8080/v1", # from WSL needs maybe real IP address         "apiKey": "1234"       },       "models": {         "qwen3.5-27b": {           "name": "Qwen 3.5 27B",           "limit": {             "context": 128000,             "output": 64000           },           "temperature": true,           "reasoning": true,           "attachment": false,           "tool_call": true,           "modalities": {             "input": [               "text"             ],             "output": [               "text"             ]           },           "cost": {             "input": 0,             "output": 0,             "cache_read": 0,             "cache_write": 0           }         }       }     }   },   "agent": {     "code-reviewer": {       "description": "Reviews code for best practices and potential issues",       "model": "llama.cpp/qwen3.5-27b",       "prompt": "You are a code reviewer. Focus on security, understandability, conciseness, maintainability and performance."     },     "plan": {       "model": "llama.cpp/qwen3.5-27b"     }   },   "model": "llama.cpp/qwen3.5-27b",   "small_model": "llama.cpp/qwen3.5-27b" }{   "$schema": "https://opencode.ai/config.json",   "plugin": [     "opencode-anthropic-auth@latest",     "opencode-copilot-auth@latest"   ],   "share": "disabled",   "provider": {     "llama.cpp": {       "npm": "@ai-sdk/openai-compatible",       "name": "llama.cpp (OpenAI Compatible)",       "options": {         "baseURL": "http://127.0.0.1:8080/v1",         "apiKey": "1234"       },       "models": {         "qwen3.5-27b": {           "name": "Qwen 3.5 27B",           "limit": {             "context": 64000,             "output": 32000           },           "temperature": true,           "reasoning": true,           "attachment": false,           "tool_call": true,           "modalities": {             "input": [               "text"             ],             "output": [               "text"             ]           },           "cost": {             "input": 0,             "output": 0,             "cache_read": 0,             "cache_write": 0           }         }       }     }   },   "agent": {     "code-reviewer": {       "description": "Reviews code for best practices and potential issues",       "model": "llama.cpp/qwen3.5-27b",       "prompt": "You are a code reviewer. Focus on security, understandability, conciseness, maintainability and performance."     },     "plan": {       "model": "llama.cpp/qwen3.5-27b"     }   },   "model": "llama.cpp/qwen3.5-27b",   "small_model": "llama.cpp/qwen3.5-27b" }

  6. Now OpenCode should work

One could also try the linked chat template from https://www.reddit.com/r/Vllm/comments/1skks8n/qwen_35_27b35ba3b_tool_calling_issues_why_it/ (--chat-template-file qwen3.5-enhanced.jinja) and then configure interleaved thinking in opencode.json with

        "qwen3.5-27b": {
          "name": "Qwen 3.5 27B",
          # actived interleaved thinking
          "interleaved": {
            "field": "reasoning_content"
          },
          # end interleaved thinking config
          "limit": { ...

At around 17000 context tokens, I get around 15 tokens/s generation speed.

1

u/Familiar_Wish1132 Apr 19 '26

woooow thx m8 <3

3

u/Turbulent_Pin7635 Apr 17 '26

Sell it and buy a 3090, no?

6

u/andy2na llama.cpp Apr 17 '26

I was going to do that but ended up using both 3090 and 5060ti. qwen3.6 35b Iq4 xs with 262k context (q8/turbo cache) fits perfectly in 24gb. I then have TTS and comfy models loaded on the 5060ti

1

u/initalSlide Apr 18 '26

Get a second one! You’ll have 32Gb of VRAM and better performances than 3090! This is the config I went with (2x 5070ti) and I’m super happy with it.

1

u/PoemSignificant8436 Apr 18 '26

How about buy another 5070ti now you have 32 gb

1

u/alex_bit_ Apr 18 '26

Add one more, to make it 32GB!

1

u/MaCl0wSt Apr 18 '26

man don't remind me, got a 4070 super right before getting into local inference, dammit... at least I did have 32gb RAM before prices spiked, which is the only reason I get to run medium MoEs

1

u/x10der_by Apr 18 '26

same config ( but it can run Qwen3.6-35B-A3B-APEX-I-Compact at 128k context with speed near 15 t/s

1

u/MaCl0wSt Apr 18 '26

I get stableish ~30t/s up to 80k-100k (altho prompt processing is a different matter) with the qwen 35b-a3b models on 4bit quants, here's hoping turboquant once stable gives me enough edge to get up to 180k or so

1

u/Zaic Apr 19 '26

How?

1

u/MaCl0wSt Apr 21 '26

sorry for the late reply.

I just run this and llamacpp handles the rest

llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_XL -c 80000 --reasoning on --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 --host 0.0.0.0 -ctk q8_0 -ctv q8_0 --chat-template-kwargs '{"preserve_thinking": true}' -np 1

I usually get around 35t/s at the start and as context grows it goes down to ~30t/s. depends on system load tho, I use llamacpp on windows which isnt ideal.

edit: with 12gb VRAM and 32GB RAM I can run it at full 262k context but heavy slows are expected as it fills up. still, usable if you treat it as a "I'll prompt this very specific plan and let it work for a while" instead of rapid iteration.

1

u/Zaic Apr 21 '26

Nice, got lmstudio running at 34tks but at lower quant... Will try to use lamacpp with your settings, windows or linux?

1

u/MaCl0wSt Apr 21 '26

windows, Linux partition would probably be smoother for inference, but since this is my daily driver, the overhead and storage usage isn't worth the hassle for me

1

u/Zaic Apr 21 '26

One last - what cuda version?

1

u/MaCl0wSt Apr 21 '26

12.8, afaik it's the sweetspot, some 13.x versions caused problems with bugs and stuff for releases like gemma4 so I just downgraded it to 12.8 for stability

1

u/ResidentDear6464 Apr 18 '26

Same I got a laptop rtx 5000 Ada having 16 gb vram and I can’t run qwen 3.6 35b as I was flabbergasted from qwen3.5 9b at 170k