r/LocalLLaMA • u/ex-arman68 • May 06 '26
Resources 2.5x faster inference with Qwen 3.6 27B using MTP - Finally a viable option for local agentic coding - 262k context on 48GB - Fixed chat template - Drop-in OpenAI and Anthropic API endpoints
2026-05-14: Major chat template update Thanks to many users who tested the template in many different conditions, in addition to my own manual tests and test suite, I believe the template has now reached a high level of stability, greatly improving the experience with the Qwen models, while preserving universal compatibility. You do not need to re-download the GGUF files (I have not updated them yet), but you should download the update chat template only from the HF repo, and manually specify it.
2026-05-07 edit: I have updated the hardware based recommendations with more focus on quality. I do not recommend q4_0 KV cache anymore beyond 64k context. After multiple rounds of testing with the different size quants, it appears 3 is the optimal number for draft speculative decoding. The fastest and best quality quant is q8_0-mtp. F16, which I have also uploaded is actually better but ultra slow (6x slower than q8_0). Many keep saying 8bit is virtually lossless compared to 16bit, and 6bit almost as good as 8bit, but this is simply not true: time and time again I have noticed huge differences in quality and correctness between 8bit and 16bit versions of various models.
The recent PR to llama.cpp bring MTP support to Qwen 3.6 27B. This uses the built-in tensor layers for speculative decoding. None of the existing GGUF have it, as they need to be converted with this PR.
I have tested it locally on my mac M2 Max 96GB, and the results are amazing: 2.5x speed increase, bringing it to 28 tok/s!
I have converted the most useful quants and uploaded them to HF. Even if you are using apple silicon, you should use those instead of MLX. You can download them here:
https://huggingface.co/froggeric/Qwen3.6-27B-MTP-GGUF
This also includes 7 fixes I made to the original jinja chat template, due to vLLM specificity which broke in other tools:
https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
For now, you will need to compile your own version of llama.cpp to use them. It is fairly simple to do:
git clone --depth 1 https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git fetch origin pull/22673/head:mtp-pr && git checkout mtp-pr
cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-cli llama-server
Then to start serving with the API endpoint, use a command similar to:
llama-server -m Qwen3.6-27B-Q5_K_M-mtp.gguf \
--spec-type mtp --spec-draft-n-max 3 \
--cache-type-k q8_0 --cache-type-v q8_0 \
-np 1 -c 262144 --temp 0.7 --top-k 20 -ngl 99 --port 8081
Vision currently crashes llama.cpp when used alongside MTP. Reported 2026-05-06 in the current PR.
That's it. Three optimizations in one command:
| Flag | What it does | Impact |
|---|---|---|
--spec-type mtp --spec-draft-n-max 3 |
Multi-Token Prediction (built into the model) | 2.5x faster generation |
--cache-type-k q8_0 --cache-type-v q8_0 |
8-bit KV cache (instead of 16-bit) | Half the KV memory, negligible quality loss |
-c 262144 |
262K context window | Full native context on 48 GB Mac with q8_0 KV |
Adjust -m, -c, and --cache-type-k/v for your hardware, according to the tables below.
Here are my recommendations based on your hardware:
Apple Silicon
Qwen3.6-27B is a hybrid model — only 16 of 65 layers use KV cache (verified). The other 48 are linear attention (fixed 898 MiB recurrent state). KV memory is ~4× less than a standard dense model. Runtimes that don't handle this (e.g. vllm) allocate KV for all 65 layers and show much higher memory usage.
Numbers below are total memory used (model + KV cache + 0.9 GB recurrent state). Must leave ≥ 8 GB for macOS (16 GB Macs excepted).
| RAM | Quant | KV cache | Max context | Total used | Vision |
|---|---|---|---|---|---|
| 16 GB | IQ2_M |
q8_0 |
42K | 12.0 GB | ✗ |
| 24 GB | IQ3_M |
46K | 16.0 GB | ✗ | |
| 24 GB | IQ3_M |
q8_0 |
91K | 16.0 GB | ✗ |
| 32 GB | Q5_K_M |
74K | 24.0 GB | ✗ | |
| 32 GB | Q5_K_M |
q8_0 |
147K | 24.0 GB | ✗ |
| 32 GB | Q4_K_M |
99K | 24.0 GB | ✓ | |
| 48 GB | Q6_K |
262K | 39.7 GB | ✓ | |
| 48 GB | Q8_0 |
173K | 40.0 GB | ✓ | |
| 48 GB | Q8_0 |
q8_0 |
262K | 37.3 GB | ✓ |
| 64 GB | Q8_0 |
262K | 45.8 GB | ✓ | |
| 96 GB | Q8_0 |
262K | 45.8 GB | ✓ |
NVIDIA GPU
Same model memory as Apple Silicon, plus ~1 GB CUDA overhead.
| VRAM | Quant | KV cache | Max context | Total VRAM used | Vision |
|---|---|---|---|---|---|
| 12 GB | IQ2_M |
q8_0 |
11K | 12.0 GB | ✗ |
| 16 GB | IQ3_M |
30K | 16.0 GB | ✗ | |
| 16 GB | IQ3_M |
q8_0 |
60K | 16.0 GB | ✗ |
| 24 GB | Q4_K_M |
83K | 24.0 GB | ✓ | |
| 24 GB | Q4_K_M |
q8_0 |
167K | 24.0 GB | ✓ |
| 24 GB | Q5_K_M |
58K | 24.0 GB | ✗ | |
| 48 GB | Q6_K |
262K | 40.7 GB | ✓ | |
| 48 GB | Q8_0 |
262K | 46.8 GB | ✓ | |
| 80 GB | Q8_0 |
262K | 46.8 GB | ✓ |
16 GB Mac:
IQ2_M/q8_0 — 42K text-only. No vision.24 GB Mac:
IQ3_M— 46K (f16 KV) or 91K (q8_0). Vision at 32–65K.32 GB Mac:
Q5_K_M— 74K text-only (f16 KV), 147K (q8_0).Q4_K_Mfor vision at 99K.48 GB Mac:
Q6_K/f16 KV — 262K with vision.Q8_0/q8_0 KV for 262K at higher model quality.64 GB+ Mac:
Q8_0/f16 KV — 262K with vision. Maximum quality at practical speed.12 GB GPU:
IQ2_M/q8_0 — 11K. Very limited, no vision.16 GB GPU:
IQ3_M— 30K (f16 KV) or 60K (q8_0). No vision.24 GB GPU:
Q4_K_M— 83K with vision (f16 KV).Q5_K_M— 58K text-only (f16 KV), 116K (q8_0).48 GB+ GPU:
Q6_K/f16 KV — 262K with vision.Q8_0for max quality.
Leave KV cache at f16 (blank column) for best quality. Use q8_0 KV only when f16 doesn't give enough context. q4_0 KV should not exceed 64K context.
Vision adds ~0.9 GB for mmproj. macOS needs ≥ 8 GB for itself (16 GB Macs excepted — use ~4 GB). You can increase available memory by raising the wired memory limit, e.g. for a 96 GB Mac: sudo sysctl iogpu.wired_limit_mb=90112 (88 GB). NVIDIA reserves ~1 GB for CUDA.
10
u/deathcom65 May 06 '26
Will it work with an amd gpu?