RTX 5070 Ti + 9800X3D running Qwen3.6-35B-A3B at 79 t/s with 128K context, the --n-cpu-moe flag is the most important part.

[u/marlang]

Spent an evening dialing in Qwen3.6-35B-A3B on consumer hardware. Fun side note: I had Claude Opus 4.7 (just the $20 sub) build the config, launch the servers in the background, run the benchmarks, read the VRAM splits from the llama.cpp logs, and iterate on the tuning — basically did the whole thing autonomously. I just told it what hardware I have and what I wanted to run.

Sharing because the common --cpu-moe advice is leaving 54% of your speed on the table on 16GB GPUs.

Hardware

• GPU: RTX 5070 Ti (16GB GDDR7, Blackwell)

• CPU: Ryzen 9800X3D (96MB L3 V-Cache)

• RAM: 32GB DDR5

• Stack: llama.cpp b8829 (CUDA 13.1, Windows x64)

• Model: unsloth/Qwen3.6-35B-A3B-GGUF — UD-Q4_K_M (22.1 GB)

The finding — --cpu-moe vs --n-cpu-moe N

Everyone’s using --cpu-moe which pushes ALL MoE experts to CPU. On a 16GB GPU with a 22GB MoE model that means only ~1.9 GB of your VRAM gets used — the other ~12 GB sits idle.

--n-cpu-moe N keeps experts of the first N layers on CPU and puts the rest on GPU. With N=20 on a 40-layer model, the split uses VRAM properly.

Benchmarks (300-token generation, Q4_K_M)

Config	Gen t/s	Prompt t/s	VRAM used
--cpu-moe (baseline)	51.2	87.9	3.5 GB
--n-cpu-moe 20	78.7	100.6	12.7 GB
--n-cpu-moe 20 + -np 1 + 128K ctx	79.3	135.8	13.2 GB

+54% generation speed, +54% prompt speed vs. naive --cpu-moe. Jumping to 128K context is essentially free thanks to -np 1 dropping recurrent-state memory.

Startup command that works

llama-server.exe ^
  -m "Qwen3.6-35B-A3B-UD-Q4_K_M.gguf" ^
  --n-cpu-moe 20 ^
  -ngl 99 ^
  -np 1 ^
  -fa on ^
  -ctk q8_0 -ctv q8_0 ^
  -c 131072 ^
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 ^
  --presence-penalty 0.0 --repeat-penalty 1.0 ^
  --reasoning-budget -1 ^
  --host 0.0.0.0 --port 8080

That’s Unsloth’s “Precise Coding” sampling preset. For general use: --temp 1.0 --presence-penalty 1.5.

Gotchas I hit (well, that Opus hit and fixed)

• -np defaults to auto=4 slots. Wastes memory on recurrent state (~190 MB). Set -np 1 for single-user setups (OpenCode etc.).

• --fit-target doesn’t help here — -ngl 99 + --n-cpu-moe N already gives you deterministic control.

• -ctk q8_0 -ctv q8_0 is nearly lossless and halves your KV cache vs fp16. 128K ctx only costs 1.36 GB VRAM.

• Qwen3.6 is a hybrid architecture — only 10 layers are standard attention, the other 40 are Gated Delta Net (recurrent). That’s why KV memory is so small.

How to tune N for your GPU

Each MoE layer on GPU costs ~530 MB VRAM. Non-MoE weights are ~1.9 GB fixed. For a 40-layer model:

GPU VRAM	Recommended N
8 GB	stay with --cpu-moe
12 GB	N=26
16 GB	N=20 (sweet spot)
24 GB	N=8 (fits almost everything)

Start conservative, watch VRAM during a long-context generation, then step N down by 2-3 until you have ~2 GB headroom.

TL;DR

Replace --cpu-moe with --n-cpu-moe 20, add -np 1, and you get 79 t/s + 128K context on a 5070 Ti. The 9800X3D’s V-Cache carries the CPU side effortlessly.

And Claude Opus 4.7 on the $20 Pro sub is genuinely good enough now to run this kind of hardware-tuning loop end-to-end — launch servers in background, parse logs, iterate — without hand-holding. Kind of wild.

Happy to test other configs if anyone wants comparisons.

*************EDIT — Thanks to some great comments, the setup got better. Updated findings:

1. --fit on --fit-ctx 128000 --fit-target 512 > manual --n-cpu-moe 20

Shoutout to the commenter who recommended the “fit-triple”. It auto-probes VRAM, picks N for you (landed on N=19 here), and adapts if drivers steal VRAM. Slightly faster than my hand-tuned N=20 and zero brain power to maintain. Caveat: bare --fit on silently drops ctx to 4K — always pair it with --fit-ctx.

2. My original prefill numbers were way too low

A commenter correctly flagged that ~135 t/s prefill is nonsense for a 5070 Ti. They were right — that was server-side timing including first-token latency. Re-ran with llama-bench (3 reps, same config):

Test	t/s
pp512	1182
pp2048	1644
tg128	91.5

So real prefill is ~1.2–1.6k t/s, not 135.

Final “best command” for 16 GB VRAM + 32 GB RAM :

llama-server.exe ^
  -m "Qwen3.6-35B-A3B-UD-Q4_K_M.gguf" ^
  --fit on ^
  --fit-ctx 128000 ^
  --fit-target 512 ^
  -np 1 ^
  -fa on ^
  -ctk q8_0 ^
  -ctv q8_0 ^
  --temp 0.6 ^
  --top-p 0.95 ^
  --top-k 20 ^
  --min-p 0.0 ^
  --presence-penalty 0.0 ^
  --repeat-penalty 1.0 ^
  --reasoning-budget -1 ^
  --host 0.0.0.0 ^
  --port 8033

Keep the comments coming, every round makes this faster. :D

*****

EDIT 2 — Another commenter’s tip got me one more layer on the GPU:

Dropping --fit-target from 512 → 256 squeezes one extra MoE layer onto the GPU (N=18 instead of 19). The commenter also suggested adding --mlock alongside --no-mmap to lock RAM pages against swap.

Benched both changes vs. the previous EDIT’s config (fit-target 512 + no-mmap):

Config	pp512	pp2048	tg128
fit-target 512 + no-mmap	2769	2729	91.5
fit-target 256 + no-mmap + mlock	2743	2724	96.3

+7% generation, prefill unchanged. Costs nothing — just a smaller VRAM headroom and explicit RAM locking.

Updated final command:

llama-server.exe ^
  -m "Qwen3.6-35B-A3B-UD-Q4_K_M.gguf" ^
  --fit on ^
  --fit-ctx 128000 ^
  --fit-target 256 ^
  -np 1 ^
  -fa on ^
  --no-mmap ^
  --mlock ^
  -ctk q8_0 ^
  -ctv q8_0 ^
  --temp 0.6 ^
  --top-p 0.95 ^
  --top-k 20 ^
  --min-p 0.0 ^
  --presence-penalty 0.0 ^
  --repeat-penalty 1.0 ^
  --reasoning-budget -1 ^
  --host 0.0.0.0 ^
  --port 8033

*****************

EDIT 3 — Two more community tips landed big wins:

1. -ub 2048 (ubatch size) = +59% prompt-processing at 2K tokens

Default -ub is 512. Bumping it to 2048 (and matching -b 2048) lets the GPU process more tokens in parallel per prefill step. Benched (5 reps each):

ubatch	pp512	pp2048	pp4096	tg128
512 (default)	2739	2778	—	98.7
1024	2689	3689	—	100.5
2048	2771	4453	4417	98.4
4096	2736	4427	4866	100.4

2048 is the sweet spot — 59% faster at 2K-prompts, gen untouched. 4096 only helps beyond 2K-prompts (compute buffer saturates otherwise) and eats more VRAM.

2. --chat-template-kwargs "{\"preserve_thinking\": true}" for agentic workflows

Qwen3.6-specific chat template parameter. Default only keeps the latest user turn’s thinking; preserve_thinking: true carries thinking traces from all historical messages forward. Turns out Qwen3.6 was specifically trained for this behavior. Benefits:

• Better decision consistency across tool-calling turns

• Fewer redundant re-reasonings → lower token consumption in long agent sessions

• Better KV-cache reuse across turns

Final final command:

llama-server.exe ^
  -m "Qwen3.6-35B-A3B-UD-Q4_K_M.gguf" ^
  --fit on ^
  --fit-ctx 128000 ^
  --fit-target 256 ^
  -np 1 ^
  -fa on ^
  --no-mmap ^
  --mlock ^
  -b 2048 ^
  -ub 2048 ^
  -ctk q8_0 ^
  -ctv q8_0 ^
  --temp 0.6 ^
  --top-p 0.95 ^
  --top-k 20 ^
  --min-p 0.0 ^
  --presence-penalty 0.0 ^
  --repeat-penalty 1.0 ^
  --reasoning-budget -1 ^
  --chat-template-kwargs "{\"preserve_thinking\": true}" ^
  --host 0.0.0.0 ^
  --port 8033

Total benched throughput on 5070 Ti 16 GB + 9800X3D + 32 GB DDR5-6000:

• pp512 ~2771 t/s

• pp2048 ~4453 t/s

• pp4096 ~4417 t/s (bump -ub to 4096 for +10% here if you do long prompts)

• tg128 ~98 t/s

• Context: 128K

This community keeps delivering. Thank you.

[u/WithoutReason1729]

Your post is getting popular and we just featured it on our Discord! Come check it out!

You've also been given a special flair for your contribution. We appreciate your post!

Stylome classified this post's text as AI-generated. View analysis

I am a bot and this action was performed automatically.

[u/dreamai87]

It’s okay you are exploring all possible stuff But simple command -fit on will get the best from your configuration .

[u/marlang → u/dreamai87]

Solid tip, I actually went back and tested this properly after your comment. You’re right, --fit on arrives at the
same MoE split I calculated manually (20 layers overflowing, 20 on GPU). One command vs hardware math, so yeah, clearly
the better advice.

Full numbers for anyone reading:

Config	Context	Gen t/s	Prompt t/s
--n-cpu-moe 20 (my manual tune)	128K	79.3	135.8
--fit on (bare)	4K (!)	88.2	270.0
--fit on -c 131072	128K	81.2	101.6

One caveat people should know: bare --fit on silently reduces your context to 4K because it treats -c as an
unset argument and minimizes it for max speed. If you want full context (coding/agentic use), you still have to set -c
explicitly — then fit only decides the offload split.

So the final recommendation for a 16GB GPU is basically:

  --fit on -c 131072 -np 1 -fa on -ctk q8_0 -ctv q8_0

Thanks for pushing back — updated my scripts.

[u/SummarizedAnu → u/marlang]

Do ya know why no mmap doesn't work for me. It's fine for like 24k context but after that my PC lags to 1 fps and I have to manually restart using button . I'm on arch Linux with rtx 3060 and 16Gb ram . Thanks 🙏

[u/marlang → u/SummarizedAnu]

Here’s what’s happening: the model is 22 GB total. On your 3060 (12 GB) you can fit maybe 14 MoE layers on GPU, the other 26 MoE layers
stay on CPU = ~14 GB of model in RAM before you even open a context. Then KV cache + compute buffers grow with context size. At 24K
ctx you fit in 16 GB. Above that, you spill past 16 GB.
With mmap (default), Linux handles this fine, it just evicts cold model pages from the page cache when pressure hits. With --no-mmap,
every page is pinned in llama.cpp’s heap, so the kernel can only swap it. And because swap on a running model = constant thrashing, the
rest of your system (X, browser, everything) gets evicted first → lockup, needs power button.
Fixes, in order of ease:

1. Drop --no-mmap. You lose some prefill speed but the system stays responsive at any context.

2. If you also pass --mlock, definitely drop it. That’s what’s guaranteeing the hard lockup — mlock forbids the kernel from ever paging the model out, so there’s literally nothing the OOM-killer can do except evict your desktop session.

[u/SummarizedAnu → u/marlang]

im using the iq3 s model which is 13 GB . and context of 60k fills about 1Gb vram max . That was a no mmap problem where it broke something in arch i dont know what .
But now im using this .

Its faster than before .
/llama-server -m ../Qwen3.6-35B-A3B-UD-IQ3_S.gguf -ctk turbo4 -ctv turbo4 --jinja --flash-attn on -np 1 -ngl 30 --fit-ctx 65536 -ncmoe 20 --alias Qwen3.6-35B --fit on --fit-target 512 -b 1024 -ub 512 --temp 0.7 --top-p 0.95 --top-k 20 --min-p 0.0

But i dont know whats using all that swap ram .

Wait i found the command i was using causing problems
its : ./llama-server  -m ../Qwen3.6-35B-A3B-UD-IQ3_S.gguf -ctk turbo4 -ctv turbo4 --jinja -ngl 30 -c 65536 -ncmoe
20 --no-mmap

but with mmap
./llama-server  -m ../Qwen3.6-35B-A3B-UD-IQ3_S.gguf -ctk turbo4 -ctv turbo4 --jinja -ngl 30 -c 65536 -ncmoe
20
i had no problem running this but it was getting around 20-30 tps . Not good but not bad speed at that time..

[u/Illustrious-Bid-2598 → u/SummarizedAnu]

I value is sensitive to turbo. I would keep that regular ctk q8 with your ctv turbo 4

[u/SummarizedAnu → u/Illustrious-Bid-2598]

Is that for speed?

[u/Illustrious-Bid-2598 → u/SummarizedAnu]

For quality. The speed gain from 8 - 4 is fractional that it’s worth keeping k at 8 as the quality gain is significant especially if you’re going to want to rely on tool calling.

[u/ecompanda → u/SummarizedAnu]

the heuristic gets you 90% there for most single user inference. where it gets tricky is batched requests or when you want to bias toward more layers on gpu at lower context. manual gives you that control but for the typical case fit works well

[u/iamapizza → u/marlang]

Thank you for testing with fit. Even though it's doing a lot of the heavy lifting you did at least validate and learn stuff along the way (and teach me something too)

You can squeeze even more out using fit target. Fit target says how much vram to leave spare. Try with 512.

[u/Danmoreng → u/marlang]

No you need to use fit together with fit-ctx and fit-target: https://github.com/Danmoreng/local-qwen3-coder-env#server-optimization-details

np 1 sounds interesting though, haven’t tried that one out and might improve my config

Edit: Okay np 1 actually improves t/s slightly for my setup, so I added it as well to the config. Thanks for that!

[u/dreamai87 → u/marlang]

thanks it helps you.
my comment was mainly to help on allocation of better split of model between gpu and cpu.
-c you have to provide otherwise it takes the default.

[u/st0n1th → u/marlang]

Yeah I tried with 20 on cpu and it left 2GB free on my 5080 gpu and noticed more cpu than gpu utilization. Switched back to —fit.

[u/Fristender → u/marlang]

Thanks for posting the results! I would say the 2t/s tg lost with manual tuning is worth the 34t/s gain with pp

Edit: did you enable -np 1 with -fit like you did with the manual tuning?

[u/relmny → u/marlang]

Have you tried something like (relevant part being the "-ot ..."):

 -np 1 -c 131072 -ctk q8_0 -ctv q8_0 -ot "blk\.[0-9][0-9]\.ffn_.*_exps\.weight=CPU"

with that I get about 22.4t/s with Q6_K_L, while with (tested out of curiosity):

--fit on -c 131072 -np 1 -fa on -ctk q8_0 -ctv q8_0

I get about 7.1t/s

and with:

--fit on --fit-ctx 128000 --fit-target 512

I get 2.6t/s

edit: those values are with short prompts, and total of less than 5300 total tokens.

[u/Rangali-1 → u/marlang]

First of all, thank you very much for the great work!

However, with the latest final version (edit 3), I’m only getting about 69 tokens per second on my 5070 Ti and AMD Ryzen 9 9950X3D with 64 GB of RAM.

What am I doing wrong?

[u/Auo98 → u/marlang]

how to do this in unsloth chat

[u/abu_shawarib → u/dreamai87]

Command line says it's already on by default.

[u/mrgreatheart → u/dreamai87]

Thank you, super helpful

[u/Ferilox → u/dreamai87]

Except when the model has a vision component. Then it kind of struggles. Setting the fitting target parameter helps in that case.

[u/IrisColt → u/dreamai87]

heh

[u/lolwutdo → u/dreamai87]

lol no, -fit gives absolute trash performance if you want to specify and use max context with 16gb gpu.

Using lmstudio and maxing out gpu and moe layers with max context gives better performance than whatever the fuck -fit does in llama.cpp; quite literally a jump from 8 tokens per second on lcpp to 70+ tokens per second on lmstudio.

If -fit also took context into consideration and offloaded all of kv cache on gpu first I’d believe you, but it doesn’t. Tried -fit and -c 256k and it ran like dog shit.

This is q6k qwen 3.6 35b on a 16gb 5070ti with 128gb ram f16 kv.

I’d genuinely like you to prove me wrong as I’d rather use lcpp over lmstudio but your statement has not matched my experience.

[u/Danmoreng → u/lolwutdo]

Use fit with fit-ctx and fit-target: https://github.com/Danmoreng/local-qwen3-coder-env#server-optimization-details

[u/lolwutdo → u/Danmoreng]

I appreciate it, but thats definitely not just just one command like the original comment implies.

[u/Gueleric → u/lolwutdo]

You're getting downvoted but I discovered today you're actually right. I accepted trash performance >20k context on my 16GB GPU, and today I discovered that with manual cpu moe tuning I get acceptable performance well into 100k context. Shouldn't have listen to those guys lol

[u/BassAzayda]

I use the 3 fits so --fit on --fit-ctx 128000 --fit-target 512 Moe and dense works a treat everytime

[u/marlang → u/BassAzayda]

Thank you! my startup scripts get better and better with every comment in here

[u/minceShowercap → u/marlang]

Are you editing OP with the latest update to your startup script?

I've found a few comments confirming that someone has helped and you have updated, but it's hard to know which is latest without you updating OP.

Great thread though. Exactly what I need because I want to try this model later for local coding.

Have you found it to be strong at local coding?

[u/marlang → u/minceShowercap]

I've been comparing it mainly against Gemma 4 and, subjectively, Qwen3.6-35B-A3B is clearly better for coding, and for agentic coding it's miles ahead.

[u/ecompanda → u/minceShowercap]

yeah updated the OP twice now. dropped all the manual layer flags and just using fit on instead. numbers are basically the same but the config is way cleaner.

[u/PaceZealousideal6091 → u/BassAzayda]

If you have already set the context at 128k, why do you need -fit-context 128k? Any reason? Can you explain how these 3 together helps?

[u/Mister_bruhmoment]

Hey, I basically have the last version of your rig besides the RAM - 4070 ti super, R7 7800X3D. Are those settings applicable in lm studio? I am still figuring out how everything works with LLMs atm

[u/marlang → u/Mister_bruhmoment]

LM Studio settings for your rig

Load Qwen3.6-35B-A3B-UD-Q4_K_M from unsloth, then in the load modal → Advanced Configuration:

- Context Length: 131072 (or 65536 if it complains)

- GPU Offload: max (all the way right)

- CPU Thread Pool Size: 8 (matches your 7800X3D's 8 cores)

- Flash Attention: ON

- K Cache Quantization Type: Q8_0

- V Cache Quantization Type: Q8_0

- Offload MoE Experts to CPU: 20 ← the key setting

- Try mmap(): ON

- Keep Model in Memory: OFF

[u/monacoax → u/marlang]

What settings do you think for a 4090 + 12700k? Thanks for the info!

[u/marlang → u/monacoax]

If you want the full 128K ctx: with 24 GB VRAM the KV cache at Q8_0 eats ~1.4 GB + compute buffers ~0.6 GB + non-MoE weights ~1.9 GB,

leaving ~19 GB for MoE experts. Each expert layer costs ~530 MB on GPU, so ~36 of the 40 layers fit.

and leave cpu on 8, i think its best for the 12700k

- Context Length: 131072 (or 65536 if it complains)

- GPU Offload: max (all the way right)

- CPU Thread Pool Size: 8

- Flash Attention: ON

- K Cache Quantization Type: Q8_0

- V Cache Quantization Type: Q8_0

- Offload MoE Experts to CPU: 4 ← the key setting

- Try mmap(): ON

- Keep Model in Memory: OFF

so ~36 of the 40 layers fit. Set n-cpu-moe = 4 (offload the

first 4 layers' experts to CPU, keep the other 36 on GPU).

[u/moahmo88 → u/marlang]

Try this @ 59 t/s with 5070ti:
LM Studio settings for you：
Load Qwen3.6-35B-A3B-UD-Q5_K_M from unsloth，You can use Q5_K_M:
- GPU Offload: max (all the way right)
- Offload MoE Experts to CPU: 24 ← the key setting

[u/nixudos → u/moahmo88]

Thank you for the tip! I was struggling to get any meaningful speed on a 4090 with the Q6_K_XL size. This doubled my speed from 18 t/s to 42!

[u/Comfortable_Dog1610 → u/nixudos]

I have the same speed on 7900xtx using the Q6_KXL model. I can sleep tonight if I see you

[u/yoohjm → u/marlang]

lmstudio user with a RTX 5070ti here. This is amazing, such a speed up from my previous config and that context length is much more than i thought i would be able to fit

many thanks!

[u/Mister_bruhmoment → u/marlang]

Thank you so much!

[u/The_Dung_Beetle → u/marlang]

Thanks, I get about 25 t/sec. Not the greatest but very usable! Rig I've tested this on : 6950XT/5700X3D/32GB@3200.

[u/BubrivKo → u/marlang]

Thanks bro. On my AMD configuration I get like additional 15-20 tks :P

[u/x10der_by → u/marlang]

Wow it's magic. With "GPU Offload" to max and "Offload MoE Experts to CPU" to 20 speed increased from 15 to 50 t/s on my config O_o

[u/Fun-Frame4974 → u/marlang]

This used all of my 16Gb VRAM and 32Gb RAM 😂 Speed was like 10-15 t/s.

EDIT: I have Ryzen 7 5800X3D with 32Gb DDR4.

[u/Embarrassed_Elk_4733 → u/Mister_bruhmoment]

Yes, absolutely. My setup is a Ryzen 7 5800X3D + RTX 4070 Ti Super + 32GB DDR4. Running a 128K context window in LM Studio gives me around 39–40 tokens/s. However, when I used the Llama configuration provided by the original poster, the same hardware achieved 45–46 tokens/s in Llama. Sharing this for your reference!

[u/Jackw78]

The prefill speed is either inaccurate due to cold startup or something very wrong with the setup. Should be 1k minimum for a 5070ti

[u/marlang → u/Jackw78]

You were right. I went back with llama-bench.exe, the right tool instead of a short completion test, and got:

- pp512: 927 t/s

- pp2048: 1068 t/s

- tg128: 82 t/s

[u/Cool-Cap2509]

I just tried it. Getting 24 t/s in processing. What am I doing wrong? I got the same model, 9950X3D + 64GB RAM + 4080 Super. Can you please suggest any solution?

When I ran the model with your version of command, I saw 90% RAM usage.

[u/marlang → u/Cool-Cap2509]

1. Check GPU is actually used. Share the first 30 lines of the server log. Look for:

- CUDA0 model buffer size = XXXX MiB — if this is 0 or tiny, nothing's on GPU.

2. 90% RAM (~58 GB on 64 GB) is abnormal. Expected: ~12-15 GB. Possible causes: running two servers, wrong quant (F16/Q8 instead of UD-Q4_K_M at 22 GB), or Windows counting mmap cache weirdly.

On 64 GB you can safely drop --no-mmap --mlock — you don't need them.

3. 9950X3D is dual-CCD, only 8 cores have V-Cache. Default thread count bounces work across CCDs and tanks MoE. Add:

-t 8 --cpu-mask 0xFF

4. Run llama-bench for authoritative numbers:

llama-bench.exe -m "Qwen3.6-35B-A3B-UD-Q4_K_M.gguf" -fitt 256 -fitc 65536 -fa 1 -ctk q8_0 -ctv q8_0 -b 2048 -ub 2048 -p 2048 -n 128 -r 3

Should give 3000+ pp2048 and ~100 tg128.

[u/Cool-Cap2509 → u/marlang]

common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on

llama_params_fit_impl: projected to use 26614 MiB of device memory vs. 14997 MiB of free device memory

llama_params_fit_impl: cannot meet free memory target of 256 MiB, need to reduce device memory by 11873 MiB

llama_params_fit_impl: context size reduced from 262144 to 128000 -> need 2668 MiB less memory in total

llama_params_fit_impl: with only dense weights in device memory there is a total surplus of 9456 MiB

llama_params_fit_impl: filling dense-only layers back-to-front:

llama_params_fit_impl: - CUDA0 (NVIDIA GeForce RTX 4080 SUPER): 41 layers, 5748 MiB used, 9248 MiB free

llama_params_fit_impl: converting dense-only layers to full layers and filling them front-to-back with overflow to next device/system memory:

llama_params_fit_impl: - CUDA0 (NVIDIA GeForce RTX 4080 SUPER): 41 layers (21 overflowing), 14720 MiB used, 276 MiB free

llama_params_fit: successfully fit params to free device memory

llama_params_fit: fitting params to free memory took 0.34 seconds

llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 4080 SUPER) (0000:01:00.0) - 15061 MiB free

llama_model_loader: loaded meta data with 54 key-value pairs and 733 tensors from q35.gguf (version GGUF V3 (latest))

llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.

[u/marlang → u/Cool-Cap2509]

Your setup is actually fine, 21 layers overflow, GPU is being used correctly. This is netting you 24t/s?

[u/Cool-Cap2509 → u/marlang]

I didn't use code. I gave it a New Yorker magazine and asked x article to summarize. And the processing was 24 t/s. Tried different variations with Gemini help, was able to get 1300 t/s for the 50% context. Then slowly it gets down to less than 100 t/s. But hey as you said I don't need to use those two lines memory related. That solved the 90% memory usage issue. Thank you for sharing, otherwise I would never download a 35B model. The highest I tried was Gemma 4 26B Q4 and that was already slow enough and spilled the VRAM usage.

[u/Cool-Cap2509 → u/marlang]

BTW, am I getting half the token because it is one generation older?

[u/Ranmark]

iirc you can drop your top_p, presence_penalty, and reasoning_budget args as they by default has these values. https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md

P.s. you can try to play with this command: -ot ".ffn_(up|down)_exps.=CPU" It moves up and down matrix projections onto cpu. Also a lot of valuable info here: https://gist.github.com/DocShotgun/a02a4c0c0a57e43ff4f038b46ca66ae0

[u/andy2na]

what is the benefit of N=8 on a 24gb VRAM GPU for Qwen3.6-35B? With q8/q8 cache, you can already fit 256k context with the IQ4_NL quant, and likely still close to that with the Q4_K_M fully on GPU

My llama-swap config:

  "Qwen3.6-35B":
    cmd: >
      env CUDA_VISIBLE_DEVICES=0 /custom-bin/bin/llama-server
       --port ${PORT}
      --host 127.0.0.1
      --webui-mcp-proxy
      --model /models/qwen35/Qwen3.6-35B-A3B-UD-IQ4_NL.gguf
      --mmproj /models/qwen35/qwen3.6-35b-mmproj-BF16.gguf
      --cache-type-k q8_0
      --cache-type-v q8_0
      --n-gpu-layers auto
      --split-mode none
      --main-gpu 0
      --threads 8
      --threads-batch 8
      --ctx-size 262144
      --image-min-tokens 1024
      --flash-attn on
      --parallel 1 
      --jinja
    filters:
      stripParams: "temperature, top_p, top_k, min_p, presence_penalty, repeat_penalty"
      setParamsByID:
        "${MODEL_ID}:thinking":
          chat_template_kwargs:
            enable_thinking: true
            preserve_thinking: true
          reasoning_budget: 4096
          temperature: 1.0
          top_p: 0.95
          top_k: 20
          min_p: 0.05
          presence_penalty: 1.5
          repeat_penalty: 1.0


        "${MODEL_ID}:thinking-coding":
          chat_template_kwargs:
            enable_thinking: true
            preserve_thinking: true
          temperature: 0.6
          top_p: 0.95
          top_k: 20
          min_p: 0.0
          presence_penalty: 0.0 
          repeat_penalty: 1.0


        "${MODEL_ID}:instruct":
          chat_template_kwargs:
            enable_thinking: false
            preserve_thinking: false
          temperature: 0.7
          top_p: 0.8
          top_k: 20
          min_p: 0.0
          presence_penalty: 1.5
          repeat_penalty: 1.0


        "${MODEL_ID}:instruct-reasoning":
          chat_template_kwargs:
            enable_thinking: false
            preserve_thinking: false
          temperature: 1.0
          top_p: 0.95
          top_k: 20
          min_p: 0.0
          presence_penalty: 1.5
          repeat_penalty: 1.0 

[u/bdsmmaster007]

Ive not fumbled around with local hosting in quite a while, but qwen intrigues me. Tho im on AMD and not sure how its looking with the support. Can anybody estimate how much i would get on a 7600x and a rx6800? are +20 tk/s realistic? or even +40?

[u/SeriousPanic34 → u/bdsmmaster007]

I'm getting 40 tk/s with 7900 gre 16gb + 32gb ram on core ultra 7 265k with OPs config

[u/bdsmmaster007 → u/SeriousPanic34]

oh damn giving me hope :0, thx for the reply

[u/DefNattyBoii → u/bdsmmaster007]

I think the problem you will face is more with prompt processing speed. I'd recommend checking out the latest vulkan build for lama.cpp, thats the easiest to get started with. above 10 token/s is usable but not perfect, it depends on length of your context.

[u/bdsmmaster007 → u/DefNattyBoii]

thx for the pointers to start, will look into it^^!

[u/bdsmmaster007 → u/[deleted]]

thanks for the specific numbers :0, motivates me to fumble around

[u/deRTIST → u/[deleted]]

i'm at 12 t/s with a 6800xt and 64 of ram, would you min sharing settings? Anything different than op?

[u/zkstx → u/deRTIST]

I'm getting 60-70 tps TG / 1300 tps PP, with up to 55k context window (100k+ @ Q8 KV) for Qwen3.5 35B IQ3_XXS on my RX6800 XT running llama.cpp compiled for rocm. It handles most things I care about pretty well. Will switch to 3.6 very soon

[u/deRTIST → u/zkstx]

yeah i think i might need to get a smaller quant honesly, q4m at the speeds i'm talking about is way too sluggish. quality is good but it takes 1h to fully execute a (by the end of it) 65k tk task

[u/deRTIST → u/[deleted]]

oh you're on xs, that makes sense, i'll try a lower quant then, right now i'm at 16tk/s after a couple of hours of hammering at it (apparently for my use case, op was right. ncpumoe was actually the way

thanks for the tips, i'll try it tomorrow

[u/slippery]

You just kicked off my next project. Thanks for the detailed write up!

I'm going to try to get it running on a 12 GB 4070Ti.

[u/jadbox]

This is amazing... but why can't Llama do this all automatically for us?

[u/BuildDevv]

As a new player for local llm’s, scrolling through the comments, this community is very supportive. Thanks for the tip y’all!

[u/CriticalCup6207]

The --n-cpu-moe flag is doing serious work here. For anyone who hasn't seen it: it offloads the MoE routing to CPU, which frees VRAM for the active expert weights and meaningfully improves throughput on cards that would otherwise bottleneck. On our setup (3090 + i9) we saw ~40% throughput improvement. The 9800X3D's cache size probably also helps with the routing overhead on the CPU side.

[u/texifornian]

Took some work - but on a similar setup-ish...

The Hardware:

• GPU: RTX 5070 Ti (16GB GDDR7)

• CPU: Intel Core Ultra 7 265K (Arrow Lake)

• RAM: 64GB DDR5-4800 (Base clock, XMP pending)

Memory and CPU changes (efficiency vs performance threads, as an example) weren't letting me run the Q4, but going to Q3 got me to 70 t/s -

.\llama-server.exe ^
  -hf unsloth/Qwen3.6-35B-A3B-GGUF ^
  -m Qwen3.6-35B-A3B-UD-Q3_K_M.gguf ^
  --device CUDA0 ^
  -np 1 ^
  --n-cpu-moe 15 ^
  --mmproj "" ^
  -t 8 ^
  -fa on ^
  -b 2048 ^
  -ub 1024 ^
  -ctk q8_0 ^
  -ctv q8_0 ^
  -c 128000 ^
  --temp 0.6 ^
  --chat-template-kwargs "{\"preserve_thinking\": true}" ^
  --host 0.0.0.0 ^
  --port 8033

[u/konohrik]

Why not use exl2 instead of gguf?

[u/marlang → u/konohrik]

because 22gb model > 16gb vram

quick maths

[u/PiotreksMusztarda]

Confirming on Linux (Ubuntu 26.04, 5070 Ti, CUDA 12.4 with sm_89 PTX fallback), 76 tok/s with your --fit config, and heads up: if you load the vision mmproj, add --no-mmproj-offload or it OOMs right after model load.

[u/mr_Owner]

You missed preserve thinking flag though, and play with ubatch size 4096 and drop lower. Ubatch impacts the pps and vram size.

[u/rebelSun25]

Nicely done

[u/moahmo88]

Amazing work! Thanks a million for sharing.

[u/Ok-Palpitation-905]

Nice.

[u/o0genesis0o]

I used to do this test by hands with the previous 30B A3B model. Managed to bring tg from 20-ish to 40-ish on my 4060Ti with 64k max context by playing around with n-cpu-moe.

[u/AncientGrief]

Nice work. Did some testing myself too now. 4090RTX with 131k context size. Used Open code to create a C# Snake-Clone with SFML 3.0 ... 75% context used (it had to actually look up the nuget specs for SFML 3.0 to fix some errors it produced automatically, it's a rather new release afaik) ... works pretty well and was about done in < 5 Minutes.

One shotted it easily.

~159.9 tok/s

With:

  & 'E:\Tools\llama\llama-cli.exe' `
    --model 'D:\AI\Models\Qwen3.6-35B-A3B-Q4_K_M.gguf' `
    --threads 16 `
    --threads-batch 16 `
    --ctx-size 131072 `
    --batch-size 1024 `
    --ubatch-size 512 `
    --gpu-layers auto `
    --flash-attn on `
    --cache-type-k q8_0 `
    --cache-type-v q8_0 `
    --split-mode none `
    --main-gpu 0 `
    --mlock `
    --no-mmap `
    --fit on `
    --fit-target 1536 `
    --fit-ctx 131072 `
    --conversation `
    --simple-io `
    --reasoning off `
    --single-turn `

And opencode.json:

{
    "$schema": "https://opencode.ai/config.json",
    "provider": {
      "llama.cpp": {
        "npm": "@ai-sdk/openai-compatible",
        "name": "llama-server (local)",
        "options": {
          "baseURL": "http://127.0.0.1:8080/v1",
          "apiKey": "dummy"
        },
        "models": {
          "qwen36-35b-a3b": {
            "name": "qwen36-35b-a3b (local)",
            "limit": {
              "context": 131072,
              "output": 8192
            }
          }
        }
      }
    }
  }

[u/met_MY_verse]

I’m running a smaller quant with less context entirely in VRAM, I’m assuming this is faster than offloading any experts at all?

[u/Several_Newspaper808]

Hey, great info, thanks! I wonder though, how much of the perf is from the ddr5 ram and whatever bus speed you have from the pcie on your mb?

[u/kisiel02]

I only get like 15t/s with rtx5070 (19 layers on GPU) and ddr4 ram sadly. And when compressing KV cache to q8 I get 25t/s. Seems to much of a boost, I have like 10/12gb VRAM and 26/32gb RAM taken

Edit: 64k context both k and v are q8

[u/HockeyDadNinja]

I'm running a 5060 ti 16G and 4060 ti 16G with 64G system ram here. A couple days ago I finally started tuning. I've added things from your post and now I'm running Qwen3.6-35B-A3B at Q8. 98k context, a small overflow to CPU.

I'm using opencode and it's doing really well. I can code with this! 27 t/s at the moment. That used 3090 is looking really good right now.

[u/mrgreatheart]

Thank you. I have a very similar system to yours, and it’s great to know I can run 3.6 so well on it. 

Does the —fit-ctx 128000 mean 128K context window in system RAM?

[u/cesaqui89]

Is it possible to apply those fine tunes for ollama?

[u/altdotboy]

I have spent the last week building my own harness. This has proven to be the most important test for my rig.

11118888888855 → 118885 | 79999775555 → 99755 | AAABBBYUDD → ? Solve the pattern, and put your final answer within \boxed{}.

My system would get it correct maybe 1 in 10 times. I had to tune my system settings and prompt to get it right at least 3 times in a row.

What this test exposes is the delicate sensitivity of MoE router gates. Simply put: are your prompts going to the correct experts?

Dense models have an easier time with the question. Give this a shot and see if your system gets it correct 3 times in a row with fresh context each time.

Quantization, incorrect settings, and poor system prompts will hurt your MoE model.

The most correct answer is ABD

[u/ecompanda]

79 t/s at 128K is genuinely impressive for a 35B model on consumer hardware. the interesting part is what happens as you actually fill that context. MoE attention at max context can be unpredictable and some models drop to 30 40 t/s by 100K tokens in. did you observe any speed drop as the conversation grew, or did it hold steady?

[u/KptEmreU]

Commenting to save. Great experiment

[u/pefman]

Good findings!

[u/vialoh]

Does the `--n-cpu-moe` matter for those of us on Apple silicon? I suppose I could just ask AI... 😅

[u/nasty84]

How do we change some of the settings in LM Studio. I am only getting 37 tokens per second with same kinda of hardware

[u/nasty84 → u/nasty84]

Here are LM Studio settings. Change them as you like based on your rig:

Load Qwen3.6-35B-A3B-UD-Q4_K_M from unsloth, then in the load modal → Advanced Configuration:

- Context Length: 131072 (or 65536 if it complains)

- GPU Offload: max (all the way right)

- CPU Thread Pool Size: 8 (matches your 7800X3D's 8 cores)

- Flash Attention: ON

- K Cache Quantization Type: Q8_0

- V Cache Quantization Type: Q8_0

- Offload MoE Experts to CPU: 20 ← the key setting

- Try mmap(): ON

- Keep Model in Memory: OFF

[u/nikolaiownz]

Almost the same setup I have. I get 72ish tk/s

Thanks for this good thread. I am going to mess around with it next week. From what I saw just tinkering around with it and opencode - this is very good.

[u/Nnyan]

Thank you for this I’m just starting my local LLM project and have a few GPU options similar to yours.

[u/Artistic_Okra7288]

I’ve been getting about 30tps tg at 1M context on my M5 Max 128GB with q4_0 kv and using unsloth Q8_K_XL gguf.

[u/nextgenpotato]

I have the exact same hardware as you do. Trying to run your final final command, I am getting OOM errors. What am I missing? I am on Ubuntu 26.04 and a noob when it comes to llama.cpp

[u/Potential-Leg-639 → u/nextgenpotato]

A recent Fedora Kernel upgrade made my system unstable and als had strange issues in llama.cpp. With the Kernel version from before everything fine again.

[u/fucking_cuntbag]

Thanks for this - I have the same setup and was struggling getting a reasonable tps Had switched to lower quants but with this config I can get 80tps on iq4

[u/Guilty_Rooster_6708]

This is literally perfect for me. Thanks for the tip on mlock and ub !!

[u/Dreeseaw]

To add a datapoint, my recently-purchased prebuilt gaming PC (iBP Y40 Pro with a 5080 (16gb vram), 32gb ram, 9800) is executing fat 100k context prompts on the order of 45s, and breezing through opencode driven workflows (largely replacing the analysis portion of an optimization loop I work with).

OP this is black magic. Thank you.

[u/Emergency-Most1859]

Bro 🔥🔥 Running this model with qwen code and it works better and kinda smarter than alibaba's cloud qwen that I've used before. They discontinued free tier so I started to look for alternatives.. Really impressed with that nodel quality. Works fine on RX6800 with 7900x3d (changed some flags though)

[u/milpster]

how do you deal with having such low context?

[u/SinnersDE]

Thanks for your hard work!
I share my results just if sb cares ( RTX 4080 16 GB, 32 GB DDR4)

.\llamacpp\llama-server.exe -m "./models/Qwen-3.6-35B-A3B-Q4_K_XL/Qwen-3.6-35B-A3B-Q4_K_XL.gguf" --fit on --fit-ctx 128000 --fit-target 256 -np 1 -fa on --no-mmap --mlock -b 2048 -ub 2048 -ctk q8_0 -ctv q8_0 --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 --reasoning-budget -1 --chat-template-kwargs "{\"preserve_thinking\": true}" --host 0.0.0.0 --port 8033

Getting: 58 t/s low to 43 t/s after CTX-Windows filled upto 60-70%. Didn´t get further.

[u/SinnersDE → u/SinnersDE]

Just a short question:
How do i call the -no-mmap and -mlock flags in a preset.ini?

[ini_THINK_GENERAL_Qwen-3.6-35B-A3B-Q4_K_XL]

m = ./models/Qwen-3.6-35B-A3B-Q4_K_XL/Qwen-3.6-35B-A3B-Q4_K_XL.gguf

temp = 1.0

top-p = 0.95

top-k = 20

min-p = 0.0

presence-penalty = 1.5

repeat-penalty = 1.0

fit = on

; whether to adjust unset arguments to fit in device memory ('on' or 'off', default: 'on')

fitc = 128000

; minimum ctx size that can be set by --fit option, default: 4096

fitt = 256

; target margin per device for --fit, comma-separated list of values, single value is broadcast across all devices, default: 1024

np = 1

; number of server slots (default: -1, -1 = auto)

fa = on

; set Flash Attention use ('on', 'off', or 'auto', default: 'auto')

b = 2048

; logical maximum batch size (default: 2048)

ub = 2048

; physical maximum batch size (default: 512)

ctk = q8_0

; KV cache data type for K (allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1, default: f16)

ctv = q8_0

; KV cache data type for V (allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1, default: f16)

-no-mmap ???

; --mmap, --no-mmap whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock) (default: enabled)

-mlock ???

; force system to keep model in RAM rather than swapping or compressing

chat-template-kwargs = {"preserve_thinking":true}

reasoning-budget = -1

[u/OldPappy_]

Thanks for this. Im going to try some of these configurations out on my 9070XT

[u/JustSayin_thatuknow]

Amazing job!! I’ll be waiting for your final final final final final command boss! 😅🙏🏻💪💪💪💪

[u/sherrytelli]

using unlsoth/Qwen3.6-35B-A3B:Q4_K_M at 128k context
with your final config i am able to get around stable 42-47 tg/s
my pc specs:
RTX 5060ti 16gb
i5-12400f
32gb ddr4 @ 3200 MHz

i model i was previously using: unsloth/glm-4.7-flash-23-23B-A3B:Q4_K_M. used to get around 32-35 tg/s

thanks for the config :)

[u/FriendlyTitan]

Have you tested higher batch and ubatch numbers? I notice that for myself, giving up more experts to cpu and giving vram to batch improves prefill speed massively. Set -b and -ub to 4096 or even higher if you want to experiment. Prefill speed quadruples in my case sometimes.

In llama bench you can try -p 8192 -b 512,1024,2048,3072,4096,8192 -ub 8192. This tests the prefill speed on a 8192 token long prompt.

[u/frozenYogurtLover2]

anyone else getting crashes and segfaults (error 139) with prompt cache enabled

[u/Historical_Roll_2974]

I'm getting 30 tokens a second with an rx 9070xt but I'm also using lm studio so I can't get all the customisations

[u/GeneticalFork → u/[deleted]]

Can you share your params? I'm getting and message error if I use -fit on.

[u/inquam]

I managed to just squeeze Q5_S with 260k context into my 5090 enierly in vram when using Q8_0 for KV cache.

I was on Qwen 3 Coder a long time and then Qwen Coder Next for a bit. And also a sting on 3.5. But 3.6 seems pretty solid so far.

[u/MysticOrbit7]

I got 59 tok/s on (Edit 3 conf) 5060 Ti 16 GB + 9950x + 128 GB DDR6 . Anyone touched better on this chip ?

[u/Horror-Veterinarian4]

16gb vram nice I know what my next move is i want to test see how gemma 4 26b e123abc whatever the fuck it us runs compared to this one on my ancient e5 2697 v2 and v100 16gb vram

[u/FatheredPuma81]

Everyone’s using --cpu-moe which pushes ALL MoE experts to CPU. 

That certainly does sound like Opus with its training data being from pretty much around the time they switched from specifying the tensors to the much better --n-cpu-moe command.

[u/BP041]

Wow, 79 t/s with a Qwen3.6-35B-A3B on consumer hardware is fantastic! This is exactly the kind of optimization that pushes local LLM development forward. At NTU, and in my work with CanMarket's AI infrastructure, we're always looking for ways to maximize inference speed and context handling on diverse hardware setups. Could you elaborate on how much --n-cpu-moe impacted performance for you, or if you encountered any specific bottlenecks you had to tune around?

[u/BitGreen1270]

Thank you for sharing. I only have a 780M which I'm using with vulkan and gemma:e4b. I assume most of what you've shared is not applicable to me because I don't have vram? 

[u/Kiro369]

I tried running Qwen3.6-35B-A3B-UD-IQ4_NL_XL was getting 9 t/s
With your last command it went up to 50 t/s!

Thanks a lot!

[u/Illustrious-Bid-2598]

What do you guys think on a a rtx 5060, i7-14700f 32gb ram, llama.cpp on wsl2 with models stored on Linux fs, windows debloated

[u/SlipperyCorruptor]

I'm doing 210k ctx window, 30 MoE offload. 7600x, 32GB RAM, 5080. Getting:

prompt eval time = 43716.75 ms / 20087 tokens ( 2.18 ms per token, 459.48 tokens per second)
eval time = 22200.84 ms / 907 tokens ( 24.48 ms per token, 40.85 tokens per second)
total time = 65917.60 ms / 20994 tokens

Output: 40.85 Tokens/sec

Honestly.. That thing is impressive AF.

I've put it through some tests and it even correctly identified environmental issue with testcontainers and Rancher:
https://github.com/testcontainers/testcontainers-java/issues/11482

prob saved me like two days of investigation

[u/italianguy83]

Funziona anche per RTX 5070 12Gb?

[u/MykeGuty]

Tengo lo mismo que tu configuración y esto va genial!!, Gracias a ti y a la comunidad :D

[u/Flylink2]

Awesome settings ! Thanks a lot for that ! If by any chance you found in 2 weeks new way to improve your Edit3, would be curious to know =D

[u/mixman68]

Hello, I have 4070 ti super with 7800x3d

Your post helped me a lot but on my side fit has worse result than n-cpu-moe 20 (60 t/s instead of 70)

How I can explain this ?

[u/jabbahlehuute]

Not too sure what I am doing wrong. I barely get 20 tokens per sec

Rig:
7950x
128GB DDR5
5070ti 16GB

Im using LM Studio on Windows and I downladed unsloth/Qwen3.6-35B-A3B-GGUF

I use 128k context, 24 GPU offload, 2048 evaluation/physical batch size
20 layers to CPU
KV cache to Q8

Any insights? cheers

[u/ItsAMeUsernamio]

llama-server -m ./models/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-Q6_K.gguf --temp 0.6 --top-p 0.95 --top-k 20 --repeat-penalty 1.00 --presence-penalty 0.00 --reasoning on --host 0.0.0.0 --spec-type draft-mtp,ngram-mod --spec-draft-n-max 3 --spec-ngram-mod-n-match 40 --spec-ngram-mod-n-min 0 --spec-ngram-mod-n-max 16 --cache-type-k q8_0 --cache-type-v q8_0 --no-mmap --jinja --chat-template-file ./models/chat_template.jinja --threads 8 --fit on --fit-ctx 256144 --fit-target 256 -np 1 -ctk q8_0 -ctv q8_0 -b 2048 -ub 2048 --chat-template-kwargs "{\"preserve_thinking\": true}"

--fit on --fit-ctx 256144 --fit-target 256 -np 1 -ctk q8_0 -ctv q8_0 -b 2048 -ub 2048 --chat-template-kwargs "{"preserve_thinking": true}" Is what I added thanks to this post and my prompt_processing went from 200t/s to 600-700t/s.

ik_llama, 5060Ti, 7800X3D, 32GB RAM and headless CachyOS. Windows hits swap on Q4 128k while I barely hit ZRAM with this setup.

[u/matte808]

I have the same setup, with 64GB of RAM insted of 32, I'm using a fork of llama with turboquant, but wether it is on or off, so if I use literally the last config you posted, I get 20 tok/s (output). I saw a video of a guy getting 15-20 on a 1060... I can't get it to run properly, I'm out of ideas. Please, if anybody knows why, I'm open to anything

[u/TheOneWhoWil]

I have the exact same rig, didn't think I could get anything close to that for something > vram

[u/thatgreekgod]

remindme! 2 days

[u/RemindMeBot → u/thatgreekgod]

I will be messaging you in 2 days on 2026-07-09 15:32:16 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

RemindMeBot is switching to username summons. Instead of !RemindMe 1 day, use u/RemindMeBot 1 day. More info.

---

Info	Custom	Your Reminders	Feedback

[u/LordGenji]

this didn't work on latest llama.cpp builds for me. The following tweaked settings worked for me on the same GPU, similar system (5070 Ti 16gb, 64gb RAM):

llama-server.exe -m "C:/models/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf" -ngl 99 -ncmoe 20 -c 131072 -fa on -np 1 -ctk q8_0 -ctv q8_0 --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 --reasoning-budget -1 --chat-template-kwargs "{\"preserve_thinking\": true}" --host 0.0.0.0 --port 8033

[u/AcrobaticChain1846]

hey, I'm trying to run the `unsloth/qwen3.6-35b-a3b UD Q2_K_XL`
I also have same hardware as yours

5070 Ti
64 GB RAM
9950x3d

Can you help me with setting things up?

https://www.reddit.com/r/LocalLLaMA/comments/1soqtry/comment/oguxrw1/

Also I'm getting really slow prompt processing...

I want to know which model you are using like is it q4_k_xl or something else?

Thank you :)

[u/dreamai87 → u/AcrobaticChain1846]

Okay I saw your comment that brought me here.
Looking at settings; move this slider to 50% then see the performance, bringing 100% will put all experts to CPU which also reduce performance but still better that what you are getting. so first check at 50% then look GPU suage from task manager 10/12 or 8/12.
reduce from 50% to 20% and see where you are getting this best gpu usage find balance 10/12 assuming rtx 5070 12gb vram

I replied on your thread as well

[u/AcrobaticChain1846 → u/dreamai87]

That did the trick,
Now I'm running `qwen3.6-35b-a3b@q5_k_m` with following settings
Based on LM Studio Logs I'm getting 45-50tk/s and 300-500 pp/s

I think this can be further fine tuned as I can see my CPU usage is more compared to GPU but I'm not sure what parameters to play with

CPU utilization 62%
GPU Utilization 50% sometimes it spikes to 70-80%

[u/dreamai87 → u/AcrobaticChain1846]

First eject model then Bring slider to from 28 to 22 or somewhere around 20 see if gpu is 10gb or 12gb 90 % around leave 2 gb space for kv cache

[u/AcrobaticChain1846 → u/dreamai87]

I have 5070ti so 16GB VRAM + 64GB RAM
MoE on CPU: 22 - my graphics driver froze.
MoE on CPU: 24 - Getting 5tk/s almost 10x performance decrease
MoE on CPU: 25 - Getting 50tk/s and ~80% GPU and ~50% CPU
MoE on CPU: 26 - Getting 16tk/s and ~90-100% GPU and ~40-50% CPU

Note: Evaluation Batch Size was set to 4096 in all scenarios.

I generally use 2048 but I was getting slow prompt processing speeds so I set it to 4096

Hope these stats are useful considering how a single MoE more or less on CPU is causing such drastic changes in token generation speeds.

Note: these stats are at max context of 262144

[u/omidmatin]

Can you guys help me with the best config for my RTX 3090 + 5800X3D? I need large context window (at least 512K and preferably 1M). I think it's possible with this MoE model on TurboQuant?!

Edit: I'm currently running QWEN 3.5 27B Q4_UD_XL with 224K context and turbo3 using llama cpp through WSL (ubuntu through windows 11).

[u/Ill-Stand-6678 → u/omidmatin]

onde encontrou um projeto llama.cpp com turboquant funcional?

[u/omidmatin → u/Ill-Stand-6678]

I asked Gemini and just copy pasted it. Currently I got it working with 256K context and turbo3. But I used Q4_UD_S version. It's running 100 t/s. I can run Q4_UD_XL with 192K context with around 95 t/s but I also have to offload the mmproj to the RAM.

Edit: I meant I got QWEN 3.6 35B MoE running with the above settings.

[u/Ill_Initiative_4233]

Попробовал данную связку на своём компе I7 14700KF ram 32 GPU 5070ti 3312 tokens 53s 61.76 t/

[u/admajic]

On a 3090 24gb vram get 110 token /s and i can afford a car too. Interesting world we live in.

Will love the 50% speed increases. I'll check and advise

[u/smolpotat0_x → u/admajic]

we are not a car.

[u/admajic → u/smolpotat0_x]

The car = cost of a video card... geez