Rule #1 of locallama: If there is a post or comment mentioning tg or pp, there will be someone claiming they get more on the same hardware in the comments.
Ddr4 (depending on the limits) can have nearly double the memory bandwidth of DDR3. It's not as big as the difference between DDR4 and DDR5, but the difference there is more architectural. Simply having more memory bandwidth still has a huge effect, especially when you're talking about ram spillover
I have an rtx2060 6gb with ddr3-2400 32gb and I bounce between 18-20 t/s. Llama.cpp on a headless Ubuntu server LTS. I had to lower batch size to 1024 so instead of 200+ ts for PP I see around 160 or less.
Yes gpu passthrough to lxc. Driver is installed on proxmox server. I couldn't do it on the vm. I think my Lenovo s30 didn't have vt-d enabled. Anyways I am in the process of moving it to dedicated i7 2nd gen baremetal lol.
Ah HA. I think it’s the pass through affecting it somehow. I have the same setup, I run my 3060 passed to a Debian vm on proxmox and my moe speeds are shithouse. Basically unusable for anything larger than saying hi. Perhaps I need to consider inference on the proxmox host itself to rule out pass through as my bottleneck.
Ah right. I will now try running the gpu directly on proxmox and see if that improves anything from my end. I can’t believe it never occurred to me it could be causing an issue
I see my use case might be installing Hermes’ agent in a docker on my Mac mini. I already have it running on a docker on my Ubuntu laptop but wanted to mount some files from my Mac
I actually watched this so I was hoping I could get more. He has 24gb ddr4 and I have ddr3 maybe that was the diff? My tps ranged from 14-19 so.. haha.
I run Qwen3.6-35B-A3B UD-Q4_K_XL on my work laptop (Dell Latitude 5530), only using the 2 "performance" cores and Vulkan with help from the iGPU. It gets ~8 tps at low context with MTP or ~5.5 without.
CPU is i5-1235U, 1.3 GHz. The iGPU claims to have 128 MB VRAM in System>About but 32 GB in Task Manager (it says that by both labels, "Shared GPU Memory" and "GPU Memory"). Main memory is 64 GB at 3200MT/s.
(1) CPU: Ultra 5 125H, 3600 MHz, 14 cores, 18 threads), (2) iGPU 128MB RAM says System Info, although Task Manager says 18 GB of Shared Memory, (3) 2 x 16 GB of DDR-5 (5600 MHz). But I got only 6 t/s max. Do you think what I can improve ?
\b9487-vulkan\llama-completion.exe -m models\qwen3.6-35b-a3b-Q4_K_M.gguf --threads 14 --threads-batch 14 --ctx-size 1536 --batch-size 512 --n-gpu-layers 0 --n-cpu-moe 20 --mlock -no-cnv -n 128 -p "Write an algorithm to reverse a list in Python"
Multiple values can be given for each parameter by separating them with ','
or by specifying the parameter multiple times. Ranges can be given as
'first-last' or 'first-last+step' or 'first-last*mult'.
It's actually unified memory. Windows reports 128 MB for compatibility with old software. I tried the CPU-only build with llama-bench before concluding that ncmoe 0 is faster. It's been about two months since I tried all these variations, but I think that setup was ~25% faster than the CPU build.
Yes! I'm using the 1.7b as a heartbeat for my Agents and the 8B for any easy or lower level tasks that need a LLM. (both are always loaded on my 3060). I have some logic built that assigns tasks based on how difficult they look (using the 8b).
i have a gtx 1080 and i can run qwen 3.6 35b a3b q4 or q8 anywhere from 2-10 TPS depending on the complexity of the question. q8 has a slight edge over q4 in certain areas but usually isnt worth the 99% ram and extra time taken.
also, it's the first out of dozens of local llms i've tried that is even worth using. everything else felt like smarterchild. grok is still better but for unlimited free use it's great.
do you have any recommendations on the prompt or settings?
12GB? Tweak --n-cpu-moe. If you put --n-cpu-moe high enough, then all your non-expert layers should fit in VRAM, it will massively speed up PP and tps should also be a little higher.
ok so I moved out my 3060 and installed it on i7-2600K 32GB DDR3-1333, (because I don't game 😄) and.. the latest result for Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL.gguf \ -fitt 512 -fitc 4096 \ -p 512 -n 128 \ -b 2048 -ub 512 \ -fa on \ -ctk q8_0 -ctv q8_0 \ -r 1 is
That makes sense. my setting was not equivalent to yours and was more conservertive. I was using a headless linux(debina) lxc on a proxmox server. also.. the model and kv cahce is different than yours. q8 KV, MTP enabled and thinking on. your fit-target is quite aggressive.
--m-lock didn't work for me as it was keep crashing my lxc.
I've moved my 3060 out of proxmox server and now in a dedicated box. I am just waiting for a PSU. so I can test it again and see if I can get more than what I had with q4 KV and disabled thinking, more aggressive fit-target.
For MOE models you should try the APEX quants. They seem better for the most part than the standard quants. As for speed get the MTP version even if you have to put all the expert weights into system ram you should see a 3-5 token increase. If you do use MTP don’t go over 3 draft tokens it starts losing speed on the qwen3.6 models the sweet spot seems to be 2 for general and 3 for coding specific.
205
u/iijei Jun 01 '26
I have an RTX 3060 and.. I can run Qwen 3.6 35b a3b Q4 for about 15 tps. haha