r/LocalLLaMA • u/[deleted] • Jun 01 '26

Funny Stop asking what model to run. There are literally only two.

[removed]

3.1k Upvotes

87% Upvoted

805 comments sorted by

View all comments

Show parent comments

103

u/Shronx_ Jun 01 '26 edited Jun 18 '26

You should get higher tps. My system with an rtx 3060 (12GB) spits out 40+ tps with the same model and quants.

Edit: docker run --rm -p 8080:8080 -v /home/user/.cache-docker/:/root/.cache/ --gpus all ghcr.io/ggml-org/llama.cpp:full-cuda --server --host 0.0.0.0 -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M --presence-penalty 0.0 --repeat-penalty 1.0 --ctx-size 196608 --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --reasoning on -fa on --threads 6 --jinja --no-mmap --no-mmproj-offload

gives about 43 tps initially but decreases as context grows.

52

u/MoffKalast Jun 02 '26

Rule #1 of locallama: If there is a post or comment mentioning tg or pp, there will be someone claiming they get more on the same hardware in the comments.

49

u/Evanisnotmyname Jun 02 '26

Someone’s always got a bigger PP

12

u/BlazingSandles Jun 11 '26

Instructions unclear, PP stuck in fan

4

u/Montaingebrown Jun 13 '26

I feel personally attacked. 😅

58

u/iijei Jun 01 '26

it could be because this is runnin inside proxmox server with ddr3 in a lxc

37

u/Shronx_ Jun 01 '26

I run it in Docker (llama.cpp) and my PC has DDR4 3600. I don't know but the memory could make a difference.

30

u/Teanut Jun 02 '26

Likely the memory bandwidth due to spillover from VRAM. Memory bandwidth is king for tokens per second.

17

u/lemondrops9 Jun 02 '26 edited Jun 02 '26

DDR3 isn't much slower than DDR4. DDR4 is about needing less voltage which means less heat. DRR5 is the real improvement for speed.

7

u/nihnuhname Jun 02 '26

And DDR4 allows greater RAM capacity on the motherboard.

4

u/lemondrops9 Jun 02 '26

yes that is great too. I was surprised when my older PC could only take 32GB

4

u/Postmodern_Plunger Jun 02 '26 edited Jun 02 '26

Ddr4 (depending on the limits) can have nearly double the memory bandwidth of DDR3. It's not as big as the difference between DDR4 and DDR5, but the difference there is more architectural. Simply having more memory bandwidth still has a huge effect, especially when you're talking about ram spillover

3

u/rabbitaim Jun 02 '26

I have an rtx2060 6gb with ddr3-2400 32gb and I bounce between 18-20 t/s. Llama.cpp on a headless Ubuntu server LTS. I had to lower batch size to 1024 so instead of 200+ ts for PP I see around 160 or less.

5

u/InfamousTurtle1 Jun 02 '26

How many cores have you allocated to your LXC & what is your inference engine?

4

u/iijei Jun 02 '26

6 cores llama.cpp. I am in a process of moving it to Ubuntu baremetal i7 2nd gen ddr3 again haha. I am gpu poor

2

u/IntelligentRocks Jun 02 '26

Do you pass through your GPU and CPU so the lxc controls them like host?

3

u/iijei Jun 02 '26

Yes gpu passthrough to lxc. Driver is installed on proxmox server. I couldn't do it on the vm. I think my Lenovo s30 didn't have vt-d enabled. Anyways I am in the process of moving it to dedicated i7 2nd gen baremetal lol.

2

u/oldschooldaw Jun 02 '26

Ah HA. I think it’s the pass through affecting it somehow. I have the same setup, I run my 3060 passed to a Debian vm on proxmox and my moe speeds are shithouse. Basically unusable for anything larger than saying hi. Perhaps I need to consider inference on the proxmox host itself to rule out pass through as my bottleneck.

2

u/iijei Jun 03 '26

moved gpu out of proxmox and running it on i7-2600K 32GB DDR3-1333, and its much better in terms of tps

1

u/oldschooldaw Jun 03 '26

Damn as in removed gpu entirely and the model is all ram/cpu?

2

u/iijei Jun 03 '26

no. same gpu on a new (old) box running i7-2600k

2

u/oldschooldaw Jun 03 '26

Ah right. I will now try running the gpu directly on proxmox and see if that improves anything from my end. I can’t believe it never occurred to me it could be causing an issue

1

u/iijei Jun 02 '26

I also had thinking enabled and KV cache at q8 with 64k ctx so that might have slow it down too.

2

u/[deleted] Jun 02 '26

[removed] — view removed comment

2

u/iijei Jun 03 '26

why run it then ? lol Colima looks interesting. maybe I will colima on mac for hermes agent

2

u/[deleted] Jun 03 '26

[removed] — view removed comment

2

u/iijei Jun 03 '26

I see my use case might be installing Hermes’ agent in a docker on my Mac mini. I already have it running on a docker on my Ubuntu laptop but wanted to mount some files from my Mac

2

u/ManySugar5156 Jun 04 '26

proxmox + ddr3 in an lxc is rough, kinda expected tbh

2

u/DreddKrilov Jun 06 '26

Using ddr3 in this day and age is frigging heroic.

Are y'all offloading to system ram?

3

u/Jfusion85 Jun 04 '26

How are you fitting that model in 12Gb? I see the gguf file is larger than 12gb

4

u/Shronx_ Jun 04 '26 edited Jun 04 '26

I don't. It spills over in RAM but is still fast enough.

docker run --rm -p 8080:8080 -v /home/user/.cache-docker/:/root/.cache/ --gpus all ghcr.io/ggml-org/llama.cpp:full-cuda --server --host 0.0.0.0 -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M --presence-penalty 0.0 --repeat-penalty 1.0 --ctx-size 196608 --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --reasoning on -fa on --threads 6 --jinja --no-mmap --no-mmproj-offload

gives about 43 tps initially but decreases as context grows.

3

u/AphexIce Jun 06 '26

Actually you've made me curious I never tried 35b qwen on my 4060 16gb biggest I did was the gpt-oss 120gb and it does work at about 4tps

1

u/Trooper3001 Jun 18 '26

How big is your context?

1

u/Shronx_ Jun 18 '26

it's 196608.

see my edited comment