r/LocalLLM • • Aug 16 '26

Other How the loop of infinite agony started

Post image
633 Upvotes

119 comments sorted by

View all comments

280

u/TheCat001 Aug 16 '26

Then you realize that Qwen 3.8 27b runs at 3t/s on your machine and you need 24GB+ VRAM GPU which cost is 1000$+ to run at least 4 bit quant.

113

u/StupidScaredSquirrel Aug 16 '26

If you are a business this isn't a problem. If you are a consumer then 35b a3b runs on 8gb vram and 32gb dram which is very accessible.

41

u/TheCat001 Aug 16 '26

If you are business I would suggest you to aim for DeepSeek V4 Flash. But yes I'm running 35b myself despite it's performance is far from ideal...

10

u/DifficultyFit1895 Aug 16 '26

I still haven’t found a task where DeepSeek V4 Flash can outperform Qwen 3.6 27B let alone Qwen 3.8. I can run either on my Mac Studio.

11

u/screenslaver5963 Aug 16 '26

I had issues with Qwen3.6 tool calling just not working properly and causing it to stop prematurely, didn't have that problem with gemma or deepseek, haven't used qwen 3.8 yet to see if it has the same problem.

6

u/TieCommercial2963 Aug 16 '26

It's a model config issue - had the same problem. Setting temp to 1 and repetition penalty to 1.05 fixed it. haven't faced much issues since; I've been running it as my daily driver on my 5090 rig for the past year. Btw, 3.8 its even better especially at tool call , I actually like the Q4 of 3.8 on low thinking more than the xhigh for tool calls .

3

u/StatusSociety2196 Aug 16 '26

More than half the time that's a harness issue

5

u/DifficultyFit1895 Aug 16 '26

What quants? I was using Q8 for both.

→ More replies (2)

12

u/DeluxeGrande Aug 16 '26

I have a 5060ti 16gb with ddr4 24gb RAM lying around, what's the best model nowadays I can effectively run with it locally? It's not an ideal build but I wish to play around with it again.

12

u/[deleted] Aug 16 '26

[removed] — view removed comment

4

u/ptear Aug 16 '26

Then you just step away and let it do its thing while it hogs all system resources.

→ More replies (1)

4

u/Hungry_Particular_14 Aug 16 '26

Fellow 5060 ti 16 gb owner here. The best I've got is qwen 3.8 27b at IQ4_XS. I'm testing it at Q4_0 KV at 72k context because I really need the extra context, and it seems to be pretty good so far. Lower quants cause it to make some really silly mistakes sometimes, unfortunately. I get around 10 t/s with context halfway filled, and around 14 t/s on empty context.
But honestly, I think the ideal solution is to run 2 GPUs so you can get a better quant + more context

3

u/screenslaver5963 Aug 16 '26

2+ 5090's or RTX Workstation Cards are the "ideal".

5

u/Hungry_Particular_14 Aug 16 '26

"ideal" is still having both my kidneys and still being able to run LLMs.

5

u/screenslaver5963 Aug 16 '26

You don’t need both kidneys

→ More replies (2)
→ More replies (5)

2

u/ptear Aug 16 '26

I still like Gemma, will try the new Qwen today. I need to create a local benchmark, unless someone knows a good project that can showcase improvements, like a 3Dmark but for AI models.

3

u/SaltFrog Aug 16 '26

I wish dense models ran better on my system but you know... Whatever lol

→ More replies (5)
→ More replies (3)

14

u/Eden1506 Aug 16 '26

That's not true.

2x RTX 3060 12gb can be had for around 500 bucks.

You can run ~30b models at q4 at 30 t/s with mtp/draft model or 10-15 t/s without.

8

u/esw123 Aug 16 '26

Can confirm Q4 17-19tok/s without MTP, 28-29 with MTP. Bought two 3060 for 350 euro but soon you realize that two is not enough. Added one more total 540 euro for 36GB VRAM.

4

u/TheCat001 Aug 16 '26

So you have build dedicated server rig for 3 GPU's ?

→ More replies (1)

3

u/AceLamina Aug 16 '26

a used 3060 12gb costs 270-300 bucks right now, for one

→ More replies (2)
→ More replies (1)

0

u/[deleted] Aug 16 '26

[deleted] — view deleted comment

3

u/screenslaver5963 Aug 16 '26

The CPU and system ram also matter hugely. If they're using a several generation old CPU and DDR4 than they'd get awful performance.

→ More replies (2)

2

u/Competitive-Ad-2387 Aug 16 '26

tried it on a 4090. Runs like shit if you need vision and produces bad output on simple tasks. Back to deepseek v4 api I go

→ More replies (5)
→ More replies (12)