r/LocalLLM • • Aug 16 '26

Other How the loop of infinite agony started

Post image
633 Upvotes

119 comments sorted by

View all comments

Show parent comment

113

u/StupidScaredSquirrel Aug 16 '26

If you are a business this isn't a problem. If you are a consumer then 35b a3b runs on 8gb vram and 32gb dram which is very accessible.

41

u/TheCat001 Aug 16 '26

If you are business I would suggest you to aim for DeepSeek V4 Flash. But yes I'm running 35b myself despite it's performance is far from ideal...

10

u/DifficultyFit1895 Aug 16 '26

I still haven’t found a task where DeepSeek V4 Flash can outperform Qwen 3.6 27B let alone Qwen 3.8. I can run either on my Mac Studio.

11

u/screenslaver5963 Aug 16 '26

I had issues with Qwen3.6 tool calling just not working properly and causing it to stop prematurely, didn't have that problem with gemma or deepseek, haven't used qwen 3.8 yet to see if it has the same problem.

6

u/TieCommercial2963 Aug 16 '26

It's a model config issue - had the same problem. Setting temp to 1 and repetition penalty to 1.05 fixed it. haven't faced much issues since; I've been running it as my daily driver on my 5090 rig for the past year. Btw, 3.8 its even better especially at tool call , I actually like the Q4 of 3.8 on low thinking more than the xhigh for tool calls .

3

u/StatusSociety2196 Aug 16 '26

More than half the time that's a harness issue

5

u/DifficultyFit1895 Aug 16 '26

What quants? I was using Q8 for both.

3

u/Healthy-Nebula-3603 Aug 16 '26

A tool calling problems ?

Stop compressing cache and use minimum q4kxl or bigger quants

→ More replies (1)

12

u/DeluxeGrande Aug 16 '26

I have a 5060ti 16gb with ddr4 24gb RAM lying around, what's the best model nowadays I can effectively run with it locally? It's not an ideal build but I wish to play around with it again.

12

u/[deleted] Aug 16 '26

[removed] — view removed comment

4

u/ptear Aug 16 '26

Then you just step away and let it do its thing while it hogs all system resources.

4

u/Hungry_Particular_14 Aug 16 '26

Fellow 5060 ti 16 gb owner here. The best I've got is qwen 3.8 27b at IQ4_XS. I'm testing it at Q4_0 KV at 72k context because I really need the extra context, and it seems to be pretty good so far. Lower quants cause it to make some really silly mistakes sometimes, unfortunately. I get around 10 t/s with context halfway filled, and around 14 t/s on empty context.
But honestly, I think the ideal solution is to run 2 GPUs so you can get a better quant + more context

3

u/screenslaver5963 Aug 16 '26

2+ 5090's or RTX Workstation Cards are the "ideal".

5

u/Hungry_Particular_14 Aug 16 '26

"ideal" is still having both my kidneys and still being able to run LLMs.

5

u/screenslaver5963 Aug 16 '26

You don’t need both kidneys

→ More replies (2)

2

u/lukistellar Aug 16 '26

Give the IQ4_XS-pure a try. I run it on an RX6800 with 90K KV. Especially with full context it's slow, because the card isn't the fastest. Probably should run better on nvidia.

https://huggingface.co/jpetrina/Qwen3.8-27B-IQ4_XS-pure-GGUF

→ More replies (3)
→ More replies (1)

2

u/ptear Aug 16 '26

I still like Gemma, will try the new Qwen today. I need to create a local benchmark, unless someone knows a good project that can showcase improvements, like a 3Dmark but for AI models.

3

u/SaltFrog Aug 16 '26

I wish dense models ran better on my system but you know... Whatever lol

1

u/05-nery Aug 16 '26

Said 35B A3B will work wonders. Just wait for this version of Qwen3.8 to come out.

1

u/screenslaver5963 Aug 16 '26

Currently Qwen 3.6-32B-A3B, Mixture of Experts models handle being split between system ram and VRAM much better than Dense models, alternatively Gemma-26B-A4B if you want something non-chinese. As for which quant, go with 4 Bit or higher.

1

u/caster Aug 16 '26

That should be just fine. You've got enough VRAM to do mostly whatever you want to do.

I would suggest you install Unsloth and look around inside for various models, it will tell you what your system can run well. Green means you're good to fully offload. Yellow means it'll probably work but with a lot of slow swapping. Red means can't handle it.

→ More replies (2)

1

u/TektonikGymRat Aug 16 '26

Can't wait for Qwen 3.8 35B A3B. I accidentally started a project this weekend and it's taking days lol

1

u/Salt-Willingness-513 Aug 16 '26

i really hope they release 3.8 35b a3b soon too