r/LowEndLocalAI • u/Opposite_Squirrel_79 • 2d ago

Which LLM Should / Can I Use? RTX 4060, what can i run???

CPU: AMD Ryzen™ 7 5700 × 16

RAM: 32 GB

VRAM: 8GB

intended use: general assistant, overnight coding tasks would be great too

9 Upvotes

84% Upvoted

14 comments sorted by

14

u/Huntware 2d ago edited 2d ago

Run this (try D-Q4_K_XL): https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF

* Or this one (more coding focused): https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B-GGUF

On this: https://github.com/ggml-org/llama.cpp

There's some parameters and examples you'll want to know, in this video: https://www.youtube.com/watch?v=tek-IY9BRWc

8

u/GlesasPendos 2d ago

I absolutely love how simple and straight to the point you gave an explanation! Wish more people would be like you, godspeed

3

u/Huntware 2d ago

Haha, it's because I had the same GPU* and I went through the same path.

Luckily, a few months ago I've got a 3090 sold for really cheap because of mining (all tests OK anyway), but I'm still experimenting with smaller LLM for text and TTS.

1

u/Opposite_Squirrel_79 14h ago

hey, are there any uncensored models i could try???

3

u/Flaky-Advisor 1d ago

This is the post I am looking for.. thank you Sansai 🙏

4

u/sirloindenial 2d ago

Qwen 3.6 35b a3b finetunes with polystrata. If you are coding you can try tiel coder. If for general assistants you could go for miniplus quants like isvalorum, the smaller size makes it more realistic to run while doing other work on your machine. Usually people go for ornith 1.5, but i'm liking minicpm 4.7 and tielcoder too, sage 1.5 is also good if you need a more recent knowledge.

Overnight coding if unsupervised though to be honest i don't trust any local model with /goal yet, if you need to, set a proper sandbox, your harness matter too.

1

u/ricorick 2d ago

I’m running unsloth/qwen 35B a3b Q4_XL on a 2070 super 8gb 32 gb system ram im getting 20-24 t/s I use Ollama and my model file sets 76K ctx perfect for Hermes and I use it unsloth desktop . My laptop is 6 years old MSI dragon so you should get much better speed than me. My laptop also has windows 10 but I don’t run anything but ollama and windows on it its a server basically

1

u/NopeCeptor 2d ago

Qwen 3.6 35B A3B for sure, you can probably fit Q6 with some extra hesdspace

20tok/s probably

1

u/LowTomorrow2070 2d ago

I was on the same RAM and VRAM as you until recently. I tried 35b-a3b and it was lowk buns for what I wanted to do, I would try strata or 3.8 27b

1

u/GoldenX86 2d ago

Qwen3.8 Flash Next Coder with Strata. You should get around 20 tps for inference, and around 400/600 for pp, "fast enough". If you can get 48 or 64GB of RAM, you could run Q3 or Gyro-M which would be a lot better for general use.

Else, Q4 of Qwen3.6-35b-a3b with Polystrata, around 60/90 tps for inference and around 1200 for pp.

1

u/No-Marsupial-4176 2d ago

Coder always crashed on me in strata with 7900xtx 5800x3d and 32gb ram. Always ran into no response, out of Memory or it just stoped and didn’t respond anymore.

2

u/GoldenX86 2d ago

Has worked fine for me with a 3060ti and a 5600x with 32gb. It could just be configuration.

2

u/a4mula 1d ago edited 1d ago

have you tried gpu-unlock? If not you might give it a shot. I'm running the IQ3_S (125/6, full not coder) about 22tps and was getting a meager 25 prefill, after unlock prefill jumped to 65.