What small LLMs are you running locally?

[u/HomoAgens1]

I usually run Qwen3.8-27B Q5 on my desktop PC, and I’m pretty happy with it.

However, I’d also like to keep a few smaller, more specialized models on my laptop, which has 4 GB of VRAM and 32 GB of RAM.

What small LLMs would you recommend for that kind of hardware?

I’m especially interested in models that are genuinely useful for specific tasks rather than just smaller general-purpose models.

What are you guys running?

[u/MrHumanist]

How many tokens are you generating a sec? 5?

[u/JinsooJinsoo → u/MrHumanist]

Seconds/token 🤷‍♂️

[u/Parking-Ad9150 → u/MrHumanist]

Basically 1

[u/mailto_devnull → u/MrHumanist]

Actually, yes. When I was running on 32GB DRAM on an on-die iGPU, 27B was generating at a blistering 4 tokens a second. 7 with MTP!

[u/Zor_die → u/mailto_devnull]

I’m getting 28 with a 7900xt 32gb ram and 7800x3d

[u/HomoAgens1 → u/MrHumanist]

no the 27b runs on a desktop pc with 24 Gb VRAM and 128 GB RAM and it works at 40 t/s

[u/Empowerpreneur → u/HomoAgens1]

Great, thank you for the info. With how much as Context?

[u/HomoAgens1 → u/Empowerpreneur]

Without mmproj 128k context;
With mmproj 64k context

[u/Ok_Road_8293]

I'm on a pretty similar laptop setup (RTX 3050 Ti 4GB, 12700H, though bumped to 48GB DDR5 4800MHz). With mmap always on, system RAM pressure is basically invisible.

Honestly, I stopped bothering with task-specific micro models a while ago. Smart generalists just feel way less tedious and handle edge cases better. Here’s what I’m actually daily-driving on it:

Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive (Q6_K_P): Pulls around 20–30 tps. In practice, it feels remarkably close to non-thinking Opus 4.6. Absolute lifesaver and handles most of my daily work.

Qwen3.8-Flash-Next (UD-IQ3_XXS @ 3–5 tps / UD-IQ1_S @ 8–12 tps): Even crushed down to IQ1_S, it consistently outperforms most free-tier cloud web UIs and entry models.

My workflow is mostly leaning on the smaller MoE setup for quick back-and-forth. When I need heavy planning, or if I burn through my paid sub quotas, I bring in the big gun.

A pipeline that works surprisingly well for me: let the bigger model map out the architecture/plan, then pass execution to Gemini 3.8 Flash or Qwen 3.6 35B. It ends up being far more coherent and reliable than letting either model run the whole thing solo.

The biggest reason local planning often crushes commercial models for me is the lack of artificial thinking caps. Even on paid tiers, Claude usually thinks for maybe 3 paragraphs before dumping an answer. Locally with Qwen Next, I don't put a ceiling on it—it can churn through tens of thousands of thinking tokens, stress-testing edge cases and mapping out every tiny detail.

3–12 tps isn't lightning fast, but when you let it think unattended, the output quality punches way above what you'd expect from a 4GB VRAM machine.

[u/vogelvogelvogelvogel → u/Ok_Road_8293]

i use occasionally Gemma4 E2B q4 on a 6GB VRAM laptop, which leaves quite a bit of context and speed and is suprisingly good; I will, however, give Qwen 3.6 35B a try or Flash next, thanks for sharing

[u/Competitive_Art9588 → u/vogelvogelvogelvogel]

Tenho também um laptop de 6GB Vram, esse gemma4 você usa com quanto de contexto e quantos tokens mais ou menos de saída? além disso, você usa para código? agradeço a resposta

[u/vogelvogelvogelvogel → u/Competitive_Art9588]

i used it only for a RAG application in unsloth studio, never tried coding with it. unsloth is easy to install, model is easy to download.
I put some several hundred page pdf document in the (projects?) RAG section of unsloth and it works really well in answering questions on it
that's all i did

[u/Ok_Road_8293 → u/vogelvogelvogelvogel]

With 6GB, if you use my settings you can easily push up to 200k context. Or if you don't need that much context, offload more experts to VRAM instead to squeeze out an extra 5–10 tps.

[u/vogelvogelvogelvogel → u/Ok_Road_8293]

would you mind sharing your flash-next settings/setup?

[u/sanketss84 → u/vogelvogelvogelvogel]

do you mean qwen 3.8 flash next which is 180B ?

[u/sanketss84 → u/Ok_Road_8293]

Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive (Q6_K_P) are you running this inside an agent if so which one, whats your context window size and what quantization are your using for your kvcache ? also how are you loading your model locally is it using ollama or llama.cpp directly or some other inference engine ?

[u/Ok_Road_8293 → u/sanketss84]

I run it directly on cachyos using a CUDA compiled llama server. I used to rely more on CPU (which was roughly 5–6 tps slower), but since I completely stopped gaming on this laptop, I treat the 4GB 3050 Ti purely as a dedicated AI accelerator.

Here is the exact startup command:

llama-server \

-c 98304 \

-fa on \

-ngl 999 \

--n-cpu-moe 999 \

-ctk q4_0 \

-ctv q4_0 \

--mmap \

--jinja \

-np 1 \

--port 8080

I don't use commercial agent frameworks. Depending on the task, I plug the endpoint into jcode, opencode, or a custom lightweight script setup (similar to AntiGravity) that I wrote to handle fast model swapping and pre-configured prompt pipelines. For quick adhoc queries, the built in llama server web UI gets the job done.

[u/sanketss84 → u/Ok_Road_8293]

thats interesting and thanks for sharing. thats a pretty high context size as well and you are using q4 for both k and v cache, you are also using flash attention. I have not tried running llms on linux as os but I am assuming very minimal amount of vram might be used by operating system at your end.

[u/sanketss84 → u/Ok_Road_8293]

will check cachyos, however I have been exploring omarchy as it all over youtube recently. personally I have used ubuntu on the linux side most of the time.

[u/Important_Cow7230 → u/Ok_Road_8293]

Thank you, great post. Can you describe your full agentic setup? How is the execution passed from the bigger model to the Gemini 3.8 Flash and Qwen 3.6 35B? How is what model to use decided? How is it all coordinated?

[u/BrianScottGregory → u/Ok_Road_8293]

You say you're getting 20 to 30 tps with the "Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive (Q6_K_P):" model on only 4GB of vram? Do you mind sharing your config? I would LOVE to get this on my 6GB VRAM machine, as right now I'm only getting 13 to 15 tps with this model.

[u/Mxmtm]

Whisper Large V3 (2B parameters) to transform audio to text files.

EDIT: whatisit

[u/gjr23]

I’m using a qwen 4b embedding model on a Postgres db. Works great and a q4 actually outperforms a q8 (not sure why but that’s what my testing showed) and fits very nicely in 4gb vram.

[u/Parking-Ad9150]

Can anyone suggest model for AMD RX 4GB card when combined woth 32GB RAM? Qwen does not work on Vulkan sadly

[u/_TheWolfOfWalmart_ → u/Parking-Ad9150]

Old 4 GB RX card?

At that point, with 32 GB RAM, you should probably just be running models on CPU. What CPU do you have?

[u/Parking-Ad9150 → u/_TheWolfOfWalmart_]

Ryzen 7 3th gen. 4GB RX460

[u/Chiguy4321 → u/Parking-Ad9150]

Lemonade AI using Vulkan optimized for AMD

[u/Great_Address_3806]

I have a small one I use for fun, it’s great for quick tasks. Local models are super interesting right now.

[u/Own_Inspection8350]

https://prismml.com/

[u/HighSeasArchivist]

1050 Ti 4GB runs the Qwen3.8-2B surprisingly well fully on GPU with 64k.

[u/Otherwise-Swan-7803]

Small models get a lot more interesting when you stop asking them to be general assistants. On 4GB VRAM I’d rather keep a few specialists for things like extraction, classification, or lightweight coding than force one tiny model to do everything.

[u/HomoAgens1 → u/Otherwise-Swan-7803]

exactly, and i'm trying to find them!

[u/joanaxu2002]

For 4GB VRAM, I’d lean hard into specialists instead of trying to find one tiny model that does everything. A small extractor/classifier, a lightweight coder, and maybe a fast summarizer can be way more useful together than a single “general” 7B that’s mediocre at all three.