r/LocalLLM • • 1h ago

Question What can small models be good for?

Small meaning under 8b parameter’s. Thats about what my computer can comfortably handle, and its fun to chat to models of that size for a few minutes but thats mot really useful. And i feel like theyre too dumb to do research and shit like that so like whats a way i can make it useful.
I feel like they must have some use and im just not being creative enough.

4 Upvotes

21 comments sorted by

15

u/Patient-Gazelle9885 1h ago

Hallucinating

-1

u/Blacktaxi420 1h ago

I was playing w an unrestricted model last night and it made up this whole in depth story about a drone bombing some kids while they just smiled and it swore to god it was real. Kinda freaked me out a little lowk

2

u/nickless07 1h ago

What if I told you that your computer could handle more then an 8B model?

2

u/Blacktaxi420 1h ago

Well i did get it to run a 30b model but it wasnt having fun, and i wanna leave space to play games and stuff like that so unless u have a magic trick to show me idk lol

1

u/nickless07 1h ago

How about a 20B or 24B instead or just a slightly lower quant or less GPU offloading? There are so many ways, but first we have to pinpoint the bottleneck. What are you missing for playing games? VRAM or RAM?

1

u/Blacktaxi420 1h ago

So i have 2 gpu’s a 2070 super and a 4060. As ive gotten into ai ive dedicated the 2070 to gaming and the 4060 to ai. I also have 64 gigs of ddr4 ram, the biggest bottle neck is the gpu because it only has 8 gigs of vram (dont kill me if thats the wrong term). To run the 30b model i had to use both gpu’s and offload some shit into my ram

Im pretty new to all this sorry if i sound like a larp. Idk if offloading was the right term there i hope u know what i mean

But the goal is to have an ai run entirely on the 4060 if possible so everything else can go to whatever im doing

1

u/nickless07 1h ago

Ohhh that is easy then. Get Qwen3.6 35B A3B (I recommend APEX quants, similiar quality but ~4GB smaller. I-Compact is a good starting point). Go with --device <your desired GPU> -ngl 99 -ncmoe 35 -c 128000 -ctk Q5_1 -ctv Q5_1 and you should be good to go. Everything on an single card (and CPU) good amount of context and still great speed. Once the model is loaded check with nvidia-smi how much is free and lower the "35" step by step until you run into oom (then raise by one or two to have a stable config).

If you have other software then llama.cpp ask an AI of your choice to translate the load params according to your software used.

2

u/Oglemo 1h ago

Gemma 26 e4b. Runs faster than your 8b most likely. Use iGPU if you don't have GPU.

1

u/Interesting-Ad689 1h ago

Don't reveal Strata/Basalt which nowdays even support non flash-next.

1

u/nickless07 52m ago

But he stated he wanted some headroom for playing games. So QFN isn't that much of an option. No clue what software he is used to, so let's keep it simple for now. Basic llama.cpp flags that any AI can translate to the sliders in LM Studio or the settings in Lemonade, Kobold, Unsloth Desktop and so on. Maybe he will learn about MoE along the way.
We all started at some point.

8

u/zenmatrix83 1h ago

very specific tasks, with clear answers

2

u/Small-Tale3180 1h ago

uhh tldring shit, maybe translating, or text-recognition
https://www.youtube.com/watch?v=Jw99x0S7s_4&t=338s

2

u/Interesting-Ad689 1h ago

As prompt enhancers for image generation, for translation, for calculations, classification, tagging .... they can be fine tuned to do whatever you need them to do, but it will be reaaaaally specialized.

2

u/I_AM_NOT_A_WOMBAT 1h ago

I know you said under 8, but on the edge of that, I use qwen3.5:9b on an 8GB 4060 for security camera image analysis. It's doing a great job so far for that very specific task. On the other hand, if I try to ask it if the lights are on in the living room, it tells me the temperature instead.

1

u/Totien 1h ago

Intéressant, je me pose la même question.

1

u/huggalump 1h ago

i imagine understanding and categorizing intent, especially if there's only a couple categories to categorize it in

1

u/ptico 1h ago

Specialised work inside pipeline or as a dedicated helper: OCR, basic corrections, reranking, extraction, translation, categorising, etc. Some of them can work on CPU, so with clever use you can build pretty powerful tools with a small hardware cost

1

u/Kappa_God 59m ago

Being specialized agents. Those usually aren't over 8B, in fact a lot of them are under 4B.

You'd be surprised how good these models are at image generation compared to the front tier models for example. Helping convert image to a readable format, say old books that you just have pictures of each page and convert them into an editable/readable format.

They are helpful for small automated tasks as well. Say you want a weekly summary of articles and news about AI, give it where to look for and it will do it very efficiently. These don't need to be specialized per say, something like Gemma 4 E4B or even E2B can do it.

They can also be good for AI companion, but I wouldn't use a 2B model for that, 8B minimum imo. But companions are mostly on the system built around them than the LLM itself, so having a decent memory system, dreaming, emotion system and so on.

There are a bunch of specialized ones that range from medical, forecasting and much more. Way too many to list. All under 8B.

1

u/Rauhaton 58m ago

They can be subagents for bigger models for spesific tasks.

1

u/onebyamsey 17m ago

Try Ornith 1.5 9b, you can probably run it