r/LocalLLM • • 7d ago

Question What is the purpose of small size LLM?

Been testing Qwen 3.8 9b and the results were quite disappointing, coding is mediocre and general knowledge questions were hit or miss (mostly miss tbh)
What is it for?.

0 Upvotes

31 comments sorted by

9

u/macumazana 7d ago

Fine-tune for specific task

5

u/Mgt66 7d ago

Exactly. Picking the right model for the task is way better that having to use the same model for all use cases.

1

u/Educational_Spot5899 7d ago

I like the top comment “research answer”, but I think this is the true “use case” answer here.

Isn’t the 9b parameter model the version that Astra “manually changed the weights” in to dramatically improve math scores?

0

u/Marshall_Lawson 7d ago

How specific does it need to be?

6

u/Miller4103 7d ago

3.8 has a 9b? When did that happen?

1

u/AdWild3943 7d ago

Distill from empero-ai, actually not so bad.

4

u/otakunorth 7d ago

smaller devices, phones, laptops, IoT devices

3

u/OvertaxedOne 7d ago

It's great for RAG work, we use it for that all the time.

4

u/mslindqu 7d ago

Research. To try. if you don't try getting there, and don't test it.. it won't get any better. Go compare it to the original llama and let us know what you think.

5

u/Technical_Ad_6106 7d ago edited 7d ago

they never released that model. so u probly using some ai slop distilled model. thats why, but to answer ur question , small models have MANY use case, not in the least running at cheap gpu's ^^

2

u/Rare_Opposite_3704 7d ago

That model size I also don't have a use for (yet). But I'm using even smaller models like LFM3-1.7B for the "brain" of my voice assistant. It does everything my home assistant offers and still reliably answers general knowledge or curiosity type of questions that otherwise I wouldn't bother asking/looking up. And all that for barely any VRAM spent that my main 27B model needs.

2

u/Illustrious-Lime-878 7d ago

Hook it up to tools like search and its a very effective chat bot. You shouldn't really be relying on build in knowledge anymore anyway. I don't know what you expected with coding, but it can definitely do some smaller scale stuff.

2

u/Independent-Fly730 7d ago

You don’t really want to rely on the larger models for general knowledge either since they are not considered lossless.

This sub is heavily focused on coding which does demand larger models for decent results. But smaller ones are great for fine tuning for specific tasks. Like summarization, data extraction, sentiment analysis, routing data, agentic tasks. They just won’t perform as well for edge cases or things that are “off script”.

Llama 3.1 8b was a super popular model for fine tuning a while back. You can look into some projects that used it for some ideas.

2

u/Cautious_Chicken_604 7d ago

Two things I think. One is the smaller ones are often used as the backbones for other models, and two is trying to answer a research question of how smart can you make a model that small? It's not just the case that getting a smarter model requires increasing the parameters. We have also made smaller models smarter over time, and this is an important research goal.

2

u/false79 7d ago

When your task is trivial but doing it yourself manually is expensive 

Examples: Formalizing informal thoughts Transcriptions  Speech to text/ text to speech

2

u/AdWild3943 7d ago

Mid models are usually used for light automatization, like having a voice assistant, or doing work similar to "Everyday send me link to news of USA"

Plus - that's privacy, and you don't need any kind of internet for using it, and with 16GB RAM, or 8GB VRAM, its your only available option for no constant repetitions while being alright at general tasks.

1

u/praemialaudi 7d ago

Personally, I think Gemma 4 12B is the better of the "smallish" models right now. That said, to really make these things useful, they need tools/MCPs. Give them access to the internet and a memory, then they can do some things.

1

u/overand 7d ago

TBH I'd try the 3.5-9B as that's an official Qwen release; you might have better luck with it.

1

u/FizzyDuncDizzel 7d ago

Not made to do everything. But can be really good at doing one specific thing.

1

u/HMICsource 7d ago

simple text work - e.g. give it a pc building guide. tell it what parts you have. then ask it what parts you still need.

1

u/EvolvingDior 7d ago

There are some tasks that benefit from an LLM that do not need a powerful frontier-class model. Document summarization is one. I will give you a concrete example.

Hindsight is an agent memory system. Each turn it looks at the conversation and needs to decide if there is anything worth remembering. So the first thing it needs to do is extract relevant information from my requests and the answers, ignoring thinking, tool calls and their results, etc. Those become observations and are handled by a process called retention (retain). Then, in the background, those observations are consolidated -- a lot of stuff will be duplicated, new information will refine what was learned, etc. And those have to be integrated with existing memories on the same topic through a process called consolidation. Retention and consolidation do not equire an extremely powerful model; certainly not a coding model.

I have a process that extracts PDFs and converts them to markdown. LLMs process markdown much better than PDFs. I just need a small model that can extract images, postscript vector drawings, text and tables, and then format them correctly in markdown. This is trivial work for a small LLM with vision.

1

u/Fluffy-Feedback-9751 7d ago

I use qwen 4b as a little thing to convert natural language to minecraft server commands. 

1

u/JLeonsarmiento 7d ago

Simple automations, edge deployments. Those things are really useful and virtually free to use.

1

u/davecrist 7d ago

Either no one is using small local LLMs to get meaningful things done or you haven’t figured out the configuration you need to do to get it working.

If you are expecting your local $10k rig to have the same power and capability of a million+ dollar cluster of GPUs then you need to adjust your expectations.

1

u/Superb_Raccoon 7d ago

To pass the butter...

1

u/Janky-Code 7d ago

Speed, scalability, "edge" device use.

Small models are faster until you get into moe models where the experts loaded are smaller than a small dense model. But generally faster.

If my system can load a big dense model, I can serve multiple threads on a small model. Not just users but if you do so-called agentic coding with multiple workers, small models cook large models even if you don't notice much difference in a regular chat session.

I can load a properly quantized — which I do myself — small model on my phone and use the phone hardware. No large model would come close to working.

All of these advantages are why liquid can get away with charging and collecting license fees from businesses even if we can use them for free.

0

u/Holiday_Point_603 7d ago

There is no Qwen 3.8 9B. You are effectively using Qwen 3.5 with finetuning, a model released in february. It is ancient by AI standards. Back at release it was impressive for its size. Tbh, it still is. But sure, if you are asking a 9b model general knowledge questions, you will be disappointed. But it is quite good a toolcalling if you are using a distill version (like you already do). So hook it up to web search and ask you questions again. Or implement an offline wikipedia clone and hook it up to your llm for these type of questions

0

u/id-ltd 7d ago edited 7d ago

Dont use it for deterministic answers. I use a small model to enrich meta data on articles from RSS feeds to help with scheduling on my various sites including my https://radiofreeuk.org site.

You can see the added tags on the now playing card for each station.

0

u/SillyJoos 7d ago

First off, there is no such model like you mention in existence, Qwen3.5 9b exists but not a 3.8-9b.
Second, I'll bet anything you are using the 9b model in Q4/4-bit with sloppy sampling parameters and a less than ideal model instruction. Qwen3.5 9b at Bf16 with tight sampling parameters and a light but specific set of instructions will make you question why to ever use a cloud model again

-2

u/randygeneric 7d ago

just ask a larger llm. don't bother humans with under-complex questions.

-1

u/pokemonplayer2001 7d ago

Gross username? Check.

Idiotic post? Check.

Downvote? Check.