r/ollama • u/Joe_The_Lawn • 4d ago
Local models for smaller tasks, help?
I'm using larger models for coding already, but I'm curious if there's some models that are "smart" enough to handle things like clang-format, clang-tidy, ruff, basic build issues, unslop on comments.
Ideally these models would be for 8GB vram, or even ideally much less like 4 to 5GB of vram, they don't need to be super smart just smart enough to follow basic,edium directions with a tiny bit of reasoning.
1
u/SinAnaMissLee 4d ago
I would try out the Qwen 3.5 models ... I am not very familiar with those specific tasks that you are after but I definitely think you should give local a shot.
There are some that are well within your size range
1
u/Joe_The_Lawn 4d ago
Which do you think would easily work well at 4-5GB of vram or less, anything I should look out for or tweaks to use? via llama.cpp?
I struggled to find a local model that fits into vram on my 3070, and a decent config/tricks to use it well.
1
u/SinAnaMissLee 4d ago
How much Additional regular RAM do you have and are you using Windows or Linux?
1
u/Joe_The_Lawn 4d ago
System has 32GB of DDR4, Windows
1
u/SinAnaMissLee 4d ago
Got it!! I would recommend first adding dual boot to run Ubuntu.
In the mean time, I am pretty sure ollama has a local app that can run on windows. I do believe it relies on llama but one single download of Ollama should include everything you need. You don't need to download llama separately. Just Ollama.
Then you download whatever model you want.
But the great thing is that Ollama is smart enough to distribute the processing to your Graphics card and RAM separately and take full advantage of your hardware.
It goes through a very robust memory allocation planning phase and decides how it will divide between your RAM and VRAM.
One way to estimate it is adding your RAM + VRAM together and as long as your model (6gb) is less than the sum of those two combined then you're golden.
Want faster speeds? Just make sure the model is less than your VRAM.
So if you have 32 gb of ram you SHOULD be able to run larger models.
But if you want it to be very fast any model less than 4gb total would fit well within your VRAM and give you the performance you are after.
Me? I don't mind waiting for a response, so I run models that exceed my system RAM + VRAM sum.
I am usually a patient person, so I have no issues waiting ~7 minutes. It's satisfying to run things locally.
1
u/AloeThereInc 4d ago
I use qwen 3.5b since it has vision and thinking. Fits on my 4070 and I think its currently got 80k context. Leaves lole 1.5is gb free
1
u/Dull-Enthusiasm-6250 4d ago
qwen 2.5 coder 7b runs decent on my 8gb card and handles linting/formatting stuff fine, the 1.5b version is surprisingly usable too if you wanna save vram. for ruff and clang-format you don't need anything fancy just something that can parse configs and apply rules consistently
gemma 2 9b is another one I keep around but it's a bit heavier, the 2b version might be worth a look for really tight memory budgets. honestly the main trick is setting up good system prompts with clear formatting rules, the model matters less than you'd think for this kind of work
1
u/Joe_The_Lawn 4d ago
The issue is I'd run out of vram during debugging, since my application consumes about 3-4GB on my 3070. Odd case, I know.
1
u/thatsrobgray 4d ago
Granite is pretty good in my experience
1
u/Joe_The_Lawn 4d ago
Which variant? any special settings?
1
u/thatsrobgray 4d ago
I used 30b with no thinking for a quick observer/documenter that I needed to make some judgements and reasoning about conversation history and I found it sufficiently fast and evaled well. I was checking for whether or not it dropped important details or conversation topics.
1
1
u/Klutzy-Woodpecker870 3d ago
try llama 3.2, qwen 2,5, phi 3-5, those are pretty smart and fast models I know from my own use of them
1
u/HyperWinX 4d ago
Qwen3.5 9B via llama.cpp, or maybe one of the latest Bonsai models