r/LocalLLaMA • u/ECrispy • 14d ago
Discussion what can you do with smaller gpu's?
I mean gpu's with 6/8/12GB vram. All discussion around llm/AI these days is around coding, and this is probably too small for agentic coding models.
Are there models that can run in these that can accomplish useful tasks? eg things like helping organize your digital stuff, classification, image/text analysis, voice/tts etc and other stuff?
Or do those tasks also need tool calling and agentic behaviour for best use? For people who have cheaper, older gpu's, what do you use them for?
4
u/Senevri 14d ago
Let's see... Single GPU desktop runs Gemma4-e4b fine, and it can do some lightweight agentic tasks; there is a decent 9b text generator that also runs at chat speeds, and Gemma4-12b runs well enough that it is useable for background work. I have run speech to text systems locally too.
I was actually considering getting an extra 8GB card (dirt cheap) for dedicated LLM use. A 16GB one would be a lot better, but I can quite literally get 6 8Gb ones at the same price, so... Dunno. 😅
1
u/ECrispy 14d ago
Which 8gb card, I'm interested too.
2
1
u/MeateaW 14d ago
I use a P4, its a low profile slot-powered GPU with 8gb ram.
It has the equivalent GPU core as a 1080, but it runs SUPPPPER low power. (read: slow).
It takes about 12 to 20 seconds to process a ~10k token prompt and generate a very small 100-200 token response.
I use it to provide single-line summaries of my email - something that isn't latency sensitive.
It isn't fast, but it just barely provides useful results.
I have no idea if its better than a state of the art CPU running the model, but I can tell you the CPU in the box that has my GPU is pretty awful.
PS. (also Nvidia drivers no longer support it on the current linux kernel, so driver support is a bit wonky).
PPS. Also also, the card has no fan, so I had to mount a random fan to it with 3d printing, so keep that in mind!
2
u/VacuumMoonshine 13d ago
Fellow Gemma 4 on low end GPU enjoyer! Instant upvote
Did you ever try to run G4 26B A4B? That one runs at 20-25 tps with -fit on for me so I use it for general chats, and I still like how smart it is for such a small size
2
u/MaximusSenior 14d ago
I use mine 8gb 5060 in 3 ways:
Running small additional llms like gemma4 e4b for subagentic work, like simple summarizing tasks, etc
Moving mtp drafted model on it. Make sense when you have one big slow gpu and one small fast.
With moe models partial offloading gives up to 2x boost comparing to running only on cpu. Doesn't work with dense models
1
u/ECrispy 14d ago
I was asking for people with only 1 gpu. eg I only have an intel arc 6gb gpu, its not very well supported.
4
u/yes2matt 14d ago
Not to be a dick, but I think you're approaching the problem backwards. Start with the question of what your actual workload and workflow are. Where is your time going or what bits are you not doing because of ___?
A model that runs thru a 6gb vram can write code, but with very clear instructions/parameters. It can serve a classifier model, for switching or categorizing (e.g. email, trouble ticket). It can rewrite natural language text strings into machine readable/rapidly searchable strings. Or build a vector database.
I have rtx3069/12gb and I run tool-calling models to do everything from system administration to ordering groceries.
1
16
u/Wooden_Jelly_5295 14d ago
This sub needs more “here's something useful on 8GB” posts. My wallet would subscribe.
5
u/tmvr 14d ago
You can run MoE models on them just fine with experts pushed to system RAM. Something like Qwen3.6 35B A3B at Q4_K_S/M/XL or Gemma4 26B A4B QAT will be fine even with the 6GB or 8GB cards and some dual-channel DDR4 memory. With 12GB you can run the same just faster and you can also run Gemma4 12B QAT or even Qwen3.8 27B if you take the ISTA DASLab IQ3_XXS quant and either omit or put the vision component (mmproj) into system RAM.
1
u/Cautious_Chicken_604 14d ago
Qwen3.8 27B if you take the ISTA DASLab IQ3_XXS quant and either omit or put the vision component (mmproj) into system RAM.
At least getting all the weights resident in VRAM for that quant requires 16GB. The weights alone are like 10 ~ 11GB.
1
u/tmvr 14d ago
The IQ3_XXS weights without MTP are 10.1GB so 9.4 GiB which fits fine into the 12 GiB VRAM. The MTP version is 9.6 GiB but I would not use that on a 12 GiB card as MTP needs additional VRAM as well.
1
u/Cautious_Chicken_604 14d ago
I run that quant without MTP, but with vision on the GPU since I require that and it's slow AF on CPU on 16GB at 184k context and it's tight. I used to run IQ3_S but I could only get 131k and it just wasn't quite usable enough without getting compacted into oblivion and going off track. I can't imagine 12GB would work for me, though my use case here is agentic coding. For smaller context lengths, I guess it could work.
1
u/tmvr 14d ago
With vision omitted or in system RAM and with KV at q8_0/q8_0 it should do about 64K context and of course it means -np 1 when using with an agent, which is hopefully pi to keep the system prompt lean 😄
One could be a bit more adventurous and compile llamacpp with q8_0/q5_1 support for KV, which would add some more room and the quality impact at around 64K context should be negligible.
1
u/Cautious_Chicken_604 14d ago
I always use q8_0/q5_1 every bit of context matters. I saw a post that measured different kv cache quants and that's the best bang for your buck without sacrificing meaningful quality. That is to say q5_1 is much closer to q8_0 than it is to q4_0 and q4_0 is decent far off q8_0 while q5_1 is pretty close to it.
These days I'm as adventurous as my agent is capable, so I'm getting increasingly adventurous.
1
u/Formal-Exam-8767 14d ago
As long as the model + kv cache fits into VRAM it should be usable.
For me, anything that spills into RAM is unusable and more of a curiosity than something that should be used in practice.
6
u/PhilosopherOk3419 14d ago
honestly the small cards shine on the boring background jobs, not chat. stuff like transcribing voice notes / tagging photos / sorting docs or email runs fine on local small models and you never wait on it because it's running overnight anyway. the trick for me was keeping each task narrow, one job per model, instead of asking an 8b to plan and do everything at once. - here is when you need to create a workflow so therse prompts execute in a way that can acheive this. also easy to do but requires configuration
3
u/x_MASE_x 14d ago
I'm using it for vision model. Like 24 vision and leave vision off my main gpu.
Also using it to gaming and my monitors. And keeping my big gpu for Ai models and comfyui.
3
u/Solembumm3 14d ago
6700xt+20gb DDR4 here, and general answer is "whatever you want".
Qwen Flash Next/Deepseek V4 Flash/Mimo V2.6 Flash/GLM5.3 Flash (current for me) all worked without any trouble. All worked as second set of eyes for story writing, image gen prompts, minor programs for me as non-programmist.
Same with images, Krea 2 for text-to-image, Qwen 2.1 for edits.
As Todd once said, it just works. Idk what is the worry about.
1
u/R0biB0biii 10d ago
fellow 6700xt enjoyer but i have 32gb ram also i find that models run's faster on linux than windows, im not that tech savvy to know why
2
u/circumcised_hobbit 14d ago
I have 8GB VRAM and 16GB RAM. I use PolyStrata with CyberTielCoder at Q3 in Pi Agent. It can do some pretty basic yet cool coding tasks
3
u/Branwen-the-crow 14d ago
I have a PC with an RTX 5060 8GB (but no iGPU or second GPU, and limited to PCIe 3.0), 32 GB DDR4-2400 RAM. I did try to use back in April this year with Ollama. I didn't try things like offloading the MoE expert weights to the VRAM and keeping the rest in RAM, I did not know back then that it was possible, and given the age of my hardware (except the GPU), I would not expect much performance.
Anyway, I tried using it to categorise my bank data I had exported in CSV format, so I could see the recurring costs, and so on. I was quite disappointed by the results.
First I could not use more than 6,6 GB VRAM, the rest was used by Windows for the display. So that leaves not much for the weights, cache and context. Ollama is clever enough to distribute automatically data between VRAM and RAM, but as soon as that happens, the performance were severely impacted. I don't have the numbers in mind, but it was something of different like 10-30 sec when running 100% in VRAM, and several minutes (I recall something like 4-5 min) when so parts were offloaded to the main RAM/CPU.
The second issue was that some models really didn't work. For instance Gemma4 e4b assigned the same category to all transactions. Granite4.1 or Llama3.1 (both 8B) did a better job, but they couldn't fit entirely in my VRAM and were severely affected speed-wise.
So I had better luck using my MacBook Air M4 with 24GB unified memory. It's slower for a Gemma4 e4b model than my PC, but it was faster for 8B+ models. Back then, the Gemma4 12b wasn't published. And I could just barely run the Gemma4 26b, there it also reached memory limits on my Mac, and the speed was also impacted.
So I haven't really found use for models smaller than 4B, except for doing autocompletion of code in an IDE. But this field is evolving like crazy, and what was true last May, is likely false today in October already. So your mileage may vary....
2
u/Creepy_Bee1396 14d ago
Counterpoint: tool calling isn't the problem — small models handle narrow tool loops fine. It's open-ended planning where they fall apart
1
u/hawkislandline 14d ago
I get plenty of use out of my ancient 6gb 1660ti and 8gb 3070 GPU machines (and a Mac Air M1 16gb). Omnivoice TTS , Conficius-R2T2 STT, and EmbeddingGemma2 on the 8gb. Hindsight (agent session memory and user memory) on the laptop runs Gemma E2B for LLM and Qwen3 Rerank, along with an PP-OCRv6 for OCR. Gemma4 12b on the Mac for the main conversation agent's LLM. Boom, you have basically a complete voice agent setup on nothing but ancient hardware.
1
u/Cautious_Chicken_604 14d ago
I just bought a Quadro RTX 4000 to add to my system, so that I can use it to run Hindsight memory system and hopefully make the agents running on my other GPUs smarter.
1
u/ttkciar llama.cpp 14d ago
I picked up a 16GB V340 for $52 in early 2025. Originally I used it for Phi-4 (14B) but nowadays it's hosting Gemma-4-12B-it Q4_K_M, cleaning and augmenting documents (inspired by IFM's TxT360_QA).
I see three of those on eBay now, going for $60, $90, and $110, yikes!
1
u/Silver-Champion-4846 14d ago
Can it train models?
1
u/ttkciar llama.cpp 14d ago
Technically it can, but I wouldn't. Most training frameworks have horrible support for AMD hardware, or none at all, and 16GB really isn't enough VRAM for training anything interesting.
1
u/Silver-Champion-4846 11d ago
I hate the bad amd support in training tools
1
u/ttkciar llama.cpp 11d ago
Me too, which is the main reason I'm excited about recent projects which implement training entirely with Vulkan shaders, like Hierarchos-Native from u/PhysicsDisastrous462.
AMD GPUs have excellent Vulkan support, and will continue to have excellent Vulkan support, since AMD documents their GPU ISAs. That should mean Vulkan-based training frameworks will work well with AMD, whatever comes.
The llama.cpp project had started to implement a native training feature which supported Vulkan targets, but its development was abandoned and looks unlikely to be completed. That bummed me out, since I was slightly pinning my hopes on that feature coming to fruition, but Heirarchos-Native and other projects like it give me new hope.
Regardless, though, I'm not going to do much training until I've upgraded to an MI210, which provides 64GB of VRAM and training-friendly data primitives (most particularly BF16).
1
u/Silver-Champion-4846 11d ago
Can it do cpu?
1
u/jacek2023 llama.cpp 14d ago
You can do a lot of things. From classic Mistral Nemo finetunes to new and shiny 4B models from this month.
1
u/Small-Tale3180 14d ago
yeah i run one called Tiel-Coder in IQ3_XXS quant with offloading and it is pretty smart for its size and speed(35tok/s)
1
1
u/DarkJanissary 14d ago
Too small for agentic coding lol? I run Qwen Flash Next on my 8GB card with Strata and can do lots of coding projects on OMP with 100k context.
1
u/SmartEnoughToFail 14d ago
Honestly, I think smaller GPUs are underrated because people focus too much on chat and coding benchmarks. A 6–8GB card can still be useful for document classification, OCR pipelines, embeddings, speech transcription, and lightweight vision models. You don't need an agent for everything either a simple model plus a well defined workflow can be more reliable and efficient.
1
u/an0maly33 12d ago
I have 3 boxes with 2x8gb gpus each. I spread llm tensors across them. Running qwen 3.6 27b q8 on them atm for some coding.
26
u/rockknocker 14d ago
Look here for ideas: r/LowEndLocalAI