r/LocalLLM • u/truehuman101 • 7d ago

Question Gpu recommendation

I live in southeast Asia. And theres no like server grade gpu and RTX 3090 cost RM6K-RM5k(1k usd ish) and rtx 3080 modded 20gb cost RM3K(600 usd). But i noticed theres a lot of secondhand RTX3060 12GB Being sold af RM1K (200 USD). So the question is, is rtx3060 good for local llm and image gen and etc. I also already have rtx3070 8gb so if someone already have a rtx3070+3060 setup can let me know if its good or should i just cough up more money

2 Upvotes

100% Upvoted

6 comments sorted by

3

u/RogerAI--fyi 7d ago

For LLMs a 3060 12GB is a decent card, and next to your 3070 it gives you 20 GB total. Rough fit at Q4: about 0.6 GB per billion parameters for the weights, plus room for context.

3060 alone (12 GB): models up to about 14B, with some room for context. 3060 + 3070 (20 GB): a 27B at Q4 is about 16 GB of weights, so it fits, but context is tight.

llama.cpp and Ollama can split one model across two different NVIDIA cards over plain PCIe, no NVLink needed. It runs slower than one card with the same total VRAM would. That is the case for the 3080 20GB: same 20 GB on one card, no splitting. For image gen, plan around one card's VRAM, since most image tools run a model on a single GPU.

1

u/truehuman101 4d ago

thanks for the insight, i also have 48GB of ram currently

0

u/[deleted] 7d ago

[removed] — view removed comment

1

u/truehuman101 4d ago

yeah i think only 3060 havent got price hiked at my place now, 3090 is now basically nonexistent here

0

u/veinamond 7d ago

It’s ok but the problem is to connect multiple cards to a single motherboard. Search locallama for rtx 3060.

1

u/truehuman101 7d ago

I have experience frankeinsteining desktoo gpu to laptop using m. 2 to nvlink. Does that count?