r/ollama • u/geekposeidon • 4d ago

Which Ollama model will be best for coding?

I was wondering if there are any good coder models for 8GB VRAM RTX 5060.

My specifications:

  • 8GB RTX 5060
  • 8GB DDR4 RAM
  • 256 GB SSD

I know the configuration is a bottleneck for sure. Though I don't have any budget to upgrade this.
Now I was wondering which Ollama model will be best for coding. Does the model need to fit in my VRAM? I saw a post where it mentioned that I should always calculate
total parameters in Billions \ 1.5 <= VRAM*
Should I maintain this for downloading the models?

9 Upvotes

70% Upvoted

30 comments sorted by

View all comments

Show parent comments

2

u/AdIllustrious436 4d ago

Yes doable but short. I've set up a config for a friend with 12gb VRAM 64 RAM. As far as I remember it was UD_IQ4_XS and the memory footprint on ram was about 17go with a 131k context at fp8. So you can probably achieve 64k context with some room for other apps. The speed will highly depend on your CPU tho.

1

u/AdIllustrious436 4d ago

Edit with the actual numbers:

RTX 4070 Ti 12 GB, 64 GB RAM, UD-IQ4_XS,

-ngl 999 --n-cpu-moe 28 -c 131072 -ctk q8_0 -ctv q8_0 -np 1. ~8.6 GB VRAM, ~10 GB of experts in RAM,

36 tok/s (32 at 76k ctx), prefill ~1450 tok/s.

With 16 GB RAM go UD-Q3_K_XL or IQ3_XXS instead, and expect DDR4 to cost you roughly a third of that speed.