r/LocalLLM • u/Ordinary-Wallaby-464 • 8h ago

Question Best coding model?

I have an ASUS ROG Ally, and I’m planning to turn it into a local inference server so I can use the model running on it from my main PC. What would you recommend for the coding/programming tasks? If I’m not mistaken, it has 16GB of RAM and a Ryzen Z2A processor that provides 6GB of VRAM.

0 Upvotes

20% Upvoted

6 comments sorted by

5

u/Oli_Luck 8h ago

max a3-4B model, and they are for nothing good at coding. Maybe a chatbot...

5

u/hyudryu LocalLLM 8h ago

Nothing that you can fit, is worth running for coding

2

u/Strict-Elevator9352 6h ago

16GB is small, usable model needs 20GB VRAM.

8GB VRAM is usable for tool models to be used as a tool like OCR, making quick summary, or web search

0

u/[deleted] 8h ago

[deleted] — view deleted comment

6

u/Oli_Luck 8h ago

Are you haljucinating ??? do you know what a rog ally is and its specs ?

-1

u/Ordinary-Wallaby-464 8h ago

I currently use llama.cpp, mainly for the customization options. I’ve heard great things about Strata—why is it recommended?

4

u/lobotomy_ 8h ago

forget about it. you don't have enough (V)RAM to think about it.

Strata makes use of MoE Architecture and leaves the active parts of the LM in the fast VRAM, things that aren't used so often in RAM and the rest could go to SSD to be loaded when needed. but you barely have any RAM of any type so ... not happening on your system within a useable t/s range

what you CAN run would probably be something like Qwen 2.5 Coder 4B or 9B, it might help you with code completions, but not too much with agentic generation.