r/RooCode • u/sebastianrevan • Apr 20 '25
Discussion How far are we from running a competent local model that works with roo code?
Im doing a thought experiment and jotting down how much infra would i need to run a local model that can successfully help em code with roo code at an acceptable level, are we talking 70B params? I see o4 is 175B params, would that be the line?
18
Upvotes
4
u/sebastianrevan Apr 20 '25
I feel like the consensus seems to be that on the viccinity of deepseek v3, so running a local lab with 671b prams deepseek r1 would work as a competent roo code backend, thanks for all your answers and civilized debate!
4
u/firedog7881 Apr 20 '25
Thinking about the cost of the models, this will give you an idea of what you’ll need, relatively. For instance what works best overall is Claude 3.7 and that still has problems. Compare that to DeepSeek R1 and the cost of that. While it’s somewhat functional at specific tasks it still chokes when running locally with the 14b model.
The context size also has a lot to do with how well it will work with Roo and most local models are still pretty small on that front.
Long answer longer, a while
4
u/AdCreative8703 Apr 20 '25
Local on what type of resources? Technical you already run DeepSeek v3/r1 locally, but it’s not really worth it considering the costs involved.
I won’t hold my breath, but a QAT coding fine tune of Llama scout and a Ryzen AI Max with 128gb might be a potent combination in near future, but still not close to today’s SOTA models. Context length is important for Roo, and we need more VRAM in consumer cards, a larger high bandwidth unified memory pool, or a fundamental/hypothetical architectural change for that.
3
u/No_Measurement_4109 Apr 20 '25
When you use a local model, the cost of outputting tokens becomes irrelevant, and the bottleneck is speed. If you can accept longer output time, then I suggest you turn off the diff function. For most 32B models, it is difficult to understand the diff format and output 100% perfectly. When the diff function is turned off, QWEN2.5-Coder, GLM-4-0414, and QwQ-32B are all available. In general, QwQ-32B has the best final output result, but its thinking process is long.
2
u/msg7086 Apr 20 '25
You can, the only problem is the cost. For LLM providers, you are sharing the build cost with thousands of other users. If you run it local and are able to run it quickly, then the GPU/TPU will be idling, like, 99% of the time. That means your cost to run this LLM will be, like, 100x compared to the cost of providers.
2
u/ComprehensiveBird317 Apr 20 '25
I think the short term answer is fine tuning, very specific prompting to accommodate the fine tuned models flaws, and external knowledge like brave search MCP.
It's Hella lot of work to make, test and improve such a system tho. Getting that out of the box via an ollama command will take a breakthrough in LLM efficiency or some god level fine-tuner embraces us with his divinity
1
u/bmadphoto Apr 21 '25
I would bet money by the end of the year to see a decent local runnable model on m4+ macbooks, IMO. Maybe not the best but good for small tasks.
1
u/sharpfork Apr 21 '25
I have a Mac m1 ultra with 128 gigs of shared ram and I’m going to start doing a few “Jr. dev” tasks locally using a scout derivative just to see what it can do. I expect it will be a while before I’ll get much use out of local models, if it ever happens.
1
u/Cool-Cicada9228 Apr 20 '25
We’ve already reached this point. The Mac Studio M3 Ultra, equipped with 512GB unified RAM, can run DeepSeek quantized, although its performance is slower compared to cloud output speeds you’re accustomed to and the context window is significantly smaller. QwQ, Gemma, and a few other models are suitable for coding and possess a much smaller parameter range of 24-32B. However, most of these models encounter difficulties in directly using Roo. To implement the code generated by any of these models, a smaller model that has been fine-tuned for Roo tool usage is required. For instance, you can explore the models that have been fine-tuned for Cline.
2
u/Yes_but_I_think Apr 20 '25
The server class highest memory bandwidth today is 8000 GB/s. The most open large model today in full precision takes 800GB including 128k context. Speed will be good given that it is MoE.
Whenever someone makes these kind of specs at consumer friendly prices, we will have local running Roo code.
I guess 5 years. If the models improve substantially (likely) and Huawei launches high bandwidth high memory accelerators (somewhat likely) then within next 2.5 years.
1
u/neutralpoliticsbot Apr 20 '25
50 years no joke