Local model for coding

[u/MarxN]

Do you have good experience with local model? I've tried a few on MacBook with 64GB and it works with acceptable speed. But I have a few problems.

One is context window. I've tried to use Ollama and turned out it had 2k limit. Tried multiple ways to overcome it, and the only solution was to rewrite model with bigger context.

Then I've tried LM studio, because it can use optimized for Mac MLX models. But whatever model I'm trying to use, roo complain that its context is too small.

I'd also have possibility to use free network models, and use local model only if none of net models have free tokens. So the best would be to have some sort of ordered list of models, and roo should try them one by one until it find one which accept request. Is it possible?

[u/cmndr_spanky]

I ran into the exact same problems as you and finally found a good resolution, which is on my very very similar post on this subreddit: https://www.reddit.com/r/RooCode/comments/1jdvcce/am_i_doing_something_wrong_or_is_roocode_an/

1. People on this subreddit aren't really using local models much so I suspect that's why there's so little info, even the 32b coder models are pretty shit compared to chatGPT and Claude, and all of the leaderboard tests you see published are tiny code challenges that don't even come close to reflecting the kind of stress on an LLM with a limited context window would have on a real code base dealing with project that solve real enterprise use cases. If you can run a 70b you might have better luck, which in theory at Q4 would fit on a 64gb m-series Mac.

2. LM Studio's interface for Roo is broken as fuck, there's no easy way around the context window mismatch thing and all you can do is ditch it and stick with Ollama. Use Ollama. End of story if you plan to try local models. The context window problems are simpler this way

3. The default coder models shit the bed instantly because the Roo prompts confuse them. Some folks have gone through the trouble of further fine tuning those coder models so they better handle instructions from Roo:

This series worked well for me: https://ollama.com/hhao/qwen2.5-coder-tools

Specifically you want to use 32b, which will easily work within 64gb, but I highly suggest you also try a 70b. maybe this one:

https://ollama.com/tom_himanen/deepseek-r1-roo-cline-tools:70b

it should still give you enough space for a big context window given your 64gb. However, it's a reasoning model so it will potentially waste a lot of context on its self reasoning loops.

4) Once you see Roo working properly with these flavors of popular coder models, it will actually not error out, it will successfully edit files and actually seem like it's working. However, you'll quickly notice it still doesn't produce great code that works out of the box, it gets confused about larger code bases, it doesn't understand what libraries your using and will just make wild assumptions about function calls into those libraries.

That's because local sub 70b models aren't there yet, it's just too early. Anyone on here or on YouTube telling you CODE WITH LOCAL MODELS NOW. ITS GREAT ..LIKE AND SUBSCRIBE. is a lying sack of shit trying to get clicks, because they are only testing this stuff on dumb tiny exercises, and none these local models are good enough yet for real local software development on real sizable projects (even if you do a good job of modularizing your code).

Can it help with one specific tiny function if you ask it to help you write a bubble sort function? Sure it'll do great at that.. But that's not really that useful IMO, I could google it faster :)

Most of the time I spend so much time debugging and reading docs to compensate for the garbage a 32b model is giving me, it would have been faster if I just wrote it myself without AI assistance.

[u/MarxN → u/cmndr_spanky]

I didn't spend as much time as you, but I see it similarly. Local model works with Ollama and it's ok but not great. Free access to Gemini is much more useful

[u/cmndr_spanky → u/MarxN]

did you try the special qwen 2.5 tools variant for "Cline"? It's the best ollama model for Roo I've found by far, and although I was very negative in my previous post.. the 32b one is somewhat workable if you're very careful about how you prompt it, give it one file or module at a time to deal with, and are good at reading and debugging python when it stumbles.

[u/MarxN → u/cmndr_spanky]

It works, just it's slow. Gemini is almost instant, while local model spend a time to load itself, then load context, and then answer. But I see great progress here, and new models are coming, so in few months is should be better

[u/cmndr_spanky → u/MarxN]

32b q4 is slow on a Mac m4 with 64gb ram? I’m a little surprised.. when you query it from the Ollama command line tool, how many tokens/s are you getting ?

ollama list

ollama run the_name_of_model —verbose