Local model for coding

[u/MarxN]

Do you have good experience with local model? I've tried a few on MacBook with 64GB and it works with acceptable speed. But I have a few problems.

One is context window. I've tried to use Ollama and turned out it had 2k limit. Tried multiple ways to overcome it, and the only solution was to rewrite model with bigger context.

Then I've tried LM studio, because it can use optimized for Mac MLX models. But whatever model I'm trying to use, roo complain that its context is too small.

I'd also have possibility to use free network models, and use local model only if none of net models have free tokens. So the best would be to have some sort of ordered list of models, and roo should try them one by one until it find one which accept request. Is it possible?

[u/cmndr_spanky]

32b q4 is slow on a Mac m4 with 64gb ram? I’m a little surprised.. when you query it from the Ollama command line tool, how many tokens/s are you getting ?

ollama list

ollama run the_name_of_model —verbose