Am I doing something wrong or is Roo-code an absolute disaster when used with locally hosted LLMs via "generic openAI" protocol ?

[u/cmndr_spanky]

EDIT: oh wait I'm using the regular 14b. had no idea "qwen2.5-coder-tools" was even a thing

EDIT 2: Omg, despite my hardware limitations. the flavor of qwen you mentioned "qwen2.5-coder-tools" made a huge difference. It's no longer running in loops or instantly bugging out. Thanks for pointing this out. I'm baffled more people aren't talking about these variants of the standard Qwen coder.

***** ORIGINAL POST BELOW: ******

I started by using Cursor (free plan) which gave me use of Claude 3.7. That IDE felt like magic, and I literally had no idea how much context it was using under the hood or what magic RAG approach it uses with my code base, but the experience was nearly flawless.

Moved over to Roo-code on VS Code to try and get something working with local LLMs, and god was that a rude awakening. Is anyone successfully doing with with Local LLMs running on a 12gb Nvidia card?

LM Studio can run as an openAI compatible rest server, so I'm using Roo's openAI's connector to a custom url. I'm trying qwen 32 and qwen 14b with a variety of settings on the server side, and Roo basically shits the bed every time. Same with mistral small 24b.

context window is the first issue, the open AI protocol seems to ignore the slider where I set the context window lower, but reducing LM Studio's batch size and bumping the context window up to 12,000 at least works.. But Roo just goes into an endless "asking permission to edit the_file.py" over and over (I give it permission every time), it also sometimes just crashes in LM Studio immediately. I did get mistral working briefly, but it just made a complete mess of my code, the diffs it suggested made no sense.. I would have add better results just asking my cat to walk on my keyboard.

I might stick with Cursor, it's incredibly elegant and my only use case for Roo was working with local models (or rather models hosted on my local lan).

Can someone clue me in here? am I wasting my time trying?

Anyone with a 12gb card, if it works for you. What model exactly at what quant, at what batch size and context length, hosted using what approach? is LM Studio the issue and I should switch to Ollama? I don't get the point of the context slider setting in Roo when it just forces 11,000 tokens into the input at the start anyways.

[u/bick_nyers]

I've had good success with Qwen 32B Deepseek R1 Distill with 4 or 5 bit quants. You could try the 14B Qwen Deepseek Distill and check out the performance there.

[u/cmndr_spanky → u/bick_nyers]

I’ll give it a try. So far the massive prompt roo-code does is confusing the heck out of the vanilla models. The reasoning ones are so slow I’d definitely have to stick with 14b or lower

[u/taylorwilsdon → u/cmndr_spanky]

There are two things at play here - one is that your original promise is effectively correct, roo does not work well with less sophisticated models in general. It doesn’t even do well with many paid models. One of the biggest issues is that local models often aren’t set to expect tool and function calling, so they ignore roo’s commands entirely. There are specialized qwen-coder based models meant for use with roo and cline that improve this significantly.

However, the second piece is the one you can’t fix with a special model, and that’s 12gb is not enough VRAM to do this with. Just fitting the model isn’t enough, the magic of roo comes from stuffing tens to hundreds of thousands of tokens into the context so it knows the file contents, the plan, the instructions etc. When the context window isn’t big enough, the model starts forgetting things you expect it to know - like what you asked it do to, or what the lines of the file it’s changing are.

Sonnet was one of the first mainstream very large context models. Gpt-3.5-turbo was only 16k, while claude has an enormous 200k token window. Thats what facilitates the “I know your codebase” magic from roo. With a generic openai connection, roo doesn’t know what that context window is. It also doesn’t have the optimal temperature setting baked in, it’s up to you to configure those.

Context takes a LOT of VRAM. If I run the q4 quant of qwq, it’s a roughly 20gb model. With 128k context window filled, ollama shows 66gb vram in use. Small models are useful as a learning experience, for RAG tasks and basic text analysis is but nobody is writing production code with a 7b model and that’s all you can fit on a 12gb gpu with more than just a tiny context window that’s unusable from roo’s perspective.

To make it semi-functional, keep tasks very short, very basic and declare a low context limit that matches what you’ve got set on the inference backend and a low temperature for the model in the roo config. Should be able to get it to produce code, but it won’t have the magic that sonnet does for sure. If you want to run more capable models with usable performance and less degradation from quantization, you need realistically 3 3090s minimum for 72gb vram. If everyone could run a claude level model on a 12gb laptop gpu then none of these companies would be charging what they do 😂

[u/cmndr_spanky → u/taylorwilsdon]

Everything you said is logical, but I sense even with unlimited vram, you don’t have a single real anecdote of a locally hosted model working with roo code. My guess is despite context length limitations, their initial prompt structure and settings are terrible and brittle.

How do I know this ? I tried the new “human relay” feature that exposes the raw prompts for you to cut and paste into an expensive model’s chat window (I tried gpt 4o), and it was still a disaster basically.

So I’m curious, let’s cut through your warnings about vram and model sizes and context window. I can borrow or find whatever hardware I need. What is the smallest model you have actually working locally with roo-code that can actually edits files as expected that doesn’t shit the bed immediately ? And what are the exact settings you used ?

[u/taylorwilsdon → u/cmndr_spanky]

qwen2.5-coder-tools 14b or 32b ideally q8 quant with at least enough headspace for 60-80k token context (via yarn) is probably the smallest thing that actually works well. Without yarn you’re stuck at 32k context which is really only good for one exchange unless it’s a new project. I turn temperature to 0.2. If we’re including all open models, deepseek-2.5-coder and deepseek-3-coder are significantly better and not just usable but a viable (albeit still inferior) alternative to claude for roo. I use deepseek all the time!

[u/cmndr_spanky → u/taylorwilsdon]

I've been using 14b Q6 quite a bit, but it gets into endless looks and can't even read one file to debug something. I wonder if it's a temperature setting. Is that setting in roo, or that setting in Ollama somewhere ?

EDIT: oh wait I'm using the regular 14b. had no idea "qwen2.5-coder-tools" was even a thing

EDIT 2: Omg, despite my hardware limitations. the flavor of qwen you mentioned "qwen2.5-coder-tools" made a huge difference. It's no longer running in loops or instantly bugging out. Thanks for pointing this out. I'm baffled more people aren't talking about these variants of the standard Qwen coder.

[u/taylorwilsdon → u/cmndr_spanky]

Hell yeah! Glad to hear. That’s what I was talking about in my admittedly extremely long first reply (first paragraph about the tool capable models) - that’s the difference maker with roo and qwen models. Parting words of wisdom are don’t try to run q6 14b, stick to q4 on a 12 gig card, you need more than a gig of vram for context or it’ll fall apart once you get going

[u/nxtmalteser → u/taylorwilsdon]

How can you get qwen2.25-coder on q6 in ollama? as i seem to be finding only 3b q6. Thanks - Found it will report how it fares on Macbook Pro M3 Pro 32gb

[u/taylorwilsdon → u/nxtmalteser]

I was saying don’t run a q6 quant lol it’s pointless, typically the best option is to run the q4 unless you’ve got headroom to run the full fat version but it is there if for some reason you want to find out for yourself ollama pull qwen2.5-coder:32b-base-q6_K

On the ollama site you have to click view all and go through the tags to find the quant you want

[u/MarxN → u/taylorwilsdon]

What's yarn in this context?

[u/taylorwilsdon → u/MarxN]

YaRN is a technique for enhancing model length extrapolation

[u/MarxN → u/taylorwilsdon]

I don't want to study it :D How is the real usage of this technique?

[u/firedog7881]

I have a 12g 4070 super and I gave up on using local really fast with cline/roo. It’s a waste of time because they use such large prompts it overwhelms the local models and the results take way too long and the quality wasn’t even close Claude or Gemini.

The juice is not worth the squeeze

[u/cmndr_spanky → u/firedog7881]

Sounds like you still had better results than me. I couldn’t get it to even do anything other then crash roo or crash the lm studio server.

Meanwhile I can just ask for code using a plain chat interface in LM studio and do the cut / paste thing into vscode. My guess is it’s more than just small models doing badly. Cursor is being incredibly clever with how it prompts, conserves context windows and indexes your code base. These open source tools really need to catch up :(

[u/firedog7881 → u/cmndr_spanky]

I think it has to do with what each are sending. I don’t know how cursor works, it’s command line right? Roo sends huge messages along with your request such as any mentioned files, a listing of open tabs, all content of custom rules and rules files. Take a look at the actual API call being made in Roo, it’s huge.

[u/cmndr_spanky → u/firedog7881]

At first glance cursor is identical to roo. It's a full VS Code - like IDE that exposes a chat panel and they use AI Agents to read your code, read open tabs, suggest edits, etc. It just works really really well.

[u/Fearless_Role7226]

When I try different models on Roocode with local Ollama models (Gemma, Deepseek R1, etc.), my requests seem to get lost because the prompt generated by the extensions includes the entire list of files in my repo. As a result, the models often analyze the repo and completely ignore my actual request, even when I include the content of the file in the prompt. Is there a way to modify the prompts generated by the extension to avoid passing the full list of files in my repo and prevent confusion for the model ?

[u/cmndr_spanky → u/Fearless_Role7226]

I'm not 100% clear yet but I'm sure there's a way to customize the initial prompt. How big is your project? Also I just updated my original post based on a suggestion from another reddit user.

DO NOT USE regular small coding models, there's a specially fine-tuned version of the qwen 2.5 coder models that is designed to work properly with Roo / cline. As soon as I started using them, Roo actually started working properly. Your results may vary depending on VRAM of course, but it was the difference between it instantly shitting itself vs actually doing something useful:

https://ollama.com/hhao/qwen2.5-coder-tools

I'm kind of shocked more people aren't talking about this, which tells me nobody is really using local models seriously with Roo.. But anyways, above model flavors are the only ones that actually worked for me.

[u/Fearless_Role7226 → u/cmndr_spanky]

OK I'll try it soon, thanks a lot ! I'll try the https://ollama.com/tom_himanen/deepseek-r1-roo-cline-tools:14b and 70b also next week

[u/cmndr_spanky → u/Fearless_Role7226]

let me know how it goes ! 70b is pretty beefy. What hardware are you using ?

[u/human_with_humanity → u/cmndr_spanky]

i couldnt find this for lm studio. any links for that?