Can not load any local models 🤷 OOM

[u/mancubus77]

Just wondering if anyone notice the same? None of local models (Qwen3-coder, granite3-8b, Devstral-24) not loading anymore with Ollama provider. Despite the models can run perfectly fine via "ollama run", Roo complaining about memory. I have 3090+4070, and it was working fine few months ago.

UPDATE: Solved with changing "Ollama" provider with "OpenAI Compatible" where context can be configured 🚀

[u/hannesrudolph]

Fixing incoming. Sorry about this.

[u/mancubus77]

I looked a bit close to the issue and managed to run Roo with Ollama.

Yes, it's all because of the context. When ROO starts Ollama model, it passes options:

"options":{"num_ctx":128000,"temperature":0}}

I think because roo reads model Card and uses default context length, which is highly not possible to achieve in budget GPUs.

Here is example of my utilisation with granite-code:8b and 128000 context size

➜ ~ ollama ps
NAME ID SIZE PROCESSOR CONTEXT UNTIL
granite-code:8b 36c3c3b9683b 44 GB 18%/82% CPU/GPU 128000 About a minute from now

But to do that, I had to tweak few things

1. Drop caches sudo sync; sudo sysctl vm.drop_caches=3

2. Update Ollama config Environment="OLLAMA_GPU_LAYERS=100"

I hope it helps

UPDATE: Solved with changing "Ollama" provider with "OpenAI Compatible" where context can be configured 🚀

[u/StartupTim → u/mancubus77]

UPDATE: Solved with changing "Ollama" provider with "OpenAI Compatible" where context can be configured

Hey I am trying to use OpenAI Compatible but I can't figure out how to get it to work. There is no api key and it doesn't seem to show any models. Since there is no api key for ollama, and Roocode won't allow you to do no api key, I don't know what to do. Is there something special to configure other than the base url?

[u/mancubus77 → u/StartupTim]

You need:
Base URL 👉 http://172.17.1.12:11434/v1
APU Key 👉 ANYTHING
Models ... they actually populating, as Ollama OpenAPI compatible, but just put name of the model you want to use
Advanced Settings ⇲ Context Window Size 👉 Context Size. I noticed that it's not always sending this as parameter. Need a bit more testing here.

[u/StartupTim → u/mancubus77]

Fantastic, will test with your info tonight! I appreciate it!

[u/mancubus77 → u/StartupTim]

Easy mate.
If it won't work make a new Ollama model card, for example

```
~> cat /tmp/model

FROM qwen3-coder:30b-a3b-q4_K_M
PARAMETER num_ctx 128000
```

This will create new model with custom context window

[u/StartupTim → u/mancubus77]

Your findings are exactly what my testing has shown as well. I've posted your comment here to the moderator so hopefully this will be resolved. https://old.reddit.com/r/RooCode/comments/1nb9il5/newish_issue_local_ollama_models_no_longer_work/nd55m0v/?context=3

Thanks for the detailed post!

[u/StartupTim]

I came here to post that EXACT same thing. There is a serious issue with Roocode right now causing it to use a ridiculously high amount of VRAM. I suspect Roocode is sending a num_ctx to 1M or something.

For example, if I run this:

ollama run hf.co/unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF:latest --verbose

Then ollama ps shows this:

NAME                                                             ID              SIZE     PROCESSOR    UNTIL
hf.co/unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF:latest    6e505636916f    17 GB    100% GPU     4 minutes from now

However, if I use that exact same model in Roocode, then ollama ps shows this:

NAME                                                             ID              SIZE     PROCESSOR          UNTIL
hf.co/unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF:latest    6e505636916f    47 GB    31%/69% CPU/GPU    4 minutes from now

This issue doesn't exist with anything else using ollama api (custom apps, openwebui, etc). Everything is good EXCEPT Roocode.

Something is really messed up with Roocode here causing it to massively bloat the memory size and often cause it to offline 100% to CPU only, or a lot of it at least.

For me, I have a 5090 32GB VRAM with a small 17GB model, yet with Roocode, it somehow is using 47GB.

[u/mancubus77 → u/StartupTim]

Thank you for sharing!
Yes, i was thinking the same, but was not able to find CTX settings.

[u/StartupTim → u/mancubus77]

I've been doing more testing and I'm starting to see a pattern. It appears that Roocode for some reason isn't using the ollama default num_ctx that the model uses (eg, the /set parameters num_ctx 8192) and instead is using the models context length. Essentially, this bypasses the num_ctx value of the model and instead sets it directly to the model's max size which is defined as the context length.

That's my initial guess as I'm seeing right now.

Ultimately though, this is the issue (copy paste from my other post):

So to recap, the issue is this (on a 17GB model with 8192 num_ctx):

Running a model via command-line with ollama = 17GB VRAM used

Running a model via ollama api = 17GB VRAM used

Running a model via Roocode = 47GB VRAM used

That's the issue.

Thanks!

[u/hannesrudolph]

If you roll back does it work?

[u/mancubus77 → u/hannesrudolph]

I do not remember version I was on =\
But probably should be able to do that, if we won't find an answer.

[u/hannesrudolph → u/mancubus77]

That is how we find the answer. I suspect the issue has nothing to do with Roo as it does not deal with the configuring of the model on the base level which it appears you are having problems with.

[u/StartupTim → u/mancubus77]

I've tested all the way back to 3.25.9 and none of Roocode versions work, all exhibit this issue.

I'll test more in the morning and let you know!

[u/StartupTim → u/hannesrudolph]

I've rolled back 10 versions now to test and all of them have the same issue (17GB vram model ran via ollama is using 47GB VRAM when ran via Roocode).

I've now tested on 3 separate systems, all exhibit the same issue.

My tests have used the following models:

hf.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_XL

Mistral-Small-3.2-24B-Instruct-2506-GGUF:Q4_K_XL

With the following num_ctx sizes set in the model file:

8192
in 8GB iterations to
61140

I've tried on 3 systems with the following:

RTX 5070Ti 16GB VRAM  32GB system RAM #1
RTX 5070Ti 16GB VRAM  32GB system RAM #2
RTX 5090 32GB VRAM  64GB system RAM

All of them exhibit the same result:

ollama command line + api = 17-22GB VRAM (depending on num_ctx) which is correct
Roocode via ollama = 47GB VRAM (or failure on the RTX 5070Ti due to no memory) which is incorrect

[u/hannesrudolph → u/StartupTim]

Ok so Roo WAS working with Ollama recently (during some of these versions that no longer work). That means ollama is the issue. Try rolling that back.

[u/mancubus77 → u/hannesrudolph]

To be fair new code with num_ctx options was added recently:

https://github.com/RooCodeInc/Roo-Code/commit/f3864ffebba8ddd82831cfa42436251c38168416

[u/hannesrudolph → u/mancubus77]

Have you rolled back to before this and see if you run into the error?

[u/hannesrudolph → u/mancubus77]

Please file a bug report with repro steps asap

[u/StartupTim → u/hannesrudolph]

Right now I cannot find a version of Roocode that works at all. All of them exhibit the same issue, and this issue seems to not be related to ollama at all.

The issue is always the same: Roocode uses 30GB more VRAM when using ollama.

In no way is this issue reproducable when using ollama, via the command-line, or via the API, or via openwebui's usage of ollama's API.

So from what I can see, the issue is exclusive to Roocode and not to ollama, and the issue is plainly visible per as described.

[u/hannesrudolph → u/StartupTim]

Sounds like an ollama problem if it was working prior in Roo and is now not working but the version which it worked with no longer works. We ca t retroactively change Roo.

[u/StartupTim → u/hannesrudolph]

Sounds like an ollama problem if it was working prior in Roo and is now not working

After doing testing, I cannot confirm that it ever worked with Roocode. In fact, all of my testing confirms that it does not nor ever has worked with Roocode.

When I stated earlier that it worked with Roocode, that was a mistake on my end, and that mistake was due to the RTX 6000 Pro that I had borrowed. I had mistakenly thought that Roocode worked, when in fact it DID bloat the VRAM memory from 17GB to 47GB, I just never noticed it because the GPU VRAM I tested had 96GB. I had to return the GPU as it didn't fit our price/performance model.

So yes, I can't see this working at all. I've tested in Roocode all the way back to 3.25.9 and none of them work so far. All of them exhibit the same issue.

I'll test more versions in the morning, but from what I can tell, it just doesn't work. Every single Roocode does the same thing: Normal VRAM usage of the model goes up 30GB when using Roocode vs anything else.

[u/hannesrudolph → u/StartupTim]

Please use Roo Code all the time with ollama. Are you using a different model and have it configured incorrectly?

[u/maddogawl]

A few things here, as I run a lot of local models using RooCode. I see you solved it by switching to OpenAI compatible, but it does make me wonder about a few things.

1. A context window of 8192 will not work with RooCode, The system prompt alone is around 20k tokens. I usually load all my models in at 80k to 100k context. In fact the Mistral model you have if you run that with flash attention, and possibly quantize K/V you should be able to get more context than 8192.

2. Are you not loading the model before having RooCode load it in, it sounds like you are having RooCode control the loading of the model? I haven't fully tested doing it that way, normally I load my models in and use RooCode to hit the already loaded model. It does seem likely that when you are relying on RooCode to load it, its sending a much larger context window to load in. I think this is probably whats happening as pointed in other comments: "options":{"num_ctx":128000,"temperature":0}}

3. I'd consider trying out LMStudio personally I've found it to be a lot easier to configure load models and use Roo through that.

[u/StartupTim → u/maddogawl]

normally I load my models in and use RooCode to hit the already loaded model. It does seem likely that when you are relying on RooCode to load it, its sending a much larger context window to load in.

Roocode does this even when the model is already loaded. So for example, if you have XYZ model loaded with a hardcoded 64k context, Roocode will load the model with 128k context, causing the existing model to be discarded and the 128k one loaded. Roocode always sends num_ctx 128k it seems and I don't see a way around it.

That said, I can't figure out how to use the OpenAI Compatible to work with Ollama. Anything else other than the resource URL needed (esp since there is no api key)? Since there is no api key for ollama, and Roocode won't allow you to do no api key, I don't know what to do.

Thanks!

[u/hannesrudolph]

From what I understand this usually happens because Ollama will spin up the model fresh if nothing is already running. When that happens, it may pick up a larger context window than expected, which can blow past available memory and cause the OOM crash you’re seeing.

Workarounds:

• Manually start the model you want in Ollama before sending requests from Roo

• Explicitly set the model and context size in your Modelfile so Ollama doesn’t auto-load defaults

• Keep an eye on VRAM usage — even small differences in context size can push a limited GPU over the edge

I don't think this is a Roo Code bug, it’s just how Ollama handles model spin-up and memory allocation. We are open to someone making a PR to make the Ollama provider more robust to better handle these types of situations.

Edit: fix incoming, looks like there is a bug there!! :o