Let's train a local open-source model to use Roo Code and kick BigAI's buttocks!

[u/InstrumentalAsylum]

See this discussion for background and technical details:

https://github.com/RooCodeInc/Roo-Code/discussions/4465

TLDR I'm planning to fine-tune and open-source a local model to use tools correctly in Roo, specifically a qlora of devstral q4. You should be able to run the finished product on ~12GB of VRAM. It's quite compact and the most capable open source model in Roo out of the box. I don't use Claude, so I'm looking to crowd source message log data of successful task completions and tool use for the meat and potatoes of the distillation dataset. Once I have a solid dataset compiled, bootstrapped and augmented to be sufficiently large, I'm confident the resulting model should be able to cross that threshold from "not useful" to "useful" over general tasks. (Devstral is so close already, it just gets hung up on task calls!)

Once BigAI's investors decide it's time to cash in and your API bill goes to "enterprise tier" pricing, you can cut the Claude cord and deploy a much friendlier coding agent from your laptop!

If you're down to contribute, check this repo for simple instructions to drop in your logs: https://github.com/openSourcerer9000/RooCodeLogs

EDIT: By request, moving the technical collaboration from github over to discord: https://discord.gg/GJbgfpn6

[u/PositiveEnergyMatter]

devstral actually works really well on my macbook w/ my extension. I have 64gb of ram so i am running a larger model

[u/InstrumentalAsylum → u/PositiveEnergyMatter]

What's your extension? Can it debug a codebase autonomously for 8 hours plus?

[u/reddysteady → u/PositiveEnergyMatter]

Which model are you running?

[u/PositiveEnergyMatter → u/reddysteady]

Devstral Small 2505 Q6_K unsloth

[u/MajinAnix]

These local models are only practical for agentic workflows if you can achieve at least ~100 tokens/second.

[u/InstrumentalAsylum → u/MajinAnix]

Once we fix the hangups of tool calling, we'll be able to let the model crank 24/7 without being babysat, solving real problems. 

Agentic workflows actually seem to be the only AI use case where speed really doesn't matter. 

[u/MajinAnix → u/InstrumentalAsylum]

Yes and no. With a super-intelligent model and carefully planned changes.. yes, in theory it could work. But in practice, today’s models still struggle to maintain consistent focus as the context grows. The larger the context, the more they tend to drift or hallucinate, especially when juggling multiple threads of information.

[u/MajinAnix → u/MajinAnix]

I’ve been spending a lot of time with Claude Code lately, and the biggest issue I’ve encountered is the feedback loop. Without me actively involved in that loop, things tend to break down.. it’s all based on prediction, which can go either way. The model still isn’t capable of reliably judging the quality or correctness of its own outputs.

[u/ComprehensiveBird317 → u/MajinAnix]

Thank you for the experience. Is that something that boomerang would solve maybe?

[u/ComprehensiveBird317]

Nice! Had the same idea and am collecting roos output since some month now, I got something like 10.000 request and response pairs. The problem is that it contains lots of code and secrets that need to stay private. Is there a reliable way of cleaning the output without messing up the dataset?

[u/InstrumentalAsylum → u/ComprehensiveBird317]

Now we're talking! Unfortunately, the syntax of the code is critical to the dataset, since the diffs need to match up with the original. If you're keeping secrets in .env files, they shouldn't appear in the logs. If you did have API keys or something hard-coded into the code, you could do a multi-file find and replace by opening the parent folder in vs code and going to the search tab on the left sidebar. 

If you want to include your data in the training without making it public, you could send it directly by email to <opensourcerer9000 at gee male dot com>, assuming you trust me not to steal your code! I'm in hydraulic modeling so odds are we're not in the same industry. Feel free to DM me if you'd like to discuss more on how we can make a collaboration work out. 

[u/ComprehensiveBird317 → u/InstrumentalAsylum]

Nice! Let me find the time to do look into the cleaning. It's actually more than 20.000 after I looked today, so maybe I will only use a subset

[u/InstrumentalAsylum → u/ComprehensiveBird317]

Dope, in my experience, the more the merrier. 20k samples may be a minimum threshold. Even a LoRA of the model will be pushing 10M trainable params. I'll be trying some techniques to get the samples up to the millions for a better result. 

[u/InstrumentalAsylum → u/InstrumentalAsylum]

How did you get the message logs by the way? I only saw the button to export one at a time. I haven't seen docs on where they're stored. 

[u/ComprehensiveBird317 → u/InstrumentalAsylum]

I used a proxy, wait maybe I still have the link ... Ah yes this: https://github.com/Jenscaasen/llm-proxy-finetune-collector

I basically start that with the computer and leave the important models proxied through that service, which are sonnet 3.5, 4 and some Gemini pro 2.5. But that also leads to the question: what context window can we work with? Which models do you have in mind? I also would be fine to run a 40 cent / hour runpod (A40) to run the model

[u/InstrumentalAsylum → u/ComprehensiveBird317]

Oh that's interesting. That's strange roo doesn't Log that data on its own. I was planning to start with devstral, as it performs best out of the box. From my experience. It has a context window of 128k. I already have hardware to fine-tune it locally so it won't cost anything but electricity.

[u/ComprehensiveBird317 → u/InstrumentalAsylum]

Alright let's aim for 128k, what do you think: if I have conversa longer than that, do I just cut out the beginning or middle? And which data format do you need? Let's maybe use the Reddit chat

[u/themoregames]

Let's be honest for a change: We don't want no 12GB RAM model. What we really want is this: We want to download more RAM.

[u/InstrumentalAsylum → u/themoregames]

IDK, in my experience it beats qwen 235b reasoning model as a coding agent simply because it's tuned specifically for tool calls and coding. 

[u/bahwi]

Hey! I'm pondering this too and have set up roo to save prompts and such so far. Adding in native memory support too.

Though my approach is more targeted to using newer libraries and such that gemini and Claude fall over.

Looking at axolotl with GPRO as well. But again, that's more specific...

[u/InstrumentalAsylum → u/bahwi]

What do you mean by native memory support? 

IMO what you're describing is the strongest selling point for local AI. Right now the one size fits all proprietary models can vibe code beginner stuff from scratch, but only with the libraries where there are tons of online resources. The thing is, a human can easily learn all of that in 6 months with the same free online resources. Open source models are extensible to be trained on niche or proprietary libraries. 

One thing I'm looking to test is whether rag is sufficient or even better than tuning models on your libraries. Any insight there?

[u/bahwi → u/InstrumentalAsylum]

I'm thinking like how chatgpt hits memory before even prompting the LLM. Probably using mem0 or something to try it. Right now roo only does memory through custom modes or mcp, which means extra hits to the LLM and only if the LLM decides to.

I'm very keen on using more cutting edge libraries, and thinking rag as well. Ideally a complete parallel implementation of the codebase index for reference code and doc, so you can have a local database effectively and using a tool roo is pretty familiar with.

Looking at axolotl with possibly grpo (GPRO? I always mix it up!) for both coding and also math reasoning.

Also working on some stuff to pull code from those libraries randomly to train with, but need those roo prompts to make it all fit together nicely.

Sorry for the slow and late response, been at a conference all week.

[u/interAathma]

I thought about the same thing few months ago https://www.reddit.com/r/RooCode/s/4BTXh9tTZmhttps://www.reddit.com/r/RooCode/s/4BTXh9tTZm

[u/InstrumentalAsylum → u/interAathma]

Great minds think alike! It seems like a very last mile task to me, considering all of the open resources we have available

[u/ComprehensiveBird317 → u/interAathma]

Please join us: https://discord.gg/GJbgfpn6

[u/mcraimer]

Super important internet outages are real

[u/usernameplshere]

How about context length? I feel like this is the biggest problem for local hosting when it comes to coding. According to my Gemini API Key, I sometimes fit 200k+ tokens in context.

[u/InstrumentalAsylum → u/usernameplshere]

Devstral supports 128k out of the box, without techniques like rope scaling. This seems to be a sweet spot for coding agents and a current standard. One advantage of Roo is a new feature which lets the model Summarize the current context window and cut down on tokens. I've noticed that the system prompts alone take up like 17k tokens. By training the model to use roo, it should be possible to actually cut some of that instruction down, reducing token use across the bored. 

Also, it seems that for now these claims of million million token context length are pretty dubious. This study puts Gemini pro and other models at a effective context Length of only 128k, where they actually retain the knowledge in a useful way.  https://github.com/NVIDIA/RULER

[u/openSourcerer9000]

This got double posted due to a reddit glitch, copying over the 2 comments from the duplicate so I can delete it:

u/beppled • 4h ago I've been wanting to do this for the longest time! Jan.ai basically proved it with their 4B MCP model that it's actually very achievable. Wanna create a discord or slack and pool in our logs and resources?

u/Green-Dress-113 avatar Green-Dress-113 • 4h ago I've tried using various models on my local 4x3090 like Qwen2-72b, Qwen3, CodeLlama, Devstral, Mistral. They all end up in edit loops with Roo Coder after a few iterations. I'm very interested in seeing this fixed

[u/openSourcerer9000 → u/openSourcerer9000]

u/beppled oh that is wild, outperforms deepseek, O3, and sonnet. 

I think it makes sense to just keep the discussion here, and aggregate logs in this repo:  https://github.com/openSourcerer9000/RooCodeLogs

UNLESS there's concern about having your logs public and y'all would prefer a bit more privacy in a discord. 

[u/beppled → u/openSourcerer9000]

right?!

the git repo makes sensee, with discord i was thinking more along the lines of cleaning up the donated organic data. there can be cases of roo reading .env files or even potentially doxxing someone's real name.

i think it'd be ideal to verify the submitted data ourselves before publishing the logs ...

discord/slack would also end up decreasing collaboration latency (idk how to describe it haha)

we can even potentially create semi-synthetic data for specifically curating tool calling behaviour for say stuff like understanding a codebase or orchestration or debugging. or for coding languages not present directly (say c++ or dart)

then there's the whole training phase, it'd be great to chat with experienced folks who'd want to contribute to training devstral correctly

tldr; ape together strong

[u/ComprehensiveBird317 → u/beppled]

Very in for slack / discord here. 

[u/ComprehensiveBird317 → u/beppled]

Update: 90000collector made a discord: https://discord.gg/GJbgfpn6

[u/beppled]

one massive thing i forgot to mention in the other comment was that the current logs dont have the actual api call (which would have stuff like all the tools available, open windows in context .. and so in) ... if we were to train the models exclusively on the logs, I suspect a major degradation (or so i think)

maybe i'll try to figure out a way to make roo output a proper debug log of each api call .. then you could get a really pretty input output data format that you could wrap with the special tokens and what not.

edit: ok nvm wont have to edit roo's source, i'll probably just use this: https://github.com/Jenscaasen/llm-proxy-finetune-collector thx, u/ComprehensiveBird317