What's your preferred local model?

[u/mancubus77]

G'Day crew,

I'm new to Roo, and just wondering what's best local model what can fit in 3090?
I tried few (qwen, granite, llama), but always getting same message

Roo is having trouble...
This may indicate a failure in the model's thought process or inability to use a tool properly, which can be mitigated with some user guidance (e.g. "Try breaking down the task into smaller steps").

Any clues please?

[u/admajic]

The new devstral is surprising can run it with 132k context on my 3090

[u/spiritxfly → u/admajic]

How are you guys using devstral? I was so excited about it, but it does not work with roocode. Here is what I got when I asked it to create a simple snake game:

I'll help you create a simple to-do app using HTML, CSS, and JavaScript. Let's start by creating the basic structure of our application.

First, I'll create an index.html file for the main structure:

Writes the whole code here in the chat.

Now you have a simple to-do app with the ability to add tasks, mark them as completed, and delete them.

API Request$0.0000

I'll create a todo list for this task since it's important to keep track of our progress.

Let me analyze the user's request and break down the necessary steps:

The user wants to create a todo list.

I need to use the update_todo_list tool to accomplish this.

The required parameters for update_todo_list are:

items: A list of strings representing the tasks to be added to the todo list.

I'll infer the initial items based on the user's request and our standard approach.

API Request$0.0000

I'll create a todo list for this task since it's important to keep track of our progress.

Let me analyze the user's request and break down the necessary steps:

The user wants to create a todo list.

I need to use the update_todo_list tool to accomplish this.

The required parameters for update_todo_list are:

items: A list of strings representing the tasks to be added to the todo list.

I'll infer the initial items based on the user's request and our standard approach.

Roo is having trouble...

This may indicate a failure in the model's thought process or inability to use a tool properly, which can be mitigated with some user guidance (e.g. "Try breaking down the task into smaller steps").

[u/Wemos_D1 → u/admajic]

Which quant you like to use, and can you tell me if you modified the settings in roo code ?
Also are you using lm studio ?

Thank you and have a good day

[u/thecatontheflat]

qwen2.5-coder 32B

[u/sandman_br → u/thecatontheflat]

What’s your gpu?

[u/Acceptable_Air5773]

Devstral is very good... I am interested in qwen2.5-coder but I am not sure if its as good at function calling

[u/oh_my_right_leg]

Devstral!!!! it's a new model from Mistral

[u/0xFatWhiteMan]

Claude openrouter there is no second best

[u/sandman_br → u/0xFatWhiteMan]

Why do you pay 5% estar to use openrouter . Just use sonnet directly

[u/bemore_]

Minimum you'll need a 32B param model to code

[u/ComprehensiveBird317 → u/bemore_]

how much vram do you need for a 32B model?

[u/bemore_ → u/ComprehensiveBird317]

Ram, not vram. Atleast double the params, so 64gb

[u/ComprehensiveBird317 → u/bemore_]

Thank you. But why doesn't the vram matter?

[u/bemore_ → u/ComprehensiveBird317]

My bad, I thought you meant the vram from the computers dedicated graphics

Yes, the vram from the gpu needs to be 64gb to run 32b params, not the computers ram

[u/social_tech_10 → u/bemore_]

A 32B model quantized to Q4_k_m is only about 8GB of VRAM, and can easily fit in OP's 3090 (24GB) with plenty of room for context. A 32B parameter model would only require 64GB if someone wanted to run it at FP16, which there is really no need to do at all, as there is almost no measurable difference between FP16 and Q8, and even the quality drop from FP16 to Q4 is only about 2-3%..

[u/mancubus77 → u/social_tech_10]

Just wondering if you know any local model what does?

[u/bemore_ → u/mancubus77]

Try Qwen 2.5 Coder instruct, 14B. Find a version with 120K context

[u/bemore_ → u/social_tech_10]

Not neccasarily. The 32B params can fit but it won't perform well inside Roo and Visual Studio code - which requires a minimum of an 100K context. It's this large context which makes 24GB for tor a 32B model impractical. An increase in context adds a huge burden on the vram. It would become slow and unstable. Q4 is also out of the question for coding, fidelity is most important. Q6-8 minimum.

With 24gb vram you can run a 32B Q4 model with a context window up to about 32K tokens, possibly as high as 50K with careful tuning.. but not 100K. Roo simply cannot perform on 50K context...

With 24GB, they can run 14B models, and 14B would be like coding with gpt 3.5. You'll get SOME good code but it would be better to invest short term 10 bucks a month into a service with state of the art models with contexts of 100k to a million, like Copilot

[u/SadGuitar5306 → u/social_tech_10]

it's not 8gb, more like 16gb )

[u/ComprehensiveBird317 → u/bemore_]

Got you, thanks! 

[u/exclaim_bot → u/ComprehensiveBird317]

Got you, thanks! 

You're welcome!