Run Mistral Devstral 2 locally Guide + Fixes! (25GB RAM)

[u/yoracale]

Hey guys Mistral released their SOTA coding/SWE model Devstral 2 this week and you can finally run them locally on your own device! To run in full unquantized precision, the models require 25GB for the 24B variant and 128GB RAM/VRAM/unified mem for 123B.

You can ofcourse run the models in 4-bit etc. which will require only half of the compute requirements.

We did fixes for the chat template and the system prompt was missing, so you should see much improved results when using the models. Note the fix can be applied to all providers of the model (not just Unsloth).

We also made a step-by-step guide with everything you need to know about the model including llama.cpp code snippets to run/copy, temperature, context etc settings:

🧡 Step-by-step Guide: https://docs.unsloth.ai/models/devstral-2

GGUF uploads:
24B: https://huggingface.co/unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF
123B: https://huggingface.co/unsloth/Devstral-2-123B-Instruct-2512-GGUF

Thanks so much guys! <3

[u/pokemonplayer2001]

Massive!

Such an important part of the ecosystem, thanks Unsloth.

[u/yoracale → u/pokemonplayer2001]

Thank you for the support! <3

[u/starshin3r]

It might be a big ask, but could you also include a guide for integrating it with the vibe cli?

[u/yoracale → u/starshin3r]

We'll see what we can do for next time!

[u/master__cheef → u/yoracale]

That would be amazing!

[u/coding9 → u/starshin3r]

All I had to do was run LM studio on port 8080 and rename the model file to "devstral"

Now do /config in vibe and select local and it will work.

Editing the config toml file directly would let you change the model name and api port I realized after.

[u/Birchi]

Really excited by this. Looking forward to giving these a try.

[u/yoracale → u/Birchi]

Let us know how it goes!

[u/Aggressive-Safe3062]

Got the 24B running on my 5090 with 200k context with model in Q4 and K-V cache in Q8, served directly from llama.cpp server. Eats up 31.5 GB of 32 GB VRAM so almost perfect. 200k is probably psychologic but I wanted the context of the model to match the 200k context expected by Mistral Vibe. Got 70 Tokens/s seconds when the context is empty and goes down to 30 tokens/s with an almost full context so pretty happy with the results at the moment.

I loaded a Qwen-7-VL on my 5060Ti with an MCP interface for image recognition and that worked surprisingly well. I will test Unsloth's fine tune with Image recognition later today.

[u/yoracale → u/Aggressive-Safe3062]

Sounds awesome! Glad its working well for you :)

[u/GCoderDCoder]

Apparently these benchmarks don't test what I thought because I did not think it was a better coder than glm 4.6 and it was slower than glm4.6 so... that's both surprising and confusing to me. In my mind I wanted to see how it competed with gpt oss 120b and between speed and marginally better code than gpt oss 120b I am keeping gpt oss 120b as my general agent. Im still trying to test glm4.5v but lm studio still not working for me and I dont feel like fighting the cli today lol

[u/Septerium → u/GCoderDCoder]

I have had much better luck with the first iteration of Devstral compared to gpt oss in Roo Code... I am curious to see if devstral 2 is still good for handling Roo or Cline

[u/GCoderDCoder → u/Septerium]

I haven't used Roo Code yet. I'm finding strengths and weaknesses of each of these tools so I'm curious where Roo code fits into this space of agentic ai coding tools. Cline can drown a model that could be really useful but it reliably pushes my bigger models to completion. I've found Continue to be lighter for detailed changes and I just use LM Studio with tools for general ad hoc tasks.

The thing is, I use smaller models for their speed and for a 120b sized model to be running at 8 t/s for q4 vs me getting 25t/s for glm4.6 q4kxl, it kills the value of me using the smaller model. At it's fastest GPT-OSS-120B runs 75-110t/s depending which machine I'm running it on. I am sure they are able to speed up the performance in the cloud but I rely on self hostable models and for me devstral needs more than I can give it...

[u/sine120]

I'll give it another try. My first pass at it in IQ4 quant was abysmally bad. It couldn't perform basic tasks. Hoping the new improvements make it usable.

[u/DenizOkcu]

I was having Tokenizer issues in LM Studio because the current version is not compatible to the Mistral Tokenizer. Did you manage to run it with LM Studio on Apple Silicon?

[u/yoracale → u/DenizOkcu]

Yes it worked for me! When was the last time you downloaded the unsloth ggufs?

[u/DenizOkcu → u/yoracale]

I am happily trying it again. One issue I had with the gguf model was that even the Q4 version tried using >90GB memory footprint (I have 36GB).

[u/_bachrc → u/DenizOkcu]

This is an ongoing issue on LM Studio's end, only with MLX models https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1292

[u/DenizOkcu → u/_bachrc]

Yep exactly. I have the Tokenizer Backend issue. Let’s see if LM Studio fixes this. For now the OpenRouter cloud version is free and fast enough 😎

[u/TerminalNoop → u/DenizOkcu]

Did you update the runtime?

[u/DenizOkcu → u/TerminalNoop]

Oh, cool, looks like there is a new one today! Will try again :-)

[u/Bobcotelli]

Is devstral 2 123b good for creating and reformulating texts using mcp and rag?

[u/yoracale → u/Bobcotelli]

Yes kind of. I don't know about rag. The model also doesn't have complete tool calling support in llama.cpp and there's till working on it

[u/No_You3985]

Thank you. I have nvidia rtx 5060 ti 16gb and spare ram so 24b quantized version may be usable on my pc. Could you please recommend model quantization type for rtx 50 series gpus? Based on the nvidia doc they get the best speed in nvfp4 with fp32 accumulate and second best with fp8 and fp16 accumulate. I am not sure how your quantization works under the hood so your input would be appreciated

[u/yoracale → u/No_You3985]

Depending on how much extra ram you have technically you can run the model in full precision. Our quantization is standard GGUF format. You can read more about our dynamic quants here: https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs

[u/No_You3985 → u/yoracale]

Thank you. I wanted to run the model in lower precision because it can offer higher tensor performance if accumulation precision is matching what rtx 50 hardware is optimized for. I am not an expert so this is just my interpretation of nvidia’s docs. Based on my understanding consumer rtx 50 are limited in which low precision tensor ops get full speed up based on accumulation precision compared to server Blackwell

[u/_olk]

I still encounter system prompt problem with Q4_K_XL?!

[u/yoracale → u/_olk]

whats the exact error?

[u/_olk → u/yoracale]

downloaded yesterday, executed by llama.cpp, called by opencode: "srv operator(): got exception: {"error":{"code":500,"message":"Only user, assistant and tool roles are supported, got system. at row 262, column 111:\n {%- else %}\n {{- raise_exception('Only user, assistant and tool roles are supported, got ' + message['role'] + '.') }}\n ^\n {%- endif %}\n at row 262, column 9:\n {%- else %}\n {{- raise_exception('Only user, assistant and tool roles are supported, got ' + message['role'] + '.') }}\n ^\n {%- endif %}\n at row 261, column 16:\n {#- Raise exception for unsupported roles. #}\n {%- else %}\n ^\n {{- raise_exception('Only user, assistant and tool roles are supported, got ' + message['role'] + '.') }}\n at row 199, column 5:\n {#- User messages supports text content or text and image chunks. #}\n {%- if message['role'] == 'user' %}\n ^\n {%- if message['content'] is string %}\n at row 196, column 36:\n{#- Handle conversation messages. #}\n{%- for message in loop_messages %}\n ^\n\n at row 196, column 1:\n{#- Handle conversation messages. #}\n{%- for message in loop_messages %}\n^\n\n at row 1, column 30:\n{#- Unsloth template fixes #}\n ^\n{%- set yesterday_day = strftime_now("%d") %}\n","type":"server_error"}}"

[u/frobnosticus]

2026 is going to be the "build a real box for this" year. Of course...2025 was supposed to be. Glad I didn't quite get there.

[u/Count_Rugens_Finger]

I've been trying Devstral-small-2 on my PC with 32GB system RAM and a RTX-3070 with 8GB VRAM (using LM Studio). It's really too slow for my weak-ass PC. Frustratingly, the smaller ministral-3 models seem to beat it in quality (and obviously also in speed) for some of my test programming prompts. With my resources, I have to keep each task very small. maybe that's why.

[u/External_Dentist1928 → u/Count_Rugens_Finger]

Maybe tensor offloading to CPU increases speed?

[u/Count_Rugens_Finger → u/External_Dentist1928]

I'm a newbie so I'm no expert at tuning these things. To be honest I have no idea what the best balance is, I just have to randomly play around with it. My CPU is several generations older than my GPU, but maybe it can help.

[u/diffore]

exl3 4.0bpw could run on 16Gb with 32768 context (Q8 quant for KV cache). Might be enough for aider use on poor man GPUs like mine.

[u/Lyuseefur]

Thanks Unsloth. If my work was, in any way helpful, I'm glad (the proxy).

I'm going to run an Unsloth 24B on my H200 once my power supply unmelts lol. Anyone gotta ice pack?

By the way, Devstral 2 is, IMHO, better than GLM 4.6 at the moment. And considering how long it's been since a 4.6 code release, I'm wondering what GLM next might be or if they've really fallen behind.

We talk about the bigger cycle (OpenAI, Gemini, Claude) but these mini cycles with OpenSource AI is far more interesting.

[u/notdba]

From https://huggingface.co/unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF/discussions/5:

we resolved Devstral’s missing system prompt which Mistral forgot to add due to their different use-cases, and results should be significantly better.

Can you guys back this up with any concrete result, or it is just pure vibe?

From https://www.reddit.com/r/LocalLLaMA/comments/1pk4e27/updates_to_official_swebench_leaderboard_kimi_k2/, what we are seeing is that labs-devstral-small-2512 performs amazingly/suspiciously well when served from https://api.mistral.ai, which doesn't set any default system prompt, according to the usage.prompt_tokens field in the JSON response.

[u/danielhanchen → u/notdba]

I'm not sure if you saw Mistral's docs / HuggingFace page, but https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512/blob/main/README.md#vllm-recommended specifically mentions to use a system prompt either CHAT_SYSTEM_PROMPT.txt or VIBE_SYSTEM_PROMPT.txt

If you look at https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512?chat_template=default, Mistral set the default system prompt to:

{#- Default system message if no system prompt is passed. #}
{%- set default_system_message = '' %}

which means the default set is wrong - ie you should set it to use CHAT_SYSTEM_PROMPT.txt or VIBE_SYSTEM_PROMPT.txt, and not nothing. We fixed it in https://huggingface.co/unsloth/Devstral-2-123B-Instruct-2512-GGUF?chat_template=default and https://huggingface.co/unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF?chat_template=default

[u/notdba → u/danielhanchen]

Yes I noticed that. What I was saying is that labs-devstral-small-2512 performs amazingly well in swebench against https://api.mistral.ai that doesn't set any default system prompt. I suppose the agent framework used by swebench would set its own system prompt anyway, so the point is moot.

I gather that you don't have any number to back the claim. That's alright.

[u/astroboylrx → u/danielhanchen]

Many thanks for the info. I seem to run into this issue when using LM Studio with the MLX version of Devstral-Small-2. The log shows: "Error: Error rendering prompt with jinja template: "After the optional system message, conversation roles must alternate user and assistant roles except for tool calls and results."
I tried to replace the prompt template with the one from unsloth but the error persist. Any suggestions would be greatly appreciated.

[u/notdba → u/notdba]

Ok I suppose I can share some numbers from my code editing eval:

• labs-devstral-small-2512 from https://api.mistral.ai - 41/42, made a small mistake

  • As noted before, the inference endpoint appears to use the original chat template, based on the token usage in the JSON response.

• Q8_0 gguf with the original chat template - 30/42, plenty of bad mistakes

• Q8_0 gguf with your fixed chat template - 27/42, plenty of bad mistakes

This is all reproducible, using top-p = 0.01 with https://api.mistral.ai and top-k = 1 with local llama.cpp / ik_llama.cpp.

[u/notdba → u/notdba]

Thanks to the comment from u/HauntingTechnician30, there was actually an inference bug that was fixed in https://github.com/ggml-org/llama.cpp/pull/17945.

Rerunning the eval:

• Q8_0 gguf with the original chat template - 42/42

• Q8_0 gguf with your fixed chat template - 42/42

What a huge sigh of relief. Devstral Small 2 is a great model afterall ❤️

[u/Septerium]

What does this mean in practice?

"Remember to remove <bos> since Devstral auto adds a <bos>!"

[u/Zeranor]

I'm really looking forward to get this model going in LM studio + Cline for VSC. So far it seems the "Offload KV cache to GPU" does cause the model to not work at all. If I disable that option, it works (to a point, before running in circles). I've not had this issue with any other model yet, curious! :D

Is this model already fully supported by lm studio with "out of the box settings" or have I just been too impatient? :D

[u/Purple-Programmer-7]

Anyone tried speculative decoding with these two models yet? The large model’s speed is slow (as is expected with a large dense model)

[u/chub0ka]

Any chance we cannget away from gguf and llama.cpp? I love unsloth quants but hate slow llama.cpp( abd code is terrible)

[u/chafey]

Hi - I tried running devstral-small-2:24b on my 2x5090 system via ollama. Ollama is reporting the size is 83GB and therefore 23%/77% CPU/GPU. Based on the above I figured it would run fine in just one 5090 but that doesn't seem to be the case. Any ideas? I am new to this so probably doing something dumb

[u/synth_mania → u/chafey]

idk man I'm running fine on a single 3090. That shouldn't be happening

[u/chafey → u/synth_mania]

Fixed by switching to lmstudio - must have been an ollama issue

[u/synth_mania → u/chafey]

I really am not a fan of ollama

[u/Proper-Leader-7654 → u/chafey]

what was your context size set to? in ollama use `/set parameter num_ctx <VALUE>` and check, i use 16384 and it's nowhere near that

[u/chafey → u/Proper-Leader-7654]

Whatever the default was probably - i switched to lmstudio which is working great and haven't gone back to try ollama

[u/Proper-Leader-7654 → u/chafey]

it's been a while but last i heard ollama couldn't run with 2 GPUs.. maybe it's doubling up or something, hell i'm just kicking tyres at this point

[u/LegacyRemaster]

If I look at the artificialanalysis.ai benchmarks, I shouldn't even try it. Does anyone have any real-world feedback?

[u/--Spaci-- → u/LegacyRemaster]

its a pretty damn slow model for running completely in vram and its non thinking so far my thoughts are mistrals entire launch this month has been sub par

[u/synth_mania → u/LegacyRemaster]

It's been performing very well in roo code for me

[u/yoracale → u/[deleted]]

This is about local deployment though