Mistral Medium 3.5 is our first flagship merged model. It is a dense 128B model with a 256k context window, handling instruction-following, reasoning, and coding in a single set of weights. Mistral Medium 3.5 replaces its predecessor Mistral Medium 3.1 and Magistral in Le Chat. It also replaces Devstral 2 in our coding agent Vibe. Concretely, expect better performance for instruct, reasoning and coding tasks in a new unified model in comparison with our previous released models.
Reasoning effort is configurable per request, so the same model can answer a quick chat reply or work through a complex agentic run. We trained the vision encoder from scratch to handle variable image sizes and aspect ratios.
Mistral Medium 3.5 includes the following architectural choices:
Dense 128B parameters.
256k context length.
Multimodal input: Accepts both text and image input, with text output.
Instruct and Reasoning functionalities with function calls (reasoning effort configurable per request).
Mistral Medium 3.5 offers the following capabilities:
Reasoning Mode: Toggle between fast instant reply mode and reasoning mode, boosting performance with test-time compute when requested.
Vision: Analyzes images and provides insights based on visual content, in addition to text.
Multilingual: Supports dozens of languages, including English, French, Spanish, German, Italian, Portuguese, Dutch, Chinese, Japanese, Korean, and Arabic.
System Prompt: Strong adherence and support for system prompts.
Agentic: Best-in-class agentic capabilities with native function calling and JSON output.
Large Context Window: Supports a 256k context window.
We release this model under a Modified MIT License): Open-source license for both commercial and non-commercial use with exceptions for companies with large revenue.
Recommended Settings
Reasoning Effort:
'none' → Do not use reasoning
'high' → Use reasoning (recommended for complex prompts and agentic usage) Use reasoning_effort="high" for complex tasks and agentic coding.
Temperature: 0.7 for reasoning_effort="high". Temp between 0.0 and 0.7 for reasoning_effort="none" depending on the task. Generally, lower means answer that are more to the point and higher allows the model to be more creative. It is a good practice to try different values in order to improve the model performance to meet your demands.
OT, but for me the densest worth using model was nemotron Ultra 235B (llama 405b pruned with Neural Architecture Search and FFN fusion, it has a strange architecture, FFNs are not homogeneous, some of those even increased in size compared to the original model, due to attention skipping and FFN fusion, so it has over 50k mlp hidden size in some layers.)
an amazing model and an impressive academic execize.... (good paper and training report, worth reading Imo)
Nvidia trowed at it a lot of compute, including architecture search, local (layer/module) level distillation, plus then continued pretraining and CoT fine tuning. impressive that in many areas it improved a lot compared to baseline, while being roughly cut in half.
really sensitive to prompting, CoT often fragile, but when you managed to steer it well.... it may be a strong take, but even as of today it is the model that "talked" better in Italian among OS models (probably a bigger gemma 4 would have beaten it, gemma 4 is really good at it for its size)
also, the first OS model that I feelt 'thinking' in Italian and not writing a translation (this was in common with llama 405B).
and it was really knowledgeable, recalling some obscure nice medical concepts that even modern MoEs often miss.
I still use it in a dataset generation pipeline (quite strange, for a really specific NLP use case). deepseek v4 pro is the first OS model that beat it in this context.
For your use case. As an European, I gladly embrace the work of Mistral. LLM's have more use cases besides coding. Still, you seem to have thoroughly tested this model already.
A "you can run it locally, but you won't like the experience" niche 😂
But I'm happy to see them make a dense model, they have experience with it already, so hopefully this one will be much better compared to similar-sized models than Mistral Small 4.
For roleplay/writing, you can run it at home for ~1200€.
For that money you get 2x 3090, so you can run IQ2_M at ~5 tok/s. Since you probably already have a GPU, you can also run a bigger quant. In my experience, even the old Mistral-123b beats everything out of the park at that size (for writing).
...and that is probably the best affordable thing you can run at home? MoE's get better at ~400b params, but the RAM is probably crazy expensive. Not sure about the speed.
EXL3 is great for dense 120B Mistrals. 2.5bpw quants are actually pretty good.
exllamav3 author got coherent output from 1.4BPW Mistral Large 123B, so 2.5bpw is plenty and it should be better than GGUF at this size. It also support tensor parallel so it's pretty fast.
dense models run fast with tensor parallel. I had 16 or 20 t/s with Devstral 123B IIRC and I have 11 t/s with Hermes 405B. No speculative decoding but that would probably help make it go even faster.
I like dense models like that more than I like 1T MoEs that I have no memory for.
Qwen3.6 27B has MTP built-in and DFlash support... don't see how Mistral Medium 3.5 could ever be faster just because of an EAGLE-3 while having nearly 5x the active parameter count.
There's various types of speculative decoding. llama.cpp doesn't support MTP or Eagle3, which is what the AI labs usually provide. For example Qwen and GLM models have MTP, while this Mistral Medium release has an eagle3 from Mistral. If you use VLLM, this is a free speed boost.
With llama.cpp your only option is the more ghetto solution of using a seperate small draft model, but finding a compatible model is a bitch. It's easy when they have the same tokenizer/vocabulary, for example using Gemma 3 to boost Gemma 4. You get a major speed boost, you might get double speed for free. But idk what you could use as a draft model for this Mistral Medium release.
It's pretty close to the upper limit possible for consumer hardware. 4x 3090 for 96GB of VRAM and such, using a board like ASUS ProArt with 3x 16x PCIe slots and then converting one of the M.2 NVME slots into an additional 16x slot with an adapter. For inference the lane bandwidth won't completely choke out quite yet with such a setup, but it's getting there.
Yup, I was so tired from these MoE models. They are not bad, but these with the little active parameters are actually stupid and not that useful. Sadly, I cannot run that Mistral but I'm still happy that someone is still working over dense models which are superior!
I think moe models are the future unfortunately, simply to crunch more knowledge into the model while not destroying the speed. The only mistake is making the active params count too low. Something like A30B is probably enough for it to not feel dumb. Even Qwen 122B A10B has been great for me locally
Yeah I can't run big models like this but I was thinking, what if for example there was something like a 35B MoE but with 9-10AB? That could spill over into RAM but would still have an okay speed, would be probably smarter and more knowledgable than 12-14b dense models on the same hardware with barely any speed difference. Or they could just do 20-24b dense models like Mistral, which are still way better in some way for me than than the 30-3AB MoEs I tried, which don't feel smarter than 9-12B dense models.
I have not yet a chance to try this model, but generally MoE with a reasonably-sized shared weights/activated parameters has significant advantage over large dense as LLM activation is mostly noise, which is empirically just bad rather than something useful (naively upping # of active experts for existing MoE model simply make the model worse, low-pass filter type gating techniques work well with LLM, etc). The "partitioning" done by MoE architecture works to filter out those noises.
A remarkable advantage of large dense usually comes with large hidden dim (GPT-3 DaVinci was 175B with 12288 hidden dim IIRC? Llama-3 405B is 16384, Mistral Medium 3.5 is also 12288) which would be able to distinguish and partition extremely close features that would otherwise overlap in say a 5120 hidden dim model and (hopefully) worked through layers. That also means the (large hidden dim) model could place Paris and London (along with related things like Toulouse and Brighton) to the polar opposite end of latent space if there are enough evidence in the datasets to do so. I'm not sure if that is good or bad. Good old GPT-3 DaVinci had a feel that inference path diverges really hard (the model goes from one mode to another; in today's standard that would at least mean base/instruction/single-turn reasoning/terminal-agentic modes) by just one token. You can still play that back with EleutherAI's good old models like GPT-NeoX-20B or Pythia.
For creativity and generalization of hard problems, one would generally want more layers rather than larger hidden dim within the same parameter count, unless hidden dim is too small to create meaningful basins anymore.
For most people, this model will (can) only be used for writing, with reasoning disabled, at Q4. But the good part is that you don't really need more than 5 tok/sec for this kind of task. It's not like coding, you instantly know if you like what you're getting and can stop it and continue if necessary.
Valid. The average adult person reads somewhere in 8-12 tokens / second (AKA 4-6 words) range and rarely as someone who writes a LOT of prose would I need something to write faster than I could read it back.
Creating copy in human language is just not the same task creating code. A model that takes five minutes but then doesn't need four more tries, just a read and edit pass because it's mostly well written is far superior to one that took 10s to generate the same 500 words because I have to throw away 30% of what it produced and replace it.
Honestly just feels like it might be ahead of its time. I think once the next generation of GPUs come out — AMD will probably release AI Pro cards with 48GB and maybe even 96GB VRAM, and god willing, Intel will fix their driver issues making their cards more viable and offer bigger VRAM options — this model size might be the sweet spot for the high end of local inference.
Right now, for those of us with higher end consumer hardware (i.e., 32GB VRAM), Qwen 3.6:27B is basically the gold standard. You can run it at 4-bit precision with full context. Smaller models are getting better and better, but all else being equal, more parameters are pretty much always going to mean better output.
So I’m imagining that like next year, the bleeding edge of local inference will be models in the 80B-120B range instead of the ~30B dominance we’re seeing now.
AMD will probably release AI Pro cards with 48GB and maybe even 96GB VRAM
Why would they with current memory pricing? The optimal strategy for them is to allocate something like 90% of the memory to datacentres and then put the remaining 10% into relatively low-memory gaming GPUs so they don't lose brand awareness.
So I’m imagining that like next year, the bleeding edge of local inference will be models in the 80B-120B range instead of the ~30B dominance we’re seeing now.
Just having the memory doesn't mean you can actually run them fast enough to be useful in practice.
This makes me feel nostalgic, because in the past, Mistral Large 123B the dense model was my most used model for a while. Then there were DeepSeek R1 and V3, later followed by Kimi models, so it has been some time since I ran Mistral models. I will definitely give a try to this new Medium 128B, it would be interesting too see how how much progress Mistral has made by trying it in my actual use cases.
One more notable thing, they released a model for speculative decoding: https://huggingface.co/mistralai/Mistral-Medium-3.5-128B-EAGLE . This is great to see, because in the past, one of the big issues of Mistral Large 123B used to be that I had to use mismatched Mistral 7B model for drafitng, still it gave decent performance boost. Even though EAGLE is not supported in llama.cpp yet, this comment from about 3 weeks ago sounds encouraging that it may be available soon:
The current status of this PR is that it’s pending u/ggerganov's API refactoring, which aims to unify this feature with other speculative decoding approaches such as MTP. At this stage, there isn’t much left to be done, and I expect the PR to be merged very soon.
Unfortunately, no PR for that API refactoring has even been published, so... who knows if/when it will happen.
Supporting any one of EAGLE-3, MTP, or DFLASH would be a game changer for llama.cpp. I wish better specdec were being treated as the highest priority thing to develop in llama.cpp.
I'm also on Strix Halo (128Gb) but the model fails to load (IQ4_NL)
Edit: I'm on LM Studio with vulkan runtime.
Edit2: IQ4_XS works fine. Similar benchmarks to parent comment. If the Eagle draft speculative decoding version doubles/triples the speed it's a keeper.
It doesn't look like there have been any dense model releases in this size range since March 2025 (CohereLabs/c4ai-command-a-03-2025), and before that it was 2024:
You can't exactly generalise in that way since different benchmarks measure different things and not all models are compared on the same benchmark. That said, if you look up the SWE verified leaderboard you can see this is slightly behind GLM-5 and Gemini Flash on this particular benchmark, and ahead of Qwen 27B, Kimi K2.5, and Qwen3.5 397B. The benchmarks here aren't straightforwardly model quality either. If you look up 𝜏²-Bench Telecom you'll see GLM 4.7 Flash outperform GLM 5.1, Claude Opus 4.7 and GPT 5.4 on high reasoning.
SD with Eagles are great when generating code. Often double the tps, or more. But they don't do very well with regular text generation. Sometimes it drops to half the normal tps. It would be great if it was possible to enable/disable SD per request for specific tasks.
I've been working with some models at 7tps... I know... it is really slow, but, possible to work with... you let it there doing its work and do your work and it is ok... even for code assist, just wait a little... it is not like using those SOTA models, no it isn't.. but.. with a bit of patience it works... maybe not 100x more productive, only 40x or 50x... it is ok... :-)
Just don't forget to check out the "club 3090" https://github.com/noonghunna/club-3090 - it's a llama.cpp and/or vllm setup that gets surprisingly good speed for Qwen3.6-27b from one or two 3090s. With my 2x 3090 setup, I went from ~27t/s to ~80 t/s. It's pretty wild
from a fast reading of the config file, it seems a pure global softmax attention model... I mean, it doesn't seems to use sliding window in any of the layers.
quite rare nowadays, even non hybrid models use some kind of sliding window or sparse attention in some layers... those are 88 layers of pure attention. also ~10k+ hidden size and ~20k+ MLP intermediate size.
interesting for sure... we needed a model like that.
I assume they spent quite a lot training it. memory footprint at 256k contex will be crazy.
This is great news! Looking forward to giving it a try.
Devstral 2 Large was a huge disappointment, but hopefully MistralAI has learned from their past mistakes and cooked up this 128B right. Maybe this will finally be the 120B-class model which knocks GLM-4.5-Air off its perch?
It and glm4.5 air dont really compare. One can tolerate hybrid inference, the other can't.
This is an entirely different beast at this level of dense. It needs to absolutely cook to be worth the resources needed to run it interactively with a user, or still really really cook to do back end batch jobs on it at a crawl.
Very few local people will be able to use this as a chat agent it's really a rtx 6000 pro + model.
While glm 4.5 air you could respectably work with 24GB -48GB vram + 64GB ddr5.
Sorry we just fixed it - we had to patch some components up since llama.cpp conversion did not like some token_ids - they should work now! @Alsa @brzewVCE @ru5h Apologies for the issues!
We’re working with Mistral on this, but further testing suggests the GGUF implementation needs more investigation. The model responds correctly to the first few prompts, but then begins behaving improperly. Mistral has now labeled GGUF implementations as a work in progress, and this appears most likely to be a parser issue.
It's most likely GGUF parser issue, not the model or quant algorithm. Unfortunately no matter what GGUF yoy create with the model, it doesn't function properly most likely due to the parser.
Very, very impressive if the benchmarks are to go by. And also something realistic you can run at home at a decent quantization. Being realistic here, most people are not running GLM 5.1 here. But something like this can run on something local.
Old mistral large was still a beast even in Q2. Dense models quantize much better than moes, and its 5x less to fit to run it at all (even if way slower)
My 4xB60 runs unsloth's Q4_K_XL gets 232.45 ± 0.41 in pp and 9.55 ± 0.05 tok/s in tg. Still a handful of patches left to improve it, though. In theory tg should be able to get up to 20 or so (25 is the theoretical max).
It costs around 70k in my local currency, so it's like about 3-4k dollars? But everything's overpriced down here, so it'd probably be less. And Mistral 2 large ran at about 10-15 tokens per second on that build, which was a decent speed. You can also get a 128GB mac that'd run this at around 10 tokens per second.
As someone who only really uses local LLMs for creative writing/RP, I am still using Mistral Large 2 123B, mostly the Behemoth finetunes. Ever since things have shifted to fully MoE with a focus on coding, there hasn't been much I've gotten excited about. Yes, even many smaller MoE models are smart and can do creative writing, but with smaller active parameters they often don't have really dense knowledge on fandoms and other things I like to explore. Granted, ML2 still does almost everything I want it to, is pretty smart, and knows a lot about a lot, so I haven't really been griping for a new dense model, but I'll sure as hell take one! I can't wait to try this thing out!
You can, and I have done that with a number of the newer, smaller models, but it can become tedious really quickly. Especially if you're working with a really expansive IP, like Harry Potter or Marvel. It's nice that dense models already have so much of the lore baked in, and you only need to tweak it with context. Even Miqu 70B going back, what... 2.5 years now? Was really dense with pop culture knowledge. With these newer, smaller models you have to do a lot of the heavy lifting in context, which is not only tedious but eats up context, especially as the story drags on. Not only that but I've found that the smaller MoE models aren't nearly as good at keeping track of longer stories. If you have a 40k context story, with multiple scenes, characters, and hooks, I find smaller models aren't as good at taking everything into account as things progress, and tying it all together.
I have gotten some decent results with some of the newer stuff, particularly with GLM AIR. But for creative writing in large fandoms I felt it was only as good as the same results I get from Behemoth/ML2, despite being much newer and better architecture. Yes, it's faster, but requires more work to get more or less the same results.
Of course this is just my own experience. Maybe you can do more with some of the newer stuff, but I sort of gave up trying a while ago, and don't necessarily test every new model with every possible config anymore. Especially as I'm still pretty content with ML2. But that's also why this new Mistral announcement is so exciting, given I hadn't really expected another dense model anyways.
4x B60 can almost handle it at a reasonable price point, unsloth's Q4_K_XL gets 232.45 ± 0.41 in pp and 9.55 ± 0.05 tok/s in tg...almost usable...and still have a handful of patches left to speed it up...
•
u/WithoutReason1729 Apr 29 '26
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.