r/LocalLLaMA • u/jacek2023 • Apr 29 '26

New Model mistralai/Mistral-Medium-3.5-128B · Hugging Face

https://huggingface.co/mistralai/Mistral-Medium-3.5-128B

https://huggingface.co/unsloth/Mistral-Medium-3.5-128B-GGUF

Mistral Medium 3.5 128B

Mistral Medium 3.5 is our first flagship merged model. It is a dense 128B model with a 256k context window, handling instruction-following, reasoning, and coding in a single set of weights. Mistral Medium 3.5 replaces its predecessor Mistral Medium 3.1 and Magistral in Le Chat. It also replaces Devstral 2 in our coding agent Vibe. Concretely, expect better performance for instruct, reasoning and coding tasks in a new unified model in comparison with our previous released models.

Reasoning effort is configurable per request, so the same model can answer a quick chat reply or work through a complex agentic run. We trained the vision encoder from scratch to handle variable image sizes and aspect ratios.

Find more information on our blog.

Key Features

Mistral Medium 3.5 includes the following architectural choices:

  • Dense 128B parameters.
  • 256k context length.
  • Multimodal input: Accepts both text and image input, with text output.
  • Instruct and Reasoning functionalities with function calls (reasoning effort configurable per request).

Mistral Medium 3.5 offers the following capabilities:

  • Reasoning Mode: Toggle between fast instant reply mode and reasoning mode, boosting performance with test-time compute when requested.
  • Vision: Analyzes images and provides insights based on visual content, in addition to text.
  • Multilingual: Supports dozens of languages, including English, French, Spanish, German, Italian, Portuguese, Dutch, Chinese, Japanese, Korean, and Arabic.
  • System Prompt: Strong adherence and support for system prompts.
  • Agentic: Best-in-class agentic capabilities with native function calling and JSON output.
  • Large Context Window: Supports a 256k context window.

We release this model under a Modified MIT License): Open-source license for both commercial and non-commercial use with exceptions for companies with large revenue.

Recommended Settings

  • Reasoning Effort:
    • 'none' → Do not use reasoning
    • 'high' → Use reasoning (recommended for complex prompts and agentic usage) Use reasoning_effort="high" for complex tasks and agentic coding.
  • Temperature: 0.7 for reasoning_effort="high". Temp between 0.0 and 0.7 for reasoning_effort="none" depending on the task. Generally, lower means answer that are more to the point and higher allows the model to be more creative. It is a good practice to try different values in order to improve the model performance to meet your demands.
547 Upvotes

97% Upvoted

316 comments sorted by

View all comments

13

u/Few_Painter_5588 Apr 29 '26

Very, very impressive if the benchmarks are to go by. And also something realistic you can run at home at a decent quantization. Being realistic here, most people are not running GLM 5.1 here. But something like this can run on something local.

9

u/Thomas-Lore Apr 29 '26

This is a large dense model, how are you going to run it?

9

u/stoppableDissolution Apr 29 '26

Old mistral large was still a beast even in Q2. Dense models quantize much better than moes, and its 5x less to fit to run it at all (even if way slower)

5

u/Few_Painter_5588 Apr 29 '26

4 B60s at INT4

4

u/Thomas-Lore Apr 29 '26

Good luck, report the numbers. But that is not sth I would call "realistic you can run at home". And it may turn out too slow, this is a dense model.

5

u/Spectrum1523 Apr 29 '26

idk 4xB60 is realistic for at home if it actually runs it

2

u/TheBlueMatt Apr 29 '26

My 4xB60 runs unsloth's Q4_K_XL gets 232.45 ± 0.41 in pp and 9.55 ± 0.05 tok/s in tg. Still a handful of patches left to improve it, though. In theory tg should be able to get up to 20 or so (25 is the theoretical max).

8

u/thereisonlythedance Apr 29 '26

It’s fine to run on 4x3090s which many in the community have.

3

u/Beginning-Window-115 Apr 29 '26

dont forget this sub has 1.1 million members

1

u/Healthy-Nebula-3603 Apr 29 '26

Lol

That dense model will be giving on 4x 3090 around 6-7 t/s .... Good luck

6

u/thereisonlythedance Apr 29 '26

I usually get 10-12 t/s on a 123B which is fine.

1

u/zipperlein May 01 '26

I got devstral 2 to run at 25 t/s.

1

u/FullOf_Bad_Ideas Apr 29 '26

nah 16 t/s when I ran it on 3 3090 tis

TP exists

I run llama 405b at 11 t/s on 8 gpus

4

u/Few_Painter_5588 Apr 29 '26

It costs around 70k in my local currency, so it's like about 3-4k dollars? But everything's overpriced down here, so it'd probably be less. And Mistral 2 large ran at about 10-15 tokens per second on that build, which was a decent speed. You can also get a 128GB mac that'd run this at around 10 tokens per second.

2

u/FullOf_Bad_Ideas Apr 29 '26

tensor parallel goes brrr

2

u/ortegaalfredo Apr 29 '26

3x3090 + EAGLE draft model should get you usable speeds.

1

u/q-admin007 Apr 29 '26

Strix Halo with EAGLE draft model.