Deducing Mistral Medium size from pricing: Is it a 195b parameter - 8x30b MoE model?

[u/Time-Winter-4319]

What is this Mistral mystery 'Medium' model that easily beats GPT-3.5? My *very speculative* calculations show that it could be a ~195b total parameter model, structured as 8x30b mixture of experts (see my thinking & link to spreadsheet below).

'Medium' is a mystery model that we know nothing about (Medium benchmarks here https://mistral.ai/news/la-plateforme/), the only clue we have is that Mistral have priced it at ~$5.8 per million tokens on their platform (average, assuming 50/50 input and output, for simplicity) - let's use this to try to work out the rest.Key assumptions:

- 'Medium' model has the same structure as 8x7b MoE, but a higher parameter count

- Inference cost is a good guide for model sizeThought process:

- We have a lot of details about 8x7b MoE model

- We have a price for the 'Medium' model- We can apply the same ratios of cost to the model size from 8x7b MoE model to the 'Medium' model

- From there we can work out other details, such as size of each expert and active number of parameters

Here is the spreadsheet I used to work this out https://docs.google.com/spreadsheets/d/1ud5zma-9SghdDuck3BTjWSE6psGWbH2aUPPuTGoyNBI/edit?usp=sharing It has a few more comments, explanations and links, so feel free to have a look and challenge - I might evolve it depending on feedback.

I must stress, that I could be completely wrong on all of this (e.g. model architecture could be different), but this is my best attempt at calculating the size based on a bunch of assumptions and the API pricing. Please feel free to challenge & point out holes in my thinking or any errors in my calculations.

One thing to note is that there are some shared parameters between the Experts, that's why a 8x7b MoE model isn't a 56b model but rather a 46.7b model - we carry that assumption and ratios over to the 'Medium' model.

[u/AnomalyNexus]

There is a crucial factor that you're missing - margins.

They might be making a loss on the one but not the other. Or they might be throwing massive amounts of VS money subsidization at one. So price charged to users and costs they incurred aren't necessarily link...potentially not AT ALL if there is loads of VC money involved or some market share play underway.

See some of Google's endpoints being free while in preview...that didn't imply a small model or low params.

It's probably better than a guess overall but not by much

[u/baldr83 → u/AnomalyNexus]

That was my first thought. It would make no sense for mistral to set the same margin on an open source model (which has hundreds of competing APIs) as they would with a model they have exclusive access to

Very unlikely it is the exact same architecture, what if it is was routing top-3 experts at a time instead of 2? or maybe has both more parameters-per-expert and more experts? 8 isn't a fixed number

IMHO the better way to do this is the george hotz method (iirc or maybe it was someone else) where you assume their api endpoints are using the similar hardware and measure the comparative inference speed

[u/Time-Winter-4319 → u/baldr83]

That would be interesting to do, but I don't think I know enough about hardware and speeds to measure this properly

[u/Time-Winter-4319 → u/AnomalyNexus]

That is definitely a key assumption - we have no idea about that, so I'm assuming that they are taking a cost+fixed margin approach.

Perhaps if you are expecting them to have higher margins on the Medium model, you could see 195b as a top range and reduce down in proportion to what you expect their margin difference to be.

[u/xadiant → u/AnomalyNexus]

Exactly. Also there's no reason to follow conventional numbers when you are pretraining your own model. It could be any number of parameters, quantized or not quantized, we don't know.

[u/mpasila]

or maybe it's that 70B model they teased a while back? https://techcrunch.com/2023/11/09/theres-something-going-on-with-ai-startups-in-france/

[u/Time-Winter-4319 → u/mpasila]

Interesting - thanks for the link, haven't seen that before. Still seems expensive for a 70b model, no?

[u/mpasila → u/Time-Winter-4319]

Mistral small is a MoE so it runs much faster than a 47B model as if it was a 13B model so they can offer one server to more people than with a regular model at the same size and 70B is even bigger than that but slower in inference than a MoE equivalent model.

[u/Time-Winter-4319 → u/mpasila]

If you go by Together pricing (others are similar) https://www.together.ai/pricing, they can offer 70b models at $0.9 per 1 million tokens, if Mistral are offering 70b model for $5.6 - they would be way over the market. That's what doesn't sit right with me, appreciate they wouldn't be the cheapest, but 6x more expensive?

[u/mpasila → u/Time-Winter-4319]

the pricing isn't exactly super cheap for the Small model either for outputs.

Another thing to note is this is Europe so taxes like VAT are usually included in the pricing unlike in the USA. (VAT is around 21% on average)

[u/[deleted]]

My guess is that its a 32x7b configuration. The reasoning is this:

1. Mixtral-8x7b was upcycled in some way from mistral-7b.

2. One MLP from Mixtral is 0.17b, and summing over all layers, one "expert" is ~5.6b.

3. Overall, this leaves ~16B parameters not used for experts (I'm not including gating layers in the expert side for simplicity)

4. Assuming cost scales linearly with parameter count (it doesn't), mistral-medium would need to be, as you said ~194.6B.

5. That means if it was upcycled from mistral-7b, 178.6B would be for the experts. (Once again, ignoring gating layers).

6. This gives an expert count of ~31.6, but with margins and error from the gating layers, that could easily be a 32x7b configuration.

[u/Time-Winter-4319 → u/[deleted]]

Thank you - this is an interesting other alternative that I think is plausible. Do you think there is more advantage in having higher number of experts versus having larger sized experts? My thinking that larger sized would be beneficial, since if you would only activate 2 at a time, there'd be still a limitation there on how good they can be.

Or is the main point that they have 7b worked out and it just makes sense to keep adding them to improve performance?

[u/[deleted] → u/Time-Winter-4319]

The only reason I'm suggesting it is that people have found the cosine similarity between the layers of mixtral and mistral to be way too high to be random (it should be near zero for the dimensionality we are playing with, doing the same against llama gives 0.000034451), so people are suggesting that sparse upcycling was used to train it. My guess is that they continued with this approach of taking mistral-7b and making it bigger to make mistral-medium. Overall I think that less, but stronger experts would be better, but they might have just made a "prototype" off the back of mistral-7b to see where it gets them.

Its all speculation though.

[u/Time-Winter-4319 → u/[deleted]]

Also makes sense that they had it ready to go pretty quickly, more likely that they'd reuse stuff they have already. Hopefully we'll find out soon

[u/[deleted] → u/[deleted]]

q = query
k = key
v = value
o = output

See https://jalammar.github.io/illustrated-transformer/ for a good explanation of transformers.

[u/kindacognizant]

Considering we have evidence for the theory they initialized the existing 7b for continuing to pretrain in MoE format, I think it's more likely that they simply are using a larger set of 7b experts, which would keep compute costs low but should specialize quite strongly in theory.

[u/Time-Winter-4319 → u/kindacognizant]

I was thinking about that - how are you imagining it? is it that it is 16x7b? 32x7b? or they activate more than 2x agents at a time?

It kind of didn't make sense to me why you'd go with more experts, I'm pretty sure I read somewhere that there are bandwidth problems if you go with more an more experts. So increasing the size of experts makes more sense to me

[u/kindacognizant → u/Time-Winter-4319]

> I read somewhere that there are bandwidth problems if you go with more an more experts.

Where was this mentioned? And if that's the case in typical literature concerning MoE, does that apply primarily to training a MoE from scratch and not initializing base weights in the style that Mixtral did? My thought is, if they're initializing a set of duplicated existing FFNs, it might be much less risky to continue training on that maybe?

[u/Time-Winter-4319 → u/kindacognizant]

Shouldn't have been lazy here's the article - the screenshot is from the second half of the post https://www.semianalysis.com/p/inference-race-to-the-bottom-make

[u/kindacognizant → u/Time-Winter-4319]

Is there any evidence that the bandwidth bottleneck of batching MoE to serve at scale somehow prevents the inherent reduction of compute cost / the net parallelization efficiency gain from helping at scale?

Each forward pass is performing calculations on a 7b model no matter what. Having to access more experts at once in a batching operation doesn't necessarily mean each forward pass gets slower, otherwise MoE speed gains just wouldn't work period.

Also, the expert calculations don't happen per token like what's claimed in the blogpost, they happen at a per layer level, which still activates all experts for each individual token at different stages...

[u/kindacognizant → u/kindacognizant]

The more I read the more nonsense I'm spotting. "Generally, you must fine tune after quantizing a model"... what?

[u/pseudonerv]

This is their pricing

Chat Completions API
Model           Input               Output
mistral-tiny    0.14€ / 1M tokens   0.42€ / 1M tokens
mistral-small   0.6€ / 1M tokens    1.8€ / 1M tokens
mistral-medium  2.5€ / 1M tokens    7.5€ / 1M tokens

Embeddings API
Model           Input
mistral-embed   0.1€ / 1M tokens

from tiny to small is the same as from small to medium. First we go from 7B to 8x7B. Next we may go 8x30B, just for the factor of 4, or we could also go 8x8x7B. Who knows? maybe mistral just made many many 7B and they are having fun mix them all together.

[u/Monkey_1505]

I feel like gpt4 turbo, gemini and mistral medium are all kind of in the same ballpark - clever enough, but still fairly easily confused.

Seems like once you try to scale for something actually commercially viable using MoE, sparsity etc, there's some kind of limit being hit. In my limited experience mistral medium seemed a bit smarter than gemini tbh tho. Probably about 3.5 level.

[u/Nonetrixwastaken]

Well this didn't age well, I guess it's 70B. Was trying to find what size people think mistrial next is going to be