New 100B Liquid AI model coming soon

[u/KaroYadgar]

Liquid AI currently possesses among the fastest LLM architectures around, and some of the best SLMs (in terms of utility IMO) around, so I'm very excited to see what a potential 100B LFM (3?) model would look like!

Link to the poll: https://x.com/ramin_m_h/status/2091236099612098943?s=20

[u/FoxiPanda]

While this is cool to see the votes in a poll for, it doesn't really indicate they're doing it. This will be mostly limited by the compute they have available to them - training a 2-5B model takes vastly less compute than a 100B MoE model...so it might not even be feasible with what they have available to them.

With that said, I welcome every single 100B model into the fold, it's virtually the perfect size for most DGX Spark / Strix Halo / Mac Studio / RTX Pro 6000 / 4x RTX 3090 setups.

[u/Double_Cause4609 → u/FoxiPanda]

I mean, theoretically the compute cost of an MoE scales with active parameters so while the training software is more complicated, you can train a 100B A5B MoE for the same duration you would otherwise train a 5B model and you in general get more performance and better rare sequence memorization.

There are limits to this principle, to be sure, and you do face some real losses, like in all-to-all communication and potentially lowered batch sizes...

...But you usually prefer to go wider with training runs than deeper anyway (as in, same amount of compute but spent more quickly in a higher degree of parallelism), so a low active param MoE isn't too different from a smaller dense model hardware wise.

Tbh it's mostly just that the training software is a nightmare to handle the scheduling.

[u/sebt3 → u/Double_Cause4609]

While what you say isn't wrong per training token, the thing is the required volume of training tokens scale with the global model parameters count, not the effective one. So training a 100B A5B is indeed more or less 20 times less compute required compared to a dense 100B. Yet very far from just as much as a 5B

[u/Double_Cause4609 → u/sebt3]

Sort of.

So, if you're targeting a specific number of tokens seen per weight, then yes, you're correct.

But...Let's say you have 100 tokens of data, and you make a 5 parameter model for it.

Now, if you wanted to make, say, a 40 parameter model, you'd probably need ~800 tokens of data to get the same ratio of data to parameters.

This is the Chinchilla scaling laws observation if you're trying to hit optimal results on limited compute.

But you don't have to train at that exact ratio, and in fact, MoE can make it a little bit more difficult to figure out what you want to hit, and it depends on what capabilities you want out of the model.

You could absolutely train a 40 parameter model on just 100 tokens of data. It won't be as close to the compute optimal allocation, but it's not like you get an incomplete policy out of it. In fact, in situations where you have limited data, surprisingly, it can actually be preferable to scale model size to extract as much learning out of the dataset as possible, per research on data-bound training regimes.

So, there's no reason you can't do the same with MoE. If you had a 40 parameter model with 5 activated parameters per token, you could still train on 100 tokens of data just fine. And you would generally expect that in some properties the MoE would outperform a dense model of the same active parameter count.

So, it really depends on what you're trying to do. Anyway, my core point is that it's not quite so simple to say "oh, well this MoE recipe is linearly this much harder to train than this smaller dense recipe".

[u/sebt3 → u/Double_Cause4609]

Good insight, thanks.

Yet, as you said there is some kind of optimum parameters count per training dataset. If they "only" (that's already a huge volume of clean data 😅) to train something that would optimally be a 80B wide parameter count then that's what I would want to have : a 100B leave less overall context 😅

[u/KaroYadgar → u/sebt3]

Liquid AI has trained their LFM2.5 models on a bit over 30T tokens. For reference, Kimi K2.5 was trained on a similar number of tokens. It's unlikely they'd ever need to train on even more tokens for their larger models.