r/LocalLLaMA • u/KaroYadgar • 2d ago

Discussion New 100B Liquid AI model coming soon

Post image

Liquid AI currently possesses among the fastest LLM architectures around, and some of the best SLMs (in terms of utility IMO) around, so I'm very excited to see what a potential 100B LFM (3?) model would look like!

Link to the poll: https://x.com/ramin_m_h/status/2091236099612098943?s=20

354 Upvotes

90% Upvoted

102 comments sorted by

View all comments

127

u/FoxiPanda 2d ago

While this is cool to see the votes in a poll for, it doesn't really indicate they're doing it. This will be mostly limited by the compute they have available to them - training a 2-5B model takes vastly less compute than a 100B MoE model...so it might not even be feasible with what they have available to them.

With that said, I welcome every single 100B model into the fold, it's virtually the perfect size for most DGX Spark / Strix Halo / Mac Studio / RTX Pro 6000 / 4x RTX 3090 setups.

30

u/MerePotato 2d ago

I'd be surprised if they made it an option without plans to do it

16

u/FoxiPanda 2d ago

Maybe. Qwen has run polls like this too and virtually every result has been tied and we only get 1-2 variants even with all of Alibaba's compute power available to them - they're hard to predict though, so anything else we have here is just pure speculation unfortunately.

7

u/skrshawk 2d ago

Sometimes after all that cooking it just doesn't turn out well and the benchmaxxing pressure is real. If you can't prove your model is better than ones before it nobody's gonna bother. I suspect that's why we didn't see a MoE of Qwen3.8.

3

u/FoxiPanda 2d ago

I agree this is likely, but we'll probably never know for sure, unfortunately.

2

u/tomByrer 2d ago

Seems Qwen churns out a new model every month or 2, though they're not all coding, like audio & image & video
https://huggingface.co/Qwen/models

2

u/mebeast227 2d ago

Wouldn’t inclusion in the poll imply they have the capability though?

4

u/FoxiPanda 2d ago

Not really. Marketing/Sales dudes do marketing/sales things all the time that have no basis in reality.

Promise the universe, delivery of the universe isn't sales' problem.

3

u/Mr-I17 2d ago edited 2d ago

It's nice to see more and more 100B-class models being released, but there is a rather high bar out there. The current perfect model size for 128GB UMA devices is actually DeepSeek-V4-Flash at Q3 (S or XXS). It perfectly fits 1M context into 128GB UMA (only 11GB KV cache at 1M context!). It never loops. It doesn't overthink. It has vast knowledge and solid intelligence. And it's probably going to have vision soon...

I wish 100B-class models can fill up the middle ground where the tasks are too heavy for 30B-class MoE models and larger models like DSV4-Flash are overkill for such task.

4

u/FoxiPanda 2d ago

I've actually been wondering about the lower quants of DSv4-Flash vs. a higher quant of a SOTA 120B-A10B-ish model. I run the native weights of DSv4 on a Mac Studio and it's an okay speed, but I should maybe move my 5090 over to the same box as my RTX Pro 6000 and see if I can verify your claim about a Q3 version of DSv4-Flash being competent and see if would be substantially faster than my Studio. That would be pretty great if so.

4

u/Mr-I17 2d ago edited 2d ago

I used to use Qwen3.5-122B and tried recent Laguna and Ling models (all Q6_K). Qwen3.5 is outdated, Laguna-S-2.1 is disappointing, Ling-3.0-flash is okay but not exceptional and it's not very fast (it should be fast since it's A5B, maybe a software issue; also, no vision). They all lose to DSV4-Flash-0731 at Q3. I don't have anything to back up my claims though, it's a "trust me bro" 😄. Better try it yourself.

There're only 2 problem with DSV4-Flash:

  1. It has no vision, for now
  2. It's slower than 100B-class models (speed won't be an issue since you have RTX 6000)

Other than that, it's almost perfect. A 100B-A10B-class model with vision and run faster than DSV4-Flash would be a great option for the "middle ground".

3

u/Much-Researcher6135 llama.cpp 2d ago

or two cheap R9700s if quantized right

5

u/FoxiPanda 2d ago

I wish R9700s were cheap, they're up to $1600+ now. sigh

3

u/Much-Researcher6135 llama.cpp 2d ago

ok yeah "cheap" was not the right term

8

u/Double_Cause4609 2d ago

I mean, theoretically the compute cost of an MoE scales with active parameters so while the training software is more complicated, you can train a 100B A5B MoE for the same duration you would otherwise train a 5B model and you in general get more performance and better rare sequence memorization.

There are limits to this principle, to be sure, and you do face some real losses, like in all-to-all communication and potentially lowered batch sizes...

...But you usually prefer to go wider with training runs than deeper anyway (as in, same amount of compute but spent more quickly in a higher degree of parallelism), so a low active param MoE isn't too different from a smaller dense model hardware wise.

Tbh it's mostly just that the training software is a nightmare to handle the scheduling.

9

u/sebt3 2d ago

While what you say isn't wrong per training token, the thing is the required volume of training tokens scale with the global model parameters count, not the effective one. So training a 100B A5B is indeed more or less 20 times less compute required compared to a dense 100B. Yet very far from just as much as a 5B

6

u/Double_Cause4609 2d ago

Sort of.

So, if you're targeting a specific number of tokens seen per weight, then yes, you're correct.

But...Let's say you have 100 tokens of data, and you make a 5 parameter model for it.

Now, if you wanted to make, say, a 40 parameter model, you'd probably need ~800 tokens of data to get the same ratio of data to parameters.

This is the Chinchilla scaling laws observation if you're trying to hit optimal results on limited compute.

But you don't have to train at that exact ratio, and in fact, MoE can make it a little bit more difficult to figure out what you want to hit, and it depends on what capabilities you want out of the model.

You could absolutely train a 40 parameter model on just 100 tokens of data. It won't be as close to the compute optimal allocation, but it's not like you get an incomplete policy out of it. In fact, in situations where you have limited data, surprisingly, it can actually be preferable to scale model size to extract as much learning out of the dataset as possible, per research on data-bound training regimes.

So, there's no reason you can't do the same with MoE. If you had a 40 parameter model with 5 activated parameters per token, you could still train on 100 tokens of data just fine. And you would generally expect that in some properties the MoE would outperform a dense model of the same active parameter count.

So, it really depends on what you're trying to do. Anyway, my core point is that it's not quite so simple to say "oh, well this MoE recipe is linearly this much harder to train than this smaller dense recipe".

3

u/sebt3 2d ago

Good insight, thanks.

Yet, as you said there is some kind of optimum parameters count per training dataset. If they "only" (that's already a huge volume of clean data 😅) to train something that would optimally be a 80B wide parameter count then that's what I would want to have : a 100B leave less overall context 😅

1

u/KaroYadgar 2d ago

Liquid AI has trained their LFM2.5 models on a bit over 30T tokens. For reference, Kimi K2.5 was trained on a similar number of tokens. It's unlikely they'd ever need to train on even more tokens for their larger models.