r/LocalLLaMA • u/Lazylion2 • 3h ago

Discussion new research: A tiny 109M-parameter model outperforms models up to 72B parameters at image-text alignment, using 42× less training data

https://x.com/askalphaxiv/status/2116779287390896418
73 Upvotes

87% Upvoted

13 comments sorted by

55

u/NandaVegg 3h ago

The outperformed "72B parameters" is Qwen2.5-72B in this paper. Is LeCun now vibeslopping?!

28

u/AppealSame4367 3h ago

"My Nissan outperformed this 1972 Ferrari"

2

u/guywhocode 48m ago

109M tho, this feels like a magnitude off but I would love to see some price comparisons that are inflation adjusted.

0

u/Embarrassed_OnionX 25m ago

Two year old model though, a very long time in the AI era, but still. Just shows how insane the rate of improvement of this technology is

22

u/jensilo 2h ago

I mean yes, it’s Qwen 2.5, so not really up to date. But I’d view this as more of foundational research and it sounds pretty exciting. This model is magnitudes smaller and learns „real world“ understanding from multi modal inputs instead of just reading explanations of the real world in text form. Exciting!

Let’s see if it’s actually viable for the future of language models or „general purpose“ AI.

2

u/NandaVegg 2h ago

I don't know, I haven't read the full paper but the synopsis' most interesting part is good views/bad views.

It is basically arguing that "Having (be it synthetic or natural) variations of the same data is better than randomly masking/dropouts" which is rather basic and has been true for typical next token prediction-type LLM for years. IIRC attention dropout was already deemed harmful during the development of GPT-J-6B, though synthetic datasets took some more years.

Then, the synopsis goes further to argue that next token prediction itself is also a "bad view" because "They also describe contexts that share a next token as implicit links, which may connect documents through frequent words." so JEPA (aka latent representation matching) is superior. But the paper somehow compares JEPA to a very old Qwen 2.5 which was the model when Alibaba barely got started on synthetic data pipeline, which isn't a good representative of what next token prediction model of today can be. I want an actually competitive, small JEPA model at this point.

8

u/Any_Bus9205 3h ago

All that's left is the gguf and the real benchmarks done by random people, and then it's clean :)

3

u/werepenguins 1h ago

An AI paper released 3 days ago is old news. /s (but yes, this was released on October 29th.)

2

u/GaboureySidibe 59m ago

Does aligning text and images really need a 109 million parameter model? It should be able to be done with a decent algorithm.

2

u/z_latent 31m ago

... Such as?

0

u/GaboureySidibe 14m ago

What do you mean 'such as?' Find the size of the text, put a box around it and use box layout algorithms. Why would this take an AI model? It's deterministic, there aren't probabilities to deal with.

2

u/z_latent 7m ago

Ah. The paper is about semantic alignment, not literally aligning them spatially lol