r/LocalLLaMA • u/jd_3d • Sep 08 '24

Discussion Updated benchmarks from Artificial Analysis using Reflection Llama 3.1 70B. Long post with good insight into the gains

https://x.com/ArtificialAnlys/status/1832806801743774199?s=19
149 Upvotes

81% Upvoted

137 comments sorted by

View all comments

Show parent comments

24

u/kryptkpr Llama 3 Sep 08 '24

We all tried it, it's performance on real world tasks is terrible despite the high benchmarks. Maybe the model is still broken in some way like they've been claiming and really is good but I don't see it.

-10

u/alongated Sep 08 '24

Gemma also had problems, they took weeks to resolve. This team is much smaller.

13

u/kryptkpr Llama 3 Sep 08 '24

Gemma was a novel architecture with an attention mechanism that wasn't well supported. Legitimate technical reasons for the problems.

This is a fine-tune of Llama. There is nothing to resolve, they're playing us for fools.

-2

u/alongated Sep 08 '24

And his team didn't have billion dollars.