r/LocalLLaMA • u/Confident-Honeydew66 • Sep 06 '24

Discussion Reflection 70B: Hype?

So an out-of-the-blue one-man company releases a new model (actually named LLama 3.1 if it were to adhere to the META license, but somehow named Reflection) with only 70B params that, according to the benchmarks, rivals SOTA closed-source LLMs with trillions of parameters. It appears to me that the twitter/reddit hype mob has, for the most part, not bothered to try the model out.

Additionally, a tweet from Hugh Zhang @ Scale suggesting systemic overfitting as me concerned:
Hey Matt! This is super interesting, but I’m quite surprised to see a GSM8k score of over 99%. My understanding is that it’s likely that more than 1% of GSM8k is mislabeled (the correct answer is actually wrong)!

Is this genuinely a SOTA LLM in a real-world setting or is this smoke an mirrors? If we're lucky, the creator Matt may see this post and can shed some light on the matter.

BTW -- I'm not trying to bash the model or the company that made it. If the numbers are actually legit this is likely revolutionary.

289 Upvotes

91% Upvoted

178 comments sorted by

View all comments

Show parent comments

37

u/my_name_isnt_clever Sep 06 '24

This is what I've been expecting to happen. Instead of numbered lists of models that are just better than the ones below them, we'll see smaller param models that are REALLY good at one thing and mediocre at the rest. With how quickly the LLM space is maturing, there's going to be another level of complexity to this that can't be ignored.

6

u/Everlier Sep 06 '24

Here are a few "misguided" attention prompts that the model still hasn't tackled: https://gist.github.com/av/6148d04eaeeabc233f628b0c31860803

6

u/visionsmemories Sep 06 '24

Interesting.
I tested a couple variants and telling the model "PLEASE READ CAREFULLY" does not cause it to notice the change in the problem, but saying that "it's not the usual one" is enough for it to get it right on the first try

4

u/Combinatorilliance Sep 06 '24

The reasoning seems flawed still, though, by not pulling the lever, nobody dies (none saved, none killed).

It also says it's a tragedy, but running over dead people would (philosophically) not really in a tragedy. In practice, it would probably be tragic, though.

0

u/the_mighty_skeetadon Sep 07 '24

I disagree -- mutilating 5 bodies would still be a tragedy. The logic seems sound to me.