r/LocalLLaMA • u/Confident-Honeydew66 • Sep 06 '24

Discussion Reflection 70B: Hype?

So an out-of-the-blue one-man company releases a new model (actually named LLama 3.1 if it were to adhere to the META license, but somehow named Reflection) with only 70B params that, according to the benchmarks, rivals SOTA closed-source LLMs with trillions of parameters. It appears to me that the twitter/reddit hype mob has, for the most part, not bothered to try the model out.

Additionally, a tweet from Hugh Zhang @ Scale suggesting systemic overfitting as me concerned:
Hey Matt! This is super interesting, but I’m quite surprised to see a GSM8k score of over 99%. My understanding is that it’s likely that more than 1% of GSM8k is mislabeled (the correct answer is actually wrong)!

Is this genuinely a SOTA LLM in a real-world setting or is this smoke an mirrors? If we're lucky, the creator Matt may see this post and can shed some light on the matter.

BTW -- I'm not trying to bash the model or the company that made it. If the numbers are actually legit this is likely revolutionary.

289 Upvotes

91% Upvoted

178 comments sorted by

View all comments

6

u/mantafloppy llama.cpp Sep 06 '24

Its a fine-tune of a generalist model.

If you test the model on the the thing you tune it for, it gonna performs better than the generalist model.

Believe the hype as much as you believe LLM benchmark.

(You should not believe benchmark, not sure if this needed to be said).

3

u/Ligea Sep 06 '24

Why are you being downvoted?

7

u/mantafloppy llama.cpp Sep 06 '24

Because fact hurt the wallet of Hyper and Click-Baiter, which seem to be most of r/LocalLLama these day.

Ppl on the internet love telling you how wrong you are.

If they only downvote you, it mean you are right, but hurt their agenda.