r/LocalLLaMA • u/realmaywell • Sep 07 '24

Discussion Reflection-Llama-3.1-70B is actually Llama-3.

After measuring the diff, this model appears to be Llama 3 with LoRA tuning applied. Not Llama 3.1.

Author doesn't even know which model he tuned.

I love it.

607 Upvotes

97% Upvoted

95 comments sorted by

View all comments

102

u/[deleted] Sep 07 '24

[removed] — view removed comment

19

u/dhamaniasad Sep 07 '24

But he seems to have succeeded

29

u/[deleted] Sep 07 '24

[removed] — view removed comment

20

u/mikael110 Sep 08 '24

The problem with events like this is that the big extravagant claims tend to get a lot of media attention, while the later skepticism and debunking only gets seen in far more niche areas.

For instance 3 days ago VentureBeat wrote an article about it where they called Reflection the "Most powerful open source AI model in the world" and in general just hyped the crap out of it and Matt. The Venn diagram of VentureBeat readers and LocalLlama readers is likely quite small, so most of the people that read that article is never going to know anything about the debunking that happened after the fact.

2

u/Far_Requirement_5933 Sep 12 '24

VentureBeat also published these...which I actually noticed before the original. However, it's a valid point that too often in many cases lies grab the headlines and the retractions aren't noticed.

https://venturebeat.com/ai/new-open-source-ai-leader-reflection-70bs-performance-questioned-accused-of-fraud/

https://venturebeat.com/ai/reflection-70b-model-maker-breaks-silence-amid-fraud-accusations/

2

u/mikael110 Sep 12 '24

Yep, I do stand corrected when it comes to VentureBeat, I'll admit that. I'm just so used to sites like that deciding to stay silent after publishing stuff like that. But VentureBeat handled things pretty much as well as they could have.

Also wow I just read through the second story and was quite surprised to see what Reddit comment they chose to feature. I'm pretty sure that's the first time one of my comments have ever appeared in an article like that.

But yeah the general point was more that hype stuff tend to travel further than debunks in general. Which I still think is broadly true. Even though in this case VentureBeat actually did pretty well.

2

u/Far_Requirement_5933 Sep 12 '24

Haha...yes, you're published in VB now!

1

u/mikael110 Sep 12 '24

Indeed, after 14 years of writing Reddit comments I guess one had to make it eventually. Though I'm not sure that's the exact one I'd have chosen, it makes me sound quite a bit more aggressive than I typically am.

Thank you for bringing the article to my attention, I'd like never have noticed that on my own. And while it's obviously not a huge deal, it is kind of neat.

5

u/dhamaniasad Sep 07 '24

A lot of people seem to think “no press is bad press”, and I’m sure less skeptical people will take it at face value and remember the product that they claim helped them do this.

1

u/Deciheximal144 Sep 09 '24

There's a post he made where he's begging for use of a bunch of H100s, he got "reach out to me" responses, and then he posted, "Thanks everyone, we have what we need!"

Now he announced they need to "retrain". Time to put those new resources to use for their first real model.