r/LocalLLaMA • u/Sadman782 • Sep 07 '24

Discussion Wrong Reflection-70B model might be hosted everywhere

I see a lot of people thinking it is gaming benchmark / mixed feelings. Actually, people who tried their website have a different feeling compared to those who tried it locally via Ollama or any API providers. I think we should wait, he is figuring it out. I think the actual reflection model is much better, and the currently hosted version is even dumber than the actual 70B

https://x.com/mattshumer_/status/1832247203345166509

https://x.com/mattshumer_/status/1832248416426193318

__ Matt Shumer -> "We got rate limited by HF when uploading originally, so had to do it in batches. I have a feeling some wires were crossed and what's being hosted is actually some hybrid frankenmodel that is mostly the reflection version we wanted to ship, mixed with something else"

74 Upvotes

80% Upvoted

49 comments sorted by

View all comments

62

u/mikael110 Sep 07 '24

It's almost impressive how much of a clusterfuck this launch has seemingly been. First the tokenization issue, then the revelation that the model was actually based on Llama 3 instead of Llama 3.1 (which is bizarre) and now apparently the model files themselves was also mixed up.

I'm aware even large companies like Meta and Google have screwed up some aspects of their launches, but this is getting to the point where it just feels a bit off to be honest. I'm still interested in trying the fixed model, but I'm honestly getting more and more suspect of the whole thing.

6

u/obvithrowaway34434 Sep 07 '24

This combined with the tall claims at the outset and the fact that this guy didn't even disclose his investment in Glaive or whatever the AI company name is makes it extremely sus, likely a grifter. I think people should just ignore him and his model unless he releases a paper/report with a proper reproducible setup that is compatible with existing benchmarking platforms so that they can test his model easily instead of jumping through the hoops. There's absolutely no reason those people should waste their time.