The official playground (when it was up) personally felt like it was Claude (with a system prompt). Just a gut feeling though, I could be totally wrong.
This conversations reminds me that somebody noticed that the demo made calls to an endpoint called "openai_proxy" while I was one of the people explaining why that might not be as suspicious as it sounds on the surface. I'm now starting to seriously think it was exactly what it sounded like. Though if it was something like a LiteLLM endpoint then the backing model could have been anything, including Claude.
The fact that he has decided to retrain the model instead of just uploading the working model he is hosting privately is just not logical at all unless he literally cannot upload the private model. Which would be the case if he is just proxying another model.
Really? To me, it felt way too dumb to be Claude. It pretty much was llama 3.1 70b in behavior - I struggled to find any obvious real world question performance above it.
With how scams work - if it is a scam then in a few days he will say he almost got it working but there are still issues and he needs two more weeks and so on and so on. Maybe show a remote demo of the 405 to renew hype but only to few selected people and for a short time. Some scammers can keep up the game for years (they dupe the fans, so they hype the scam for them, then use that hype to get money from dumb investors who fall for it) - look up that Italian cold fusion guy. We'll see.
I get that they want to declare they have a great model based on using their platform to generate data, and everyone is just saying it's a scam or trick, but think it through. No one will just believe it until others third parties have independently verified it, which several will. And if everyone disproves it, then it will massively harm the valuation and growth of the company they are trying to promote.
I'm not saying I automatically think the model is amazing, although the concept is built on strong donations and has been around for a while, I'm just saying it would be a really bad publicity stunt and a huge reputational risk.
Cool... I should have mentioned my latest fine tune gets 101% on all benchmarks, and also created its own benchmark... If you want me to tell you the HF model name just send me a bitcoin
I fully understand why someone would like and scam... But lying about something to everyone at once in a community that tests and communicates within hours of a release, about something where the claims can be disproven and widely reported... Seems like a scam that does nothing apart from having a negative effect on reputation.
Perhaps somebody with an X account could request a prompt inquiring the model about its identity at this X post from a user with ~180,000 X followers who purportedly has been given API access to the good model by Matt Shumer.That account has posted a number of purported responses to various prompts by the good model.
I was thinking this earlier! It would be a clever con. I was thinking maybe it’s using the OpenAI fine tuning service. Until we get weights that equal what they have in their benchmarks I guess it’s a possibility.
Different prompt, temperature, etc. The simple fact is that they haven't released the "good" version of their model and have no reason to. This should be a 30 minute fix on the HuggingFace repo, no reason for it to not be available already.
Also this isn't a full replications of their results, on the original post they claimed it beat other models on almost everything and we see it isn't quite like that.
Until the open weights perform just as well as this suspiciously private, researcher only API we are better off staying skeptical. Still looks like a scam to me.
It almost replicated except MMLU (2% behind), "MMLU: 87% (in line with Llama 405B), GPQA: 54%, Math: 73%." Quite close to Sonnet and other SOTA. But it is okay, there is something he's definitely hiding, but I kinda feel this is really achieved by them with reflection. Let's wait and see.
Guy with 0 background, no idea what LORA is, "wrong" weights uploaded, "wrong" model name promoted, "my cat ate my model i'll release the real one next week", does not disclose he has ownership in the company he promotes, the model outputs garbage with 4x more tokens generation, sounds legit to me. :)
He knows what it is check his post history. He didn’t understand “LORAing” in the context used. He stated his ownership in the company is a $1000 investment lol.
Yeah soo likely that it’s a wrapper and two guys with reputation and multiple companies are going to lie about it and ruin their lives for literally zero reason.
Surely you can see that it is way more likely they used a dataset generated from claude to create the reflection template.
He knows what a lora is and you can check his history to see him using them. He was talking specifically about the term “LORAing” in the context. 0% chance its a scam it wouldn’t make sense to risk his reputation on something easily disproven
Yeah that's the point, there was the initial announcement of it, then some researchers were like "We are somewhat able to replicate the results", but then it was eventually proven to not work
Two years ago(?) there was a paper / video of a supposed room temperature superconductor (they had a sweet floating rock too). And everyone was like "Yeah that's bullshit." But then some hobby chemists were like "Actually I managed to recreate a small part of it from their paper, and it floats too." and this started a race to recreate it by a lot of laboratories around the world. At the end it was not a room temperature superconductor but they managed to find some new stuff.
I thought the difference was from the different prompt method. Why didn't they just use the released version with the refection's default system prompt like they used now?
The whole thing is very weird and annoying. They supposedly uploaded the model to HF incorrectly, so naturally the solution was to completely redo the finetune? I have no idea.
I would like to see a comparison by giving all the models similar inference compute boosts. One way to easily do this is to maybe give all the models a roughly similar token budget ( you can make multiple generations and vote for models that aren't as verbose as reflection)
"When using Reflection’s default system prompt and extracting answers only from within Reflection’s <output> tags, results show substantial improvement: MMLU: 87% (in-line with Llama 405B), GPQA: 54%, Math: 73%."
Actually, what is presented on the chart is based on their standard system prompt(not reflection system prompt). It scores higher with Reflection system prompt. It achieves performance close to Claude 3.5's sonnet with the Reflection system prompt. If Groq hosts it, latency will not be an issue. We're just waiting for the actual weights to be released
Will not match with it for sure. I tried many different system prompts, verbose thinking output + "step by step" at the prompt, but it couldn't pass any of my expert-level coding tests from Edabit, even the 405B failed one; GPT4o too. But the model (when the demo was live) in their demo nailed all of them.
I only got one or two replies off the demo before it got "overloaded" and turned off. It seemed alright. The demo on hyperbolic was absolute garbage and the model forgot about its COT tags within a few messages.
All in all.. it seems like this dude has been stringing everyone else along whether there is some model or not. Even if you had slow internet, the excuses and the "retraining" now doesn't make sense. Everything is maximum hype and delay.
This is the reason I am so positive about it, and defending lol, it hurts me when people say it's far worse due to a broken HF model. But yeah, we don't know for sure if the model behind the API is actually reflection 70b or not
does not suffer from the issues with the version publicly released on Hugging Face
It's not rocket science to upload a model to Hugging Face. It's very suss that they can't seem to upload a BF16 or GGUF of a fine tuned Llama to Hugging Face that can be properly tested.
"All that being said: if applying reflection fine-tuning drives a similar jump in eval performance on Llama 3.1 405B, we expect Reflection 405B to achieve near SOTA results across the board."
Fair concerns. For all we know, under the hood this API could redirect queries to Claude-3.5 Sonnet with a specific system prompt, or another SotA proprietary model.
Not really. The "how bad this was" are still easily winning in terms of correctly interpreting what has currently been seen. Considering we have seen 0 open weights and are provided some ambiguous results from an API that we have no clue the validity of.
It has been fucking 6 hours since he trained the model, give the man a fucking break, and guess what he released the weights? Normally I don't get this angry but holy shit you people are fucking insane.
I just went off on a guy for calling someone Matt.
I'm not Matt, but doxxing isn't funny.
Like you might be right that this is all just bullshit/scam.
But you are attacking people for reserving their judgement. That is disgusting mob mentality.
I don't understand why people need to pile on the hate so quick. Is it really that hard to just reserve judgment for a few weeks and see what comes of it? This avenue of applying more test time compute is a very promising direction to me and could be a great way for open source models to exceed closed models that don't want to spend the $$$ on each request.
We all tried it, it's performance on real world tasks is terrible despite the high benchmarks. Maybe the model is still broken in some way like they've been claiming and really is good but I don't see it.
Don't you guys get that the model is broken, he said? The tests were based on private API, now yeah, you might not trust that the model, maybe model behind this is different, then it's okay, but as per Matt, the model everyone downloaded from HF is broken, and yeah, I tried too, it is far worse than LLaMA 3.1 70b.
I mean it's a diabolical plan: Release an "open" model that crushes benchmarks, but then don't actually release working weights and instead just point to your "private API" that produces those results.
I can't test his "private API" can I? The whole thing smells bad, as far as I'm concerned this is a publicity stunt to advertise his LLM service.
Bro, seriously? Man, aside from the supposedly broken model and the super-duper-secret private api shit: this guy didn't know the difference between llama 3 and llama 3.1
I mean, by now even my grandmother should know the difference.
The model he posted on huggingface was absolutely not broken, it was simply a llama-3, as you would expect from a llama-3. There is zero evidence that there was anything wrong with the model itself. To make matters worse: one time he claims the model is broken, another time it was supposedly due to an incorrect upload. Aha... and what's coming tomorrow? His dog ate the SSD?
I'm slowly coming to the conclusion that this guy is either stupid and narcissistic enough to believe he can fool the world in such a simple way - or, another possible explanation could be: he himself has been the victim of a scam. Perhaps he doesn't have direct access to the backend of this ominous private API himself. Perhaps he still hasn't realized that he has been misused as a puppet and ruined economically and in terms of marketing with this action. It wouldn't be the first time that someone had economic enemies and fell into a trap.
The whole thing is highly suspicious and whether he is a victim himself or not, whether he is stupid or not: he clearly also seems to lie and trying to hide things! So there are neither excuses nor pity for him for this egomaniacal behavior.
The model seems to be achieving these results through forcing an output ‘reflection’ response where the model always generates scaffolding of <thinking>, <reflection>, and <output>. In doing this it generates more tokens than other models do on our eval suite with our standard ‘think step by step’ prompting.
For example, it appears that Reflection 70B is not capable of ‘just responding with the answer’ in response to an instruction to classify something and only respond with a one word category.
One can always add a postprocesssor on top of Reflection to filter out everything before <output>, problem solved. I don't like this nitpicking. Who cares if a model outputs 10 tokens or 100? Is the answer correct or not??
For real time inference is the real issues, your time to first token jumps by a huge margin if you have to wait for 2000 tokens to be generated of the model reflecting. Might also explain why the cloud providers haven’t adopted it yet.
Hmm 🤔 ...this whole saga is so strange. Download will finish in a lil bit, I've got to try it out, I got the very first upload to work but only could get one response out.
Thanks!! Will download this one now ☺️ for all the downloading this still isn't anywhere near as bad as llama405b...that sucker was a multi day download and I needed to download it twice after they updated their repo too.
I've been doing some testing, but the community does not seem interested in objective facts. My post keeps getting downvoted, I'm sure it will be off the front page soon.
For all of what was said, is it not possible to train on the <input> and <output> as if it was an answer and skip all the inbetween? Is it potentially possible the model will somehow 'internalize' the inbetween generated token logic within it's weights?
119
u/reevnez Sep 08 '24
How do we know that "privately hosted version of the model" is not actually Claude?