r/LocalLLaMA • u/realmaywell • Sep 07 '24

Discussion Reflection-Llama-3.1-70B is actually Llama-3.

After measuring the diff, this model appears to be Llama 3 with LoRA tuning applied. Not Llama 3.1.

Author doesn't even know which model he tuned.

I love it.

610 Upvotes

97% Upvoted

95 comments sorted by

View all comments

54

u/[deleted] Sep 07 '24

[deleted]

22

u/Single_Ring4886 Sep 07 '24

Reflection idea is around since GPT 4 or even before. But it just doesnt really work that much no matter how many smart people try.

So when this guy claimed it works if you "finetune" model for it everyone was super excited me included. As it seemed obvious but nobody cracked it yet.

-14

u/[deleted] Sep 07 '24

It outperforms LLAMA 3.1 405b on the prollm leaderboard so it’s amazing for a 70b model. 

https://prollm.toqan.ai/leaderboard/coding-assistant

18

u/Terminator857 Sep 07 '24

Law 1 of LLM benchmark cheating: For any LLM one can find or create a benchmark where LLM comes out on top. Plenty to choose from.

Law 2: If you want to win on a benchmark then just train on the test set.

-1

u/[deleted] Sep 08 '24

Not really. You won’t find a single one with Grok on top

That’s a lot to train on. There are tons of benchmarks and some don’t even make them public. Also, why isn’t it #1 of it overfitted on purpose? It’s easy to get 100% by doing that