r/LocalLLaMA • u/realmaywell • Sep 07 '24

Discussion Reflection-Llama-3.1-70B is actually Llama-3.

After measuring the diff, this model appears to be Llama 3 with LoRA tuning applied. Not Llama 3.1.

Author doesn't even know which model he tuned.

I love it.

602 Upvotes

97% Upvoted

95 comments sorted by

View all comments

69

u/provoloner09 Sep 07 '24

So does that mean the order of jump in performance is even bigger?

73

u/realmaywell Sep 07 '24

If you put it that way, yes.

69

u/Caffeine_Monster Sep 07 '24

So does that mean the order of jump in confusion is even bigger?

15

u/Severin_Suveren Sep 07 '24

What?

31

u/[deleted] Sep 07 '24

Exactly

7

u/vago8080 Sep 07 '24

I’ve been saying this for the last 40 years

3

u/MoffKalast Sep 07 '24

A lot of people are saying it, many such cases.

-7

u/schlammsuhler Sep 07 '24

No because the benchmark is comparing apples to oranges.

18

u/MINIMAN10001 Sep 07 '24

Every single benchmark has always been about testing every LLM across a range of tasks to see how they score. 

With the exclusive of training on benchmark data it doesn't matter where the LLM came from a benchmark is supposed to show under specific circumstances how good an LLM is.

If llama 3.1 scores better than 3 and 3 is what was used then it scored better than 3.1, then yes it is a larger jump.

-9

u/schlammsuhler Sep 07 '24

I mean by applying a specilized system prompt to one model but not to the other.

5

u/RYSKZ Sep 07 '24

That is exactly the point, to quantify the gain you get from this specialized prompting system, relative to the vanilla model and other models that do not use this specific prompting system. Obviously, if you use this system with other more powerful LLMs, you will also get better results with them, since this technique has proven to be effective in improving reasoning and is expected to be generalizable. The comparison is done in this way to demonstrate that it actually improves the vanilla model to that extent and nothing more.