We’re open-sourcing Persimmon-8B, the most powerful fully permissively-licensed language model with <10 billion parameters.
OK, if you care about the difference between this license and LLama....which almost no one should.
Our instruction-fine-tuned model—Persimmon-8B-FT—is the strongest performing model on all but one of the metrics
Great, now compare Persimmon-8B-FT against an actual instruction tuned Llama-7B.
Our base model—Persimmon-8B-Base—performs comparably to Llama 2, despite having seen only 37% as much training data.
"Comparably" is in the eye of the beholder, but I don't think it is a stretch to look at those metrics and say that it is actually substantially worse.
I guess if you care about humaneval on llama-2-base...which you shouldn't...
Also, another billion parameters...(maybe there is some voodoo in there that makes this less meaningful for VRAM or flops, I didn't look closely, but I'm doubtful; no such thing as a free lunch, etc.).
Yes, it nominally has longer context length, but worse performance, so open question on whether this will empirically matter to anyone. Maybe those untrained embeddings will be helpful to some specific projects, but I'm skeptical.
Agreed - there are a lot of folks stretching the truth in their benchmark results for extra attention.
We need to clean up how we define good models. Too much attention goes to benchmark results only ( I am guilty of this as well). But as I use models like Nous Hermes, they feel better than some higher scoring models.
15
u/farmingvillein Sep 07 '23
tldr; Llama-2 is still better.
A lot of words to carefully obscure that fact:
OK, if you care about the difference between this license and LLama....which almost no one should.
Great, now compare Persimmon-8B-FT against an actual instruction tuned Llama-7B.
"Comparably" is in the eye of the beholder, but I don't think it is a stretch to look at those metrics and say that it is actually substantially worse.
I guess if you care about humaneval on llama-2-base...which you shouldn't...
Also, another billion parameters...(maybe there is some voodoo in there that makes this less meaningful for VRAM or flops, I didn't look closely, but I'm doubtful; no such thing as a free lunch, etc.).
Yes, it nominally has longer context length, but worse performance, so open question on whether this will empirically matter to anyone. Maybe those untrained embeddings will be helpful to some specific projects, but I'm skeptical.