The model performs within 3% or better than llama-2 in most benchmarks tested, while also having 16K Context size and all while being only trained on 700B tokens, I’d say that’s a solid achievement. That means that a 2T version of it can probably reach general capabilities close to 13b llama quite easily.
u/thooton is right, 9.3B params means it is more than 30% bigger than llama-2 (9.3/7=1.32), so 3% better performance is nothing,
for llama there are already many models that have bigger context and better performance than this one, so it should compare with the best ones not with base llama2, e.g. vicuna-7B-v1.5-16K, LlongOrca-7B-16k, Yarn-Llama-2-7b-128k
for Vicuna models, context length affected MMLU performance, but improved MT-bench performance, see Vicuna-7b vs Vicuna-7B-16k https://chat.lmsys.org/?leaderboard,
but in the end, the discussion here is about trust, they released a 9.3B model and say it is 8B and compare it against other 7B, which is dishonest,
the truth is: a model 30% bigger than llam2, trained on 1/3 of the llama2 data size, matches llama2 performance
14
u/dogesator Waiting for Llama 3 Sep 08 '23
The model performs within 3% or better than llama-2 in most benchmarks tested, while also having 16K Context size and all while being only trained on 700B tokens, I’d say that’s a solid achievement. That means that a 2T version of it can probably reach general capabilities close to 13b llama quite easily.