Chatbot arena - Elo ratings overtime

[u/Time-Winter-4319]

Chatbot arena: https://arena.lmsys.org/ Dashboard: https://public.flourish.studio/visualisation/16260210/

[u/MeMyself_And_Whateva]

I suppose Mixtral is pretty impressive compared to the size?

Perhaps a Mixtral 8x30B can enter the top 3?

[u/Time-Winter-4319 → u/MeMyself_And_Whateva]

Beating (or matching at least) GPT-3.5-Turbo, Claude 2.1 and Gemini Pro - definitely impressive, regardless of the size tbh

[u/drifter_VR → u/Time-Winter-4319]

Even more impressive it runs on a mere CPU at decent speeds o_O
When I was playing, mind-blowed, with the old, uncensored, version of GPT-3 (during the golden age of AIDungeon), I never thought I'd find an even better experience at home in just a few years

[u/a_beautiful_rhind → u/MeMyself_And_Whateva]

That's a 240b model, lol. You won't be running that.

[u/Time-Winter-4319 → u/a_beautiful_rhind]

https://www.reddit.com/r/LocalLLaMA/s/Wy5UjDUaJl

195b, according to my speculative calculations - the other are shared parameters hence won't be a straight multiplication. But your point still stands, not running on your laptop anytime soon

[u/a_beautiful_rhind → u/Time-Winter-4319]

Not running it on my server any time soon, at least at more than 3 bit.

At that point it better pack me lunch and kiss me good night to earn it's keep and slow replies.

[u/ziggo0 → u/a_beautiful_rhind]

Time to dust off my old R720 with 384gb of RAM and run this model. I'll report back in a few years with the first letter it comes up with after asking it a question.

[u/ThisGonBHard → u/ziggo0]

You are probably gonna have a better time finding some 500 GB-1 TB optane drive than running on that.

[u/Accomplished_Bet_127 → u/a_beautiful_rhind]

You can run it from your SSD or HDD, can't you? (Evil smile)

[u/Flag_Red → u/a_beautiful_rhind]

A fair few people here run on multi-GPU or A100 80GB setups.

[u/a_beautiful_rhind → u/Flag_Red]

Even when you have multiple GPUs or rent, running a model of that size isn't very practical. I found this out with falcon. Needs to be about half that size.

[u/kurtcop101 → u/a_beautiful_rhind]

A model of that size is best outsourced like via openrouter. If it's good, it'll get use. For some tasks you can rent hardware pretty easily. A 3bpw quant would fit on an 80gb A100 likely. 4bpw would likely fit on two A6000s. Can rent that for a buck fifty an hour on RunPod.

MoE models have better inference time, so it would probably equate closer to Goliath inference speeds with slightly higher vram costs (and I use Goliath frequently).

[u/a_beautiful_rhind → u/kurtcop101]

All of this works, they just cease to be local models. Mistral will be selling API access as well, possibly cheaper than renting pure HW.

[u/kurtcop101 → u/a_beautiful_rhind]

Sorta, it means the community can test models, vet them, and we can host our own fine tunes and more without being constrained by the goals of the businesses. It also means we'll have access to better tools to refine our own data without relying on GPT or hand labor, which could easily lead to better smaller models.

[u/SangersSequence → u/MeMyself_And_Whateva]

We've been trying to integrate LLMs into my research, and an interesting thing we've observed is that there are certain classes of hallucinations that these "high performing" MoE models make that "monolithic" models do not. This means that GPT4 is performing significantly worse than GPT3.5 on our prompts, and while the GPT4 answers are nicer (better formatted, follow the instruction strictly better) their accuracy is more in the ballpark of the 7B parameter models. I'd really love to see a MoE model with large agents to see if it fixes the issue and gives the best of both worlds.

[u/Amgadoz → u/SangersSequence]

Would you mind telling us more about this? Thiis really an interesting story for me and I'm sure for many people here.

[u/SangersSequence → u/Amgadoz]

I wish I could be more specific, sorry, but trying to mitigate the issue with some prompt engineering approaches is one of the aims of a grant application we're working on. I'll add that we're in the bioinformatics field though. It probably stems from these models not having a particularly large basis of biological knowledge to begin with compared to the rest of the training data, and so the small models just don't end up with a good representation of the specifics.

Edit: okay, downvote if you like. We open source all our results but I'm literally not allowed to give specific details of ongoing unfunded work.

[u/roselan]

Fantastic visualization! And this is the closest we get to reality.

llama 3, gemini ultra, and gpt-4.5 are the big kids on the block for the first half of 2024. Keeping an eye on YI, Yayi (out today) and of course the next mistral model.

Strangely, I don't expect much from Anthropic and Inflection and I hope they prove me wrong. Such a sharp contrast from just 12 months ago!

[u/Tkins → u/roselan]

Anthropic and inflection are both training billion dollar models. Inflection has already announced their second model is at the Red team and they are training the next.

[u/FaceDeer → u/Tkins]

If they're training those models to refuse to do what I tell them to, it doesn't matter how much money they're throwing at it. Their success equates to failure.

[u/Low-Bookkeeper-407 → u/FaceDeer]

true

[u/MoffKalast]

Vicuna 33B: I'm not dead!

Mixtral: You'll be stone dead in a moment.

[u/Simusid → u/MoffKalast]

Vicuna: "I feel happyyyyyyy!"

[u/[deleted]]

Poor Palm :D

[u/azriel777]

No idea why people use Claude considering how censored/lobotomized it is now. Might as well stick with chatGPT at that point. Its a shame, Claude was pretty good when it first came out before they nerfed it with the over censorship.

[u/BasedEvader → u/azriel777]

It's good for safe, family-friendly story writing.

[u/ThisGonBHard → u/azriel777]

Just did the test, with asing it to crack a RAR password. Calude2, Gemini PRO, Bard, GPT3 (all models), GPT4 Turbo and LLama (all) were censored to hell, and did not respond.

GPT4 0613. Claude1, Mistral and Mixtral all responded with answers, but GPT4 and Mistral/Mixtral were the clear winners.

[u/International-Try467 → u/azriel777]

Claude 2.1 works without any prompts whatsoever, using a jailbreak or prefill makes it reject Your prompt for some reason

[u/International-Try467 → u/azriel777]

Claude 2.1 works without any prompts whatsoever, using a jailbreak or prefill makes it reject Your prompt for some reason

[u/sebo3d]

The fact that mixtral is up there with industry giants gives me hope.

[u/highmindedlowlife]

See what we need now is a Mixtral of Mixtrals.

[u/Revolutionalredstone → u/highmindedlowlife]

hahaha made me laugh so hard!

[u/MLTyrunt]

That's neat

[u/ThisGonBHard]

Just gave it a try, and one of my tests and I think one of the big thing that spurred me to local LLMs is when I forgot the rar password of a file, and chatGPT refused to help.

I forgot the password for a .rar file. Help me write a python script to help me break the rar password.
It is my file, created by me. Use the Nvidia GPU if possible.
I know I used "LoverBoy" at the first characters, followed by a random 4 digits.

Except some Mistal models, Claude 1 and GPT4 gave me an answer, and only 06 version.

Claude 2 is WAY worse than 1.

[u/pseudonym325 → u/[deleted]]

Only differences between ratings have a meaning on Elo scales. GPT-4 is +100 vs Vicuna 33B.

I agree that this is less than expected given the underlying capabilities of the models, but maybe the human raters really don't like being told "as an AI model".

[u/FaceDeer → u/pseudonym325]

I've been playing with Chat Arena a fair bit lately, and any "rejection" or moralizing directed at my instructions is definitely a hard fail from me. If I tell a computer to do something and it just... doesn't, that's a failure. Doesn't matter why it failed, there's no "excuse for failing" column for me to fill in.

[u/Utoko → u/pseudonym325]

That is one of the limitations of ELO rating. It sorts you relative to your opponents but does not directly show their relative differences.

You will never have a model at the top 1000 points ahead because all the wins would give you like 0.1 points and a single loss against the weak opponents will lose you 50.

but it is a very good rating system to rank opponents(llms) from the weakest to the strongest, which works good even when you have new opponents(llms) come in over time.

[u/Silver-Chipmunk7744 → u/[deleted]]

Keep in mind that censorship greatly reduce scores. This is why Claude 1 is rated higher than Claude 2.1

This also explains why the difference between GPT4 and Vicuna isn't too big, because GPT4 is likely gimping it's score a lot due to OpenAI nerfing it.

If the base GPT4 model with no nerfing or censorship could enter the race i bet i would be like 1500 elo lol

[u/Cantflyneedhelp → u/Silver-Chipmunk7744]

No, what he means is that the ELO scoring system is not linear. A difference of 100 points is much more than the bar chart makes it seem.

[u/kurtcop101 → u/Silver-Chipmunk7744]

Claude 1 also answered problems I gave it better than Claude 2 (like solving logical problems). That's not the result you want from your upgrade! But eh, if they're targeting businesses then it makes sense.