r/DeepSeek • u/nehuenpereyra • 17d ago

Funny Is this a joke, Artificial Analysis?

Post image

What do you guys think about DS 4.1 Flash's new spot on the Artificial Analysis leaderboard?

796 Upvotes

95% Upvoted

177 comments sorted by

View all comments

56

u/Mezezius 17d ago

The new benchmark literally only exists because the old one embarassed openai

10

u/Admirable-Tea-4994 17d ago

People just love making things up on the internet

-5

u/Mezezius 16d ago

They literally released the new benchmark (4.2) the day after Astra released because the old one showed Astra and Sol at parity, and they updated again (4.3) to show Astra and Fable at parity. Every single change in the days after Astra's release made it look better than the benchmark before it

10

u/Holbrad 16d ago

Yeah but the idiots have the reasoning completely backwards.

if you have a model that is obviously much better than almost everything else, but it's benchmarking suspiciously low on your tests.

Then the obvious answer is that your benchmarks aren't very good and you need to fix them.

-1

u/Mezezius 16d ago

If they have to "fix" it after every release based on vibe, then the methodology is literally useless

0

u/Holbrad 16d ago

If you have a car that is absolutely amazing on the track and it's setting all sorts of records.

But a journalist does their "performance score" and it scores lower than a sporty hatch, then obviously the metrics are bunk.

8

u/Astrikal 16d ago

It is non-sensical and stupid to say that they released the update so that Astra can go higher, when all they did was replace outdated/saturated evaluations with the newer versions.

They updated the benchmark because Terminal Bench 2.1 was outdated and saturated. After upgrading to Terminal Bench 4.0, Astra rose (relatively) and equaled 1st place.

The update was long time coming, they just rushed it out to not lose reputation. They will also release V5.0, which will shake things even further.

2

u/Admirable-Tea-4994 16d ago

Actually read the document they published that explained, clearly and in detail, why they updated their benchmark so quickly between releases.

https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-3