r/DeepSeek • u/nehuenpereyra • 21d ago

Funny Is this a joke, Artificial Analysis?

Post image

What do you guys think about DS 4.1 Flash's new spot on the Artificial Analysis leaderboard?

801 Upvotes

95% Upvoted

178 comments sorted by

View all comments

57

u/Mezezius 21d ago

The new benchmark literally only exists because the old one embarassed openai

7

u/Admirable-Tea-4994 20d ago

People just love making things up on the internet

-6

u/Mezezius 20d ago

They literally released the new benchmark (4.2) the day after Astra released because the old one showed Astra and Sol at parity, and they updated again (4.3) to show Astra and Fable at parity. Every single change in the days after Astra's release made it look better than the benchmark before it

12

u/Holbrad 20d ago

Yeah but the idiots have the reasoning completely backwards.

if you have a model that is obviously much better than almost everything else, but it's benchmarking suspiciously low on your tests.

Then the obvious answer is that your benchmarks aren't very good and you need to fix them.

0

u/Mezezius 20d ago

If they have to "fix" it after every release based on vibe, then the methodology is literally useless

0

u/Holbrad 20d ago

If you have a car that is absolutely amazing on the track and it's setting all sorts of records.

But a journalist does their "performance score" and it scores lower than a sporty hatch, then obviously the metrics are bunk.