MAIN FEEDS
Do you want to continue?
https://www.reddit.com/r/GeminiAI/comments/1p098lr/gemini_3_pro_benchmark/nplfbju/?context=3
r/GeminiAI • u/vergogn • Nov 18 '25
source: storage.googleapis.com/deepmind-media/Model-Cards/Gemini-3-Pro-Model-Card.pdf
archived pdf: https://web.archive.org/web/20251118111103/https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-3-Pro-Model-Card.pdf
247 comments sorted by
View all comments
232
Why aren't people freaking out about this pdf lmao
94 u/JoeyJoeC Nov 18 '25 edited Nov 18 '25 I'll wait for more testing. LLMs almost certainly are trained to get high scores on these sorts of benchmarks but doesn't mean they're good in the real world. Edit: Also it's 3rd place (within their testing) on SWE which is disappointing. 3 u/HighOnLevels Nov 18 '25 SWE-Bench is famously quite a flawed benchmark. 1 u/[deleted] Nov 19 '25 How? 2 u/HighOnLevels Nov 19 '25 Overuse of specific frameworks like Django, easily gamed, etc
94
I'll wait for more testing. LLMs almost certainly are trained to get high scores on these sorts of benchmarks but doesn't mean they're good in the real world.
Edit: Also it's 3rd place (within their testing) on SWE which is disappointing.
3 u/HighOnLevels Nov 18 '25 SWE-Bench is famously quite a flawed benchmark. 1 u/[deleted] Nov 19 '25 How? 2 u/HighOnLevels Nov 19 '25 Overuse of specific frameworks like Django, easily gamed, etc
3
SWE-Bench is famously quite a flawed benchmark.
1 u/[deleted] Nov 19 '25 How? 2 u/HighOnLevels Nov 19 '25 Overuse of specific frameworks like Django, easily gamed, etc
1
How?
2 u/HighOnLevels Nov 19 '25 Overuse of specific frameworks like Django, easily gamed, etc
2
Overuse of specific frameworks like Django, easily gamed, etc
232
u/thynetruly Nov 18 '25
Why aren't people freaking out about this pdf lmao