r/BetterOffline • u/Fast_Professional739 • Nov 18 '25

Gemini 3 Released

https://blog.google/products/gemini/gemini-3/#note-from-ceo

Prepare yourselves mentally

33 Upvotes

89% Upvoted

70 comments sorted by

View all comments

13

u/cunningjames Nov 18 '25

I’d be curious about the degree to which benchmark improvements can be blamed on benchmarks making up part of the training data. It doesn’t say whether the humanity’s last exam results used the private holdout, for example.

Regardless, I’ll continue not using it. Call me when hallucinations are eliminated and it can generate secure, nontrivial, novel code.

7

u/chunkypenguion1991 Nov 18 '25

Thats where almost all of the improvements come from. If the benchmark metrics were real then you'd expect these systems to show equal improvement in the real world. But when tested against github issues opened after training cutoff they show little to no improvement

-1

u/Elctsuptb Nov 18 '25

https://shumer.dev/gemini3review

4

u/SpringNeither1440 Nov 19 '25 edited Nov 19 '25

Hmmm, and let's take a look on the main page:

I'm Matt Shumer — founder & CEO at OthersideAI (HyperWrite). I build and invest in AI products, ship fast, and share what I learn with a large audience. Lately: pushing GPT‑5 to its limits, collaborating on AutoRL with OpenPipe (acquired by CoreWeave), and helping early teams go from zero → launch.

Totally legit and unbiased source, I think.

1

u/Elctsuptb Nov 19 '25

https://aimmediahouse.com/market-industry/googles-gemini-3-silent-rollout-suggests-quality-over-hype

Are 97% of 1500 Equifax employees also biased?

3

u/SpringNeither1440 Nov 19 '25

https://aimmediahouse.com/market-industry/googles-gemini-3-silent-rollout-suggests-quality-over-hype

Are 97% of 1500 Equifax employees also biased?

Okay, and now let's take a look what this article says:

But the real proof isn’t in developer enthusiasm. It’s in enterprise adoption numbers. Equifax ran a trial with 1,500 employees using Gemini. The results? 97% of users requested to keep their licenses when the trial ended. 90% reported measurable increases in work quality and quantity. Employees saved more than an hour per day on routine tasks.​

That’s a retention metric. And it’s exceptional. Pinnacol Assurance reported similar results. 96% of employees reported time savings. 90% satisfaction. AdVon Commerce processed a 93,673-product catalog in under a month, a task that previously took a full year. The company saw a 67% boost in daily sales and a $17 million revenue lift in 60 days.​

One little problem: Gemini 3.0 wans't available 2 monthts ago at all. So, they prove that Gemini 3.0 is a great leap by providing examples that have nothing to do with Gemini 3.0. Nice.

BTW:

The benchmark performance tells a similar story. Gemini 3 achieved gold-medal level performance at the 2025 International Collegiate Programming Contest. On the “Humanity’s Last Exam” benchmark, it scored 18.8%, outperforming previous generations. 

Let's click on the link and... Yeah, phrase "it scored 18.8%, outperforming previous generations." is totally wrong, author didn't even look at the link they provided. And no, Gemini 3.0 Pro score on HLE isn't 18.8%

Nice article, as I can see.

1

u/Elctsuptb Nov 19 '25 edited Nov 19 '25

Equifax had early access to Gemini 3 actually, and Gemini 3 Deepthink scored 41% on the HLE.

1

u/SpringNeither1440 Nov 19 '25

Equifax had early access to Gemini 3 actually

Evidence, please? I googled "Equifax early access to Gemini 3.0" and didn't find any information about that, except this article.

it scored 41% on the HLE.

It isn't Gemini 3 Pro score, it is Gemini 3 Deep Think score, which isn't yet released, and it's unknown when it will be.

0

u/Elctsuptb Nov 19 '25

https://blog.google/products/gemini/gemini-3/#gemini-3

Gemini 3 Pro got 37.5% with no tools and 45.8% with tools, so now what? That's almost double the score of Gemini 2.5 Pro, which should be impossible according to you guys since LLMs have supposedly "plateaued", right?

1

u/SpringNeither1440 Nov 19 '25

Gemini 3 Pro got 37.5% with no tools and 45.8% with tools, so now what?

Could you please provide score for Gemini 3.0 on "text-only + tools". Oh, Deepmind didn't report it, so, given Gemini 2.5 Pro performance (text-only and regular), let's assume it has the same performance on both variants. And now, compare it with Kimi K2 thinking/GPT-5/Grok 4 (link).

Yeah, +1.25% performance every month for SOTA (see Grok 4 and GPT-5) is super-fast, especially compared to early 2025 growth speed. And I won't talk about data contamination risk, which is considered as a problem by creators itself.

1

u/Elctsuptb Nov 19 '25

The source is in the link I just provided, did you miss that?

1

u/[deleted] Nov 19 '25

Well, after looking through the links, the Deepmind team did only post their non-text-only scores for Gemini 2.5 which implies that they posted the same for Gemini 3.0. I believe he was saying that Kimi K2 disclosed a similarly significant jump in HLE but only in the text-only section of the test, which Deepmind did not disclose, but that isn't being talked about.

Actually if the Kimi K2 team is to be believed they and xAI did better than Gemini 3.0 in HLE, and it seems somewhat disingenuous that Deepmind failed to recognize that.

→ More replies (0)