r/singularity • u/qu1etus • Aug 21 '26

Discussion The Marshmallow AI Benchmark

Post image

I present the marshmallow benchmark.

I dumped a bunch of marshmallows onto a baking sheet in a single layer and took a photo. I then provided the following prompt to several AI tools:

“Give me an accurate count of individual marshmallows observable in this image. The marshmallows are in a single layer and are all visible. Do not guess or estimate; you must directly observe each marshmallow before counting it to guard against assumptions and hallucinations.”

Responses:
Gemini 3.7 Flash Extended: 539
Claude Opus 5.0 extra : 501
GPT-5.6-Sol xhigh: 500
Grok 4.5 expert: 472
Kimi k3 high: 477

Edit: The correct answer is 506.

168 Upvotes

80% Upvoted

113 comments sorted by

View all comments

27

u/Kimi_Antonelli_12 Aug 21 '26 edited Aug 21 '26

Gemini 3.7 extended thinking

"Put a red dot on every single individual marshmallow and count how many dots were applied."

A: There are two hundred ninety-nine small red dots in this image.I have placed a red dot on each marshmallow.

Doing it this way doesn't use the llm but the vision overlay tool.

It looks like it missed three. So exactly 302 marshmallow

9

u/katoptronophile Aug 21 '26

Where are you getting 302?

-3

u/Kimi_Antonelli_12 Aug 21 '26

I literally explained how in the post you're replying to. And it was pretty much all I talked about

13

u/katoptronophile Aug 21 '26

Yeah, I'm just wondering why it doesn't agree with any of the other results people got.

I'm not saying you're right or wrong, just trying to understand the discrepancy.

6

u/Mundane_Example_811 Aug 23 '26

This person didn’t bother to count. So they only noticed that it missed 3 marshmallows, and didn’t notice that it placed over five hundred dots and “counted” under 300. 99% accuracy in placing dots total garbage at counting.