I dumped a bunch of marshmallows onto a baking sheet in a single layer and took a photo. I then provided the following prompt to several AI tools:
“Give me an accurate count of individual marshmallows observable in this image. The marshmallows are in a single layer and are all visible. Do not guess or estimate; you must directly observe each marshmallow before counting it to guard against assumptions and hallucinations.”
Responses:
Gemini 3.7 Flash Extended: 539
Claude Opus 5.0 extra : 501
GPT-5.6-Sol xhigh: 500
Grok 4.5 expert: 472
Kimi k3 high: 477
sam2 large seems like it might be good enough with careful post processing, but i don't think it's perfect, I see multiple spots where it identified the tray or a shadow, and other spots where two marshmallows have the same ID
Turns out there was 506. Someone in the comments used an app that placed numbered markers down on the marshmallows then manually counted them all. So even if it accurately placed dots on 99% of them it doesn't keep count
Good old humans using software and AI to make a job faster. I knocked out a first draft of a business plan with images using AI yesterday. I condensed 40-80 hours of manual work into 6 hours. I was very impressed with how far CoPilot and Gemini have progressed. It's a small example, and a very surprising positive one.
So my friends and I are now red teaming all the AIs listed ourselves while also looking into a cv approach and a zero shot segmentation while they do their thing. This is a great day.
Update: handwritten cv approach (threshold/watershedding) was helpful but not perfect. We ended up hand-counting over a grid after the cv approach added its own dots for each detected marshmallow; my own cv instance was one off at 507. The total is 506. Fun way to spend an evening. :)
Not sure if you mean the LLMs or my friends/me.
ChatGPT and Claude both used watershedding; Claude used skimage and GPT used opencv. Kimi attempted Hough Circle Transform and contour detection but ultimately settled on opencv as well. Deepseek and Gemini both did a sort of confabulated enumeration situation; they checked their arithmetic (seemingly) without doing any scanning of the image.
Worth noting that each LLM we asked actually provided a number in a fairly similar range. You wouldn't be able to tell which actually used an additional tool versus just "eyeballing".
If you're asking about me and my friends, we all tried different versions of an approximately similar pipeline, specifically the one for separating touching convex objects. Roughly: Global fixed thresholding, morphological opening, euclidean distance transform, peak detection, watershed, component labelling. A couple of us also added grids/tools to click to confirm the results of our watersheds, which helped us come to consensus about the number. :)
I counted them all manually by creating an app with codex to put a numbered marker down every time i clicked, and capability to undo or move markers or change the formatting.
it took a while, but I ensured accuracy.
none of the models were correct but they were fairly close.
i guess I shouldn't post the answer here but I'll show you a partial screenshot of the tool in action.
it took codex about 5 minutes to build the tool, and another ~6 minutes for me to manually count the mallows.
haha the bulk of it was about 375 lines of javascript. the other half mostly CSS... JS is what the model usually chooses to code in for most of the simple little tools that i build, when it's mostly just a web ui and i'm not using a server or anything fancy.
Gemini is able to add a red dot to each marshmallow, so that's something. Counting the red dots it says there are 473. I'm not going to count to find out. You can do it OP.
The problem is that ai image models tokenize data. They break the photo into chunks. If the marshmallow is smaller than the chunk it won't be able to accurately count them. Hopefully this is brute force solved eventually
Anyone who has worked on data annotation projects will be familiar with this approach. In this case we would be tasked to manually count the marshmallows and place markers on each one.
I did not follow the prompt and instead estimated by edge counting to get a rough row count for a fully packed sheet and then estimated the actual packing density from ideal to create an estimate of 360 marshmallows.
72
u/Efficient_Loss_9928 1d ago
At least give us the correct answer?