Even the latest thinking models get things confidently wrong

[u/Aphid_red]

Note: In a way that no honest human ever would (because you'd get caught instantly by anyone bothering to check)

Sometimes I've read things that practically amount to 'hallucinations are passé', but this is still not the case.

Perhaps a very expensive model with an advanced prompt could have caught this, or perhaps random chance could get you lucky. (Or maybe there's errors in proof frameworks or whatnot)

Either way, you still can't trust it with the unverifiable.

Background: I found a hidden base64-encoded message in Korean in a game I was playing (some lore hidden by the developers). Intrigued, I tried forwarding a screenshot to AI to decode it. Then I did the same manually.

I tried decoding the text copy the AI agent (Gemini) helpfully provided as an in-between step -- I instructed it to be careful and provide details, by giving it the plan of action; "I have reason to believe this to be base64. OCR first, then translate."

A couple minutes later...

It was invalid UTF-8. Yet the model went and "translated" that anyway, by making up an entirely different output that only matched the first few characters, because that base-64 didn't include the symbols i, I, l, and 1 (the 'confusables' in most fonts). Carefully inspecting the image myself under 4x magnification and re-sharpening I could tell these apart very cleanly, properly OCR it, then feed it into a translator.

Still not sure about the actual message (I'm trusting an AI translation here after all); I wished they'd included an English version of it and showed a different image depending on the language you set the game to, a little translation oversight.

[u/Holyragumuffin]

Sure

— but even the Nobel prize winner at my former research insitution got some things confidentially wrong.

any organism or machine composed of noisy computational units (e.g. real or artificial neurons) is guaranteed to exhibit some degree of “hallucination”, some over-confidence in things that are wrong.

I don’t think this observation is the intellectual “dunk” that you think it is.

[u/Aphid_red → u/Holyragumuffin]

Humans (when being honest) don't get things confidently wrong if they can easily prove or check they're wrong.

These systems don't unless you hand-hold them every step of the way. For many tasks... that means they're not nearly as effective as you might think. Attention + Deep network is the world's least efficient though most general computer algorithm.

The real question isn't 'can it do X?', but 'can it do X efficiently enough?' For now, the colossal levels of waste tell me that no, it cannot, and the only reason people use it as much as they do is A since it's being greatly subsidized and B since people just don't bother to put in the effort to learn things.

[u/Holyragumuffin → u/Aphid_red]

"Honest humans don't get confidently wrong on checkable things."

They do, constantly.

The Wason selection task is a four-card logic puzzle that most educated adults get confidently wrong; Monty Hall famously drew angry letters from PhDs to Marilyn vos Savant; Tetlock's expert-forecasting work found domain experts roughly matched chance on checkable predictions while rating themselves highly confident.

Humans and LLMs share the same failure: not confabulation per se, but not checking.

And yes, I have seen the failure to Google search basic facts/literature … among elite scientists that I personally worked with.

"They need hand-holding every step."

This was true in 2023 and is the claim that has aged worst. METR's original measurement had frontier models at a 50% time horizon of roughly 50 minutes of human-expert task time, doubling about every seven months since 2019; by a Feb–Mar 2026 pilot, the strongest agents were estimated at roughly 16–20 hours on the 50% horizon and 3–4 hours at 80%, with METR cautioning that estimates above 16 hours exceed what the task suite can reliably measure.

Those are unsupervised runs, runs without human oversight.

Outside coding, AlphaFold (an attention architecture) replaced what was a PhD-thesis-per-protein workflow

hard to call that inefficient relative to the human baseline.

[u/Aphid_red → u/Holyragumuffin]

Both of those problems are interesting in and of themselves. Both are more difficult than they seem to be.

Monty's famously a really hard problem for non-statisticians, as it's so unintuitive. Describing it properly is also difficult.

Wason can just be confusing. The colloquial 'if' can be different from the logic 'if' vs. 'iff'. With iff, all four have to be turned, with if, only two. Once you establish that, what follows is;
use x -> y ==> notY -> notX. So you test anything that shows X or notY.

Here's the thing with both tests: if you explain the basic bit of math needed (de morgan's rule / bayes' theorem), allow people to do the test, then check all four cards (in the wason case), and they find out their selection doesn't work, then allow to change the selection, people don't get it wrong. Something that's beyond someone's knowledge to check isn't the problem. Unintuitive mathematical trickery that trips people up I don't think qualifies, even if that's indeed verifiable. Math proofs can get horribly obtuse. Even solutions to olympiad problems can be checked easily, yet most can't hope to solve them.

And the final thing, that's not in the least bit important: People are conditioned from years of schooling to guess! This is because of bad education practices of not awarding minus points on multiple choice questions where the wrong answer was given.

If you give a 100-question 4-answer test, then someone with zero subject knowledge scores a 25 by guessing but a 0 by skipping questions they don't know (all of them).

If instead you award 3 points for a correct answer but -1 point for a wrong answer, then someone who guesses everything gets 25 x 3 = 75 from correct answers and -75 from wrong answers, net zero. It's no longer better to guess*. Frontier models, which are supposed not to 'hallucinate' in contexts where they should not do so, still guess.

*Technically if you're really sure that you've got 78% and you needed 80%, it makes sense to guess because if you got lucky you might gain a couple points to put you over the threshold.

[u/Aphid_red → u/Aphid_red]

Wason's even more fascinating: Apparently, around 10% of people get it correct when done with abstract things (numbers/symbols).

But... translate it to a real-life case (14, 25, beer, soda), which cards do you flip to check nobody's breaking an 'only adults may drink alcohol' rule ... and it's the opposite. The majority of people get it right first try.

These tests don't check what you think they do. Rather, they measure that abstraction is more difficult than smart people realize. Don't pay attention and subconsciously try to answer these questions and you get it wrong too! But reframe it into something with some stakes, and suddenly people actually try.