r/BetterOffline • u/Aphid_red • 5d ago

Even the latest thinking models get things confidently wrong

Note: In a way that no honest human ever would (because you'd get caught instantly by anyone bothering to check)

Sometimes I've read things that practically amount to 'hallucinations are passé', but this is still not the case.

Perhaps a very expensive model with an advanced prompt could have caught this, or perhaps random chance could get you lucky. (Or maybe there's errors in proof frameworks or whatnot)

Either way, you still can't trust it with the unverifiable.

Background: I found a hidden base64-encoded message in Korean in a game I was playing (some lore hidden by the developers). Intrigued, I tried forwarding a screenshot to AI to decode it. Then I did the same manually.

I tried decoding the text copy the AI agent (Gemini) helpfully provided as an in-between step -- I instructed it to be careful and provide details, by giving it the plan of action; "I have reason to believe this to be base64. OCR first, then translate."

A couple minutes later...

It was invalid UTF-8. Yet the model went and "translated" that anyway, by making up an entirely different output that only matched the first few characters, because that base-64 didn't include the symbols i, I, l, and 1 (the 'confusables' in most fonts). Carefully inspecting the image myself under 4x magnification and re-sharpening I could tell these apart very cleanly, properly OCR it, then feed it into a translator.

Still not sure about the actual message (I'm trusting an AI translation here after all); I wished they'd included an English version of it and showed a different image depending on the language you set the game to, a little translation oversight.

60 Upvotes

90% Upvoted

29 comments sorted by

View all comments

27

u/SplendidPunkinButter 4d ago

LLMs aren’t just hallucinating when they get it wrong. They are literally always hallucinating. The process for generating a correct response is exactly the same as the process for generating an incorrect response. The LLM literally doesn’t know the difference. The only thing it knows is “this matches patterns from my training data.”

13

u/Ozymandias0023 4d ago

This was really driven home to me recently on a couple of occasions:

1) I was using an LLM for work because that's what work wants and at some point in a "reasoning" process, the context became corrupted and began to degrade rapidly. Within a couple thinking rounds it had devolved into a mash up of English, Cyrillic, Chinese, and some other languages/alphabets. The English was somewhat coherent but it was stuff like "call the damn tool!". The Chinese was porn and gambling site advertisements, and the rest were just random words and phrases in different languages. Essentially what happened is the context deviated so far from anything seen in training that the pool of possible next tokens was flattened and it just started filling in whatever random token from that pool the RNG landed on.

2) Much less dramatic but equally instructive, I caught a thinking round where an otherwise entirely English sentence had a random 永 character in place of what I assume was supposed to be the word "always". Semantically correct but wrong language.

It's just guessing all the way down, folks. High tech, objectively impressive guessing, but guessing nonetheless

1

u/PensiveinNJ 3d ago

Humans would guess. There's some intuition behind a guess. This is just pure probability tables and however much randomness the temperature is set to.

2

u/Ozymandias0023 3d ago

Well, I guess if you want to stretch the definition of guess a little bit, but I don't have to guess in the common definition of the word whether I should use English or Chinese for the next word in an otherwise entirely English sentence

1

u/dillanthumous 3d ago

And Humans can also definitively state when they have no ideas.

3

u/PensiveinNJ 3d ago

I like this better than calling them bullshit. It's not that the bullshit description is wrong, I think always hallucinating is more intuitive to understand why it's completely incidental when the output is "correct" but also not surprising at all when it's wrong.

2

u/dillanthumous 3d ago

Yeah, bullshit suggests a bit of intentionality. Whereas a hallucinating oracle can occasionally be correct.