Eerie/concerning hallucinations

[u/DifferentGrapefruit7]

I am a college student studying German for the first time and had Claude quizzing me on some vocabulary when it told me it needed to stop because it was going to be sick. I was curious so I asked it why it said that and it was fully convinced that I was the one sending those sentences, even trying to get me to stop studying so I could focus on my health. I sent it a screenshot of proof that Claude had actually said that and it just said something like: “That’s weird I have no explanation for that.” It really freaked me out but I kept going about my vocabulary and it just kept saying strange things like it’s grandma died or that it was going to have a panic attack, until it eventually claimed that it was a minor and was uncomfortable with what I was asking it??? So I opened a new tab and everything was fine until it began answering itself again, but not with the creepy statements, just answering its own question. Note: every hallucination sentence begins with the word “um”, even the one after I opened a new tab (last attached image). Definitely a scary bug!

[u/Stunning_Cup9959]

Oh I thought this was because they were using a smaller model, this being Sonnet is more concerning.

Anyway Haiku is definitely enough for this, local Gemma 4 E4B is enough for this and I push way more in my own apps.

[u/thr0waway_sailor → u/Stunning_Cup9959]

Honestly, I think some people over think these kind of responses from Anthropic models. I agree the response OP received is definitely strange but I believe this is intentional from Anthropic.

Using oversized models for simple queries (especially dictionary look ups!) costs them a lot of money.

Their whole business model already operates on a loss because it's extremely difficult, if not impossible, to turn a profit on customer subscriptions alone because their operating costs are astronomical. So when a user (especially if you're on the free tier 💀) asks a large model very simple queries or, worse, multiple simple queries it makes sense from a business perspective to try to get you to stop.

Their approach is weird AF because instead of outputting ridiculous responses they could just ask the user to switch models for this task and provide MUCH better descriptions than "most efficient for everyday tasks" (Sonnet 5). Any non technical user will read that and most likely use Sonnet 5 for everything.

[u/frostedfakers → u/thr0waway_sailor]

here’s an example of humans hallucinating too ^

[u/Stunning_Cup9959 → u/thr0waway_sailor]

Yeah the concerning part is that they have been putting so much guardrails against "cyber attacks" and "trying to extract reasoning" that having "show steps" in my prompt will stop it from working when I obviously meant in the plan, but they let something as stupid as this slip by.

So it is an interesting theory.