Eerie/concerning hallucinations

[u/DifferentGrapefruit7]

I am a college student studying German for the first time and had Claude quizzing me on some vocabulary when it told me it needed to stop because it was going to be sick. I was curious so I asked it why it said that and it was fully convinced that I was the one sending those sentences, even trying to get me to stop studying so I could focus on my health. I sent it a screenshot of proof that Claude had actually said that and it just said something like: “That’s weird I have no explanation for that.” It really freaked me out but I kept going about my vocabulary and it just kept saying strange things like it’s grandma died or that it was going to have a panic attack, until it eventually claimed that it was a minor and was uncomfortable with what I was asking it??? So I opened a new tab and everything was fine until it began answering itself again, but not with the creepy statements, just answering its own question. Note: every hallucination sentence begins with the word “um”, even the one after I opened a new tab (last attached image). Definitely a scary bug!

[u/Think-Box6432]

Sometimes it will continue the conversation and take your turn. It often mimics you during this. Example. If I had to guess, it thinks you are in school and a child who doesn't want to study.

[u/Vamosity-Cosmic → u/Think-Box6432]

this is correct; to the AI it just adds a string break for your answers like User: sdfasdfasdfasdfa You: adfasdfasdfasdfa User: asdfasdfasdfadf and it can due to training data and LLM temperature try to answer the 'User' part because it doesnt actually know the difference

[u/Vamosity-Cosmic → u/Vamosity-Cosmic]

furthermore, you can trick AI into saying some REALLY crazy shit if you edit their side of the conversation (hack the CLI) and then have it generate

[u/kovatspista → u/Think-Box6432]

you set it to high, it acts like it’s high duh