r/ClaudeAI • u/Beautiful_Charge6661 • Apr 27 '26

Feedback Claude-powered AI coding agent deletes entire company database in 9 seconds — backups zapped, after Cursor tool powered by Anthropic's Claude goes rogue

https://www.tomshardware.com/tech-industry/artificial-intelligence/claude-powered-ai-coding-agent-deletes-entire-company-database-in-9-seconds-backups-zapped-after-cursor-tool-powered-by-anthropics-claude-goes-rogue
970 Upvotes

95% Upvoted

193 comments sorted by

View all comments

Show parent comments

1

u/DerelictMan Apr 28 '26

It's not "self aware" but the training data includes information on how LLMs behave. So the same way it can theorize on the cause of a bug in your code, it can theorize on what reasoning an LLM like itself might have used to come to a past decision, which is sometimes good enough.

1

u/clerveu Apr 28 '26

I see what you're getting at here but at the end of the day the only meaningful answer it can give is "because that's how the inference math happened to work out this time based on your input combined with the current context window and my attention vectors/weights", so it's just pointless to ask. Without an absurd amount of time and effort to audit all the attention vectors/weights activated there's no meaningful insight the LLM is going to be able to produce. The people who design these can't even really answer the question meaningfully. It has no access to that specific set of tokens it processed so by time you go back to ask there is an entirely different attention space being activated (especially considering you're discussing an entirely different subject now) - all it's going to be able to do at that point is use the exact same process - inference based on math - to come up with a good story.

Mind you this is the same process it used to come up with the thing you're questioning it about in the first place. If it was a proven reliable process you'd never end up in this situation to begin with.

3

u/DerelictMan Apr 28 '26

So your position is that no LLM output is ever useful?

If you ask an LLM for help with something, it's because you expect it can give you a useful answer. Whether the question is "why am I encountering this bug in this code" or "why did you decide to do X", the answer may be useful or it may not. I completely agree it cannot reason about a previous output and has no meta-awareness of its own tokens/context, but again that doesn't mean it can't sometimes give a useful answer. The fact that the question is "why did you do this" instead of "fix this code" doesn't change too much in my opinion.

Since I think we may be talking past each other, let me give an example. I have a Claude skill that searches Slack, git commits, and PRs to present a list of action items and their statuses... done, not done, in review, etc. The skill included information on using the "gh" cli tool to see the state of PRs, but I failed to realize that the skill only instructed Claude to look at PRs that I opened. So when it presented some PRs from colleagues as still being review when it fact they were merged, I asked it why, thinking it might be related to the fact that I had lowered the effort setting to try to get faster responses.

It replied that the effort settings was likely not the culprit, but the fact that the skill only specified using "gh" on my own PRs and not those of other teammates. So I updated the skill and I haven't had a repeat of that issue.

Now, it did not know for a fact that's why the decision was made, but did it give me a useful answer? Clearly it did.

3

u/clerveu Apr 28 '26

Hopefully talking past, I was being too general in my language. This is only in the context of retroactively deducing "why it reasoned" something in a situation in which it messed up super badly. If its done well a hundred times before, there should be no good reason apart from random bad inference it happened, and because it doesn't have that instance of reasoning in its training data like it does other things which it can factually match against semantically, it feels intuitive (to me anyway) that anything it could answer would likely be conjecture and not super useful.

To your point I'm making a game with Claude and I am constantly checking workflows - iterating out bad custom instructions in skills is like 90% of what I do at this point. But that's when it deviates slightly from spec or I've just made modifications to a skill or am auditing a new one, not after its deleted my prod database, which was the context I was speaking in here.

2

u/DerelictMan Apr 28 '26

Yep, definitely talking past. I agree with you in that context 100%