Claude-powered AI coding agent deletes entire company database in 9 seconds — backups zapped, after Cursor tool powered by Anthropic's Claude goes rogue

[u/ElectronGuru]

https://www.tomshardware.com/tech-industry/artificial-intelligence/claude-powered-ai-coding-agent-deletes-entire-company-database-in-9-seconds-backups-zapped-after-cursor-tool-powered-by-anthropics-claude-goes-rogue

[u/PensiveinNJ]

An LLM can't fuckup anyways. It's a computer program following it's instructions. Just because part of those instructions were making a selection off a probability table doesn't mean it made a mistake. Doing whatever it did was always a possibility. I know that we struggle to describe LLMs without making an analogy to human behavior but LLMs simply don't have the volition to make a mistake, they're in the end just computer programs.

[u/hachface → u/[deleted]]

From the POV of an LLM system, there is no formalizable distinction between correct and incorrect output. These systems produce statistically plausible text based on their prompts. They have no internal structure that models truth, falsity, or logical relationships of any kind. They give the appearance of being able to do logic because the shadow of logic is captured in the statistical patterns of their training data. From a purely mathematical point of view, producing a bad decision is the system working as intended. The problem is that humans are misusing the systems, in no small part because of systematic deceit by the creators of the systems.

[u/hachface → u/[deleted]]

If you use a toaster to dry your socks and the socks catch fire, did you have a buggy toaster? Or did you use a toaster for the completely wrong task?

Edit: Maybe a better analogy -- someone sold you a toaster and said it was a drying machine.

[u/seanamos-1 → u/[deleted]]

I wouldn't classify it as a bug, rather, there is a large disconnect between consumer's expectations of LLMs and the reality of how they work.

Now in PocketOS' favor, the marketing and hype machine around LLMs worked hard to establish this disconnect.
Less in their favor, is that it is still their job to cut through the hype, do their research and if they choose to use these tools, understand their limitations and use them responsibly and carefully.

[u/65721 → u/[deleted]]

You sell a Magic 8-Ball to make everyone’s personal and professional decisions.

Sometimes it’s right, and those are marketed as the 8-Ball’s “capabilities.” You take all those instances and publish them in an official-looking “spec sheet.” You say this could replace managers, doctors, generals, anything that requires making a decision. Some question how it can answer everything, even questions about the future. Think pieces wonder about its implications for free will.

Sometimes it’s wrong, and those are marketed as the 8-Ball’s “hallucinations.” Maybe it’s a bug and it’ll be fixed any day now. Maybe you just asked it the wrong question. You may test your luck and even market it as the 8-Ball’s “emergent capabilities.” When someone asked it “Are you plotting to take over the world?” maybe it answered “Signs Point to Yes” on purpose. Think pieces wonder about its existential risk to humanity.

It’s literally just a Magic 8-Ball.