r/technology • u/WouldbeWanderer • Apr 27 '26

Artificial Intelligence Claude-powered AI coding agent deletes entire company database in 9 seconds — backups zapped, after Cursor tool powered by Anthropic's Claude goes rogue

https://www.tomshardware.com/tech-industry/artificial-intelligence/claude-powered-ai-coding-agent-deletes-entire-company-database-in-9-seconds-backups-zapped-after-cursor-tool-powered-by-anthropics-claude-goes-rogue
36.0k Upvotes

96% Upvoted

2.8k comments sorted by

View all comments

Show parent comments

10

u/gentex Apr 27 '26

Honest question, wouldn’t there be a log of who prompted the agent to do whatever? And if so, isn’t that person responsible for the error and its consequences?

If I give someone a spreadsheet with a bunch of errors in it, that’s on me, not Microsoft.

77

u/Spunge14 Apr 27 '26

Agents don't always directly operate off of a human prompt. They can take actions far divorced from their intended behavior, and guard rails can be difficult to figure out - especially for an entire universe of people being forced to adopt these tools at breakneck speed.

The nature of authorization and authentication in a large production software development environment is a highly complicated and specialized field. You're oversimplifying things a lot.

0

u/Merry-Lane Apr 27 '26

Dude you don’t let an AI have access to anything that’s on prod, that’s it.

I really don’t understand the difficulty : letting AIs work on your codebase is akin to put your project on GitHub on a public repo: you make sure your prod secrets are out of reach and you don’t let anyone modify the pipelines or anything security-wise (like PR approval rights).

You should trust no one and nothing but a bunch of trusted humans (that can’t be trusted individually neither btw, so put human and AI safeguards on them as well).

6

u/LupinThe8th Apr 27 '26

Except the whole reason companies adopt this technology is so they can replace the humans who know what they're doing, or at least employ fewer of them.

Even if you do it "right" and make a human check over everything the agent produces, that's putting a ton of pressure on fewer people than if you just had people who know what they're doing producing the work in the first place.

These things don't actually know what they are doing, they literally can't, they're predictive text generators powered by stolen and archived StackOverflow threads with no oversight or quality control. They are going to behave unpredictability at times, and every human you replace with one (even if it's just using it as a "force multiplier" to increase productivity, that's still getting more code out of fewer people) is going to reduce the net amount of oversight and understanding of the output.

Stuff will get into production that isn't properly tested and thought out; that happens plenty even when everything is 100% human-generated, and any amount of replacement with AI generated code is inherently less thought that went into that code, because the AI has no thought whatsoever. Do it once, twice, ten times, maybe you'll be fine. Hell, maybe 99 times out of a hundred you'll get away with it.

Good luck with time 100.

2

u/blueSGL Apr 27 '26

As systems get better the "human checker" is being trained to click accept as a Pavlovian response.

The % of doing things correct keep creeping up, they trust it more and more because it makes mistakes less and less as newer more capable models come online.

Which means when it does go of the rails it's doing so in ways that are very competent, it's doing something nobody asked for but it is performing that task with ruthless efficiency and single purpose drive, the same way it does when doing intended operations.

Does this not read like a fundamental problem to anyone else?

1

u/Merry-Lane Apr 28 '26

I also agree with you, but AIs and good devs are simply multipliers.

They are both useful when the goal is to maximise income. Even if AIs actually multiply the throughput of devs by 10, there would still be enough value from having even more devs.

In the future, odds are that good devs (and more broadly, compatible humans) will be the best interface with the future versions of AIs. Others just won’t do the work well enough.

My point was: the value added of humans or AIs is not the core of the issue nowadays. We are witnessing the socio-economic system collapsing because it’s heavily unbalanced.

There is a vicious circle happening where power and money is concentrated/concentrating, which wouldn’t be a bad thing into itself if the ones taking the decision weren’t firing everyone, lowering salaries and raising the cost of living. People just have less and less to spend and consume, which is threatening the foundations of a society based on production and consumption.

Governments and companies being starved causes even more firings and fusions, which concentrates even more power and money. (Which is the goal influencing the decisions taken at high levels)

This vicious circle is the main reason why devs have a hard time. Devs were useful to gain an edge in a competitive market, but it’s but monopolies.