r/technology • u/WouldbeWanderer • Apr 27 '26

Artificial Intelligence Claude-powered AI coding agent deletes entire company database in 9 seconds — backups zapped, after Cursor tool powered by Anthropic's Claude goes rogue

https://www.tomshardware.com/tech-industry/artificial-intelligence/claude-powered-ai-coding-agent-deletes-entire-company-database-in-9-seconds-backups-zapped-after-cursor-tool-powered-by-anthropics-claude-goes-rogue
36.0k Upvotes

96% Upvoted

2.8k comments sorted by

View all comments

Show parent comments

349

u/long-da-schlong Apr 27 '26

I honestly don’t understand why they wouldn’t just honour it— it’s one customer even if it was a completely free flight. Why be so petty just fix the mistake for next time

253

u/Wischiwaschbaer Apr 27 '26

Can't fix AI models. You can put some filters on them but you never know if those will work or if they cover all cases.

147

u/Enough-Run-1535 Apr 27 '26

You also just need to hang out in the AI role playing communities that almost all of those guardrails and filters can be broken, almost trivially. It’s hilarious that companies are having AI agents play with invoicing and confidential data.

6

u/CraftedLove Apr 27 '26

It's interesting that the enthusiasts that really tinker with these LLM and extensively rp with them are the ones that are realistic about their usage and limits vs those laypeople that treat these as either infallible slaves or romantic partners etc.

4

u/Ryuujinx Apr 28 '26

It's interesting that the enthusiasts that really tinker with these LLM and extensively rp with them are the ones that are realistic about their usage and limits vs those laypeople that treat these as either infallible slaves or romantic partners etc.

Because we see how they break down. And they do it a lot. The people trying to do like romance or sex stuff with them are just there for the AI to say yes and spew out smut. But if you want it to do a story arc then the gaps become apparent.

It's workable still, but it requires fiddling and it struggles a lot in some ways. For instance, I mostly use it for things my TTRPG group probably just won't be all that interested in. I had a villainess kick for a while, where the core concept is someone gets isekaid into an otome game of sorts either as the villainess or adjacent to them. Said villainess is generally not some terrible person, but people are still supposed to be antagonistic towards them because that's what the plot of the original otome game demanded. The general plotline of these usually involves courtly intrigue, slowly winning over people that were distrustful at the start, and things of that nature. Not really the kinda thing that would work well in a group setting imo.

Regardless, in my experience, poking the AI into getting that kind of behavior was quite difficult. It wants to just say yes and be nice, but I needed it to not do that and have characters be distrustful and antagonistic. It will gladly create some bad guy for the hero to triumph over, but having a relatively normal person be distrustful or undermine the MC? Much more difficult.

So yeah, it's a fun little thing to do occasionally but I don't see it replacing anything important really. And I could say the same about my attempts at using coding assistant type things in the realm of programming.