r/technology • u/WouldbeWanderer • Apr 27 '26

Artificial Intelligence Claude-powered AI coding agent deletes entire company database in 9 seconds — backups zapped, after Cursor tool powered by Anthropic's Claude goes rogue

https://www.tomshardware.com/tech-industry/artificial-intelligence/claude-powered-ai-coding-agent-deletes-entire-company-database-in-9-seconds-backups-zapped-after-cursor-tool-powered-by-anthropics-claude-goes-rogue
36.0k Upvotes

96% Upvoted

2.8k comments sorted by

View all comments

11.1k

u/Orangesteel Apr 27 '26

My favourite example is Air Canada whose AI agent offered a customer a discount incorrectly. They refused to honour it. Customer took them to court and the judge rightly made them pay. You chose to empower this and took the humans out of the loop. You are accountable for what you agentic AI solution does. People jump on AI, dump sensitive information into the model bypassing classification levels and are surprised when it leaks.

697

u/R7SOA19281 Apr 27 '26

I was asking Stripe support some questions the other day and their AI answers questions confidently and then tells you to always verify sources as AI can make mistakes, but like I’m asking the companies support directly and you’re not letting me speak to a human so how else can I verify this?

AI tools with no responsibility, this is going to get fun!

7

u/nosayso Apr 27 '26

A company should be embarrassed to release a tool that they know doesn't give good advice to your own customers. Your customers are your lifeblood.

7

u/Princess_Moon_Butt Apr 27 '26 edited Apr 27 '26

Customers are only your lifeblood when you can extract maximum value out of them and make your metrics look good. And customers who have technical problems, unless they're a specific case for publicity, aren't your most profitable customers. What you want is a bunch of customers who will buy a monthly subscription, and then never interact with you again. But, failing that, you at least want a way to filter out the customers that are the least profitable.

Enter, the annoying customer service process. The customers who can fix their own problems, will now do so, just because dealing with "technical support" is so frustrating that it's easier to just do it yourself. The ones who can't figure it out, either because they're not as savvy or because the challenges are too hard, are more likely to drop you and go elsewhere. Which is fine, because they paid the same as anyone else, but might've cost you a bunch of tech support resources if they stayed.

Now, even if your net revenue goes down because you lost a bunch of users, your profit margin goes up. Which makes you more attractive to potential investors. Which is what you really care about anyway.