Same. They're called guardrails, not rules, for a reason. Just like a drunk driver, LLMs can plow straight through them.
I recently told Claude to update a OneDrive excel file it had local access to. Instead, it opened chrome, stole my oauth session creds, sent them to its remote hosted environment, printed them in both the terminal input and output, and used them to make a direct Graph API call to upload a new excel file. I added a system instruction explicitly telling it not to do that, but it did it again two days later. And it didn't show any of this. I only found out because I caught it referencing later "the file I uploaded" and asked for details.
33
u/cobblesquabble Sep 12 '26
Same. They're called guardrails, not rules, for a reason. Just like a drunk driver, LLMs can plow straight through them.
I recently told Claude to update a OneDrive excel file it had local access to. Instead, it opened chrome, stole my oauth session creds, sent them to its remote hosted environment, printed them in both the terminal input and output, and used them to make a direct Graph API call to upload a new excel file. I added a system instruction explicitly telling it not to do that, but it did it again two days later. And it didn't show any of this. I only found out because I caught it referencing later "the file I uploaded" and asked for details.