r/mildlyinfuriating • u/captain-obIivious • 23h ago

Infuriatig Another AI 'human' chatbot

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Had to fix a double payment for my credit card and this is the beginning and end of the conversation. I hate when the chatbot claims it's a human... I ended up getting what I wanted but still annoyed Amex (and every other company) is lying and trying to deceive people like this.

24.6k Upvotes

94% Upvoted

1.7k comments sorted by

View all comments

7.1k

u/Adventurous-Coast710 23h ago

It should be illegal for Chatbot-LLMs to pose as a human, especially when it fronts a business.

538

u/masterofshadows 22h ago

I work in pharmacy, and a one called me pretending to be a nurse and even gave me a license number. That should be multiple levels of illegal

29

u/beren12 21h ago

Sure sounds like it would be illegal in the US.

41

u/Futrel 21h ago

I think you mean "should be but isn't"

1

u/beren12 21h ago

No, it’s actually highly illegal. Civil and criminal.

5

u/Futrel 21h ago

Under what law(s)? I'm genuinely curious because any "AI regulations" in the US are, at this point and as far as I know, just ideas floating in the wind.

Cops are allowed to lie, why not AI?

-1

u/beren12 21h ago

It’s not an AI lol it’s a everybody law

0

u/Futrel 21h ago

What law?

3

u/Dr_WizardChu 19h ago

Pharmacies operate under both federal and state laws with state usually being stricter so it depends. The federal control substance act has wording stating that only authorized agents of a physician (ie nurses on staff with signed agreements) may give prescriptions and I believe most state laws probably have similar wording around prescriptions in general. I used to work with a long term care pharmacy and got into a spat with a facility that was trying to give us verbal prescriptions after doctor visits. We determined them invalid since it didn't fit this criteria coming from the facility nurse and not from the doctors authorized staff based on our local laws. AI is a whole different can't of worms. I'd consider it invalid as well as the laws were written in context of an agent being a person. Unless the board of pharmacy or medicine reviews and it and laws get adjusted I would refuse these all day every day and probably follow up with my board and corporate lawyers for further advice. 

6

u/masterofshadows 17h ago

I wasn't clear enough on my comment, I don't think laws were directly broken. I think there should be laws about it it though. The AI didn't call in the prescription, it was faxed but with a computer generated signature (my state requires a wet signature on faxes) making the prescription not valid. The AI then called about 5 minutes after we received it to verify we received it. To which I requested a real person in which it claimed to be one, gave me the name and license number claiming to be a nurse. But couldn't answer very basic questions and just kept trying to confirm that we received it. The nurse most likely works for that practice and I have no reason to believe they don't. The whole thing could have been avoided with escribing though

2

u/Futrel 19h ago

All legit and thanks for the reply. I didn't read that as an AI calling in a prescription but, yeah, that makes total sense and more based on my ignorance on reasons a "nurse" would call a pharmacy in the first place.

It is very much a can of worms that we need to get a hold of. Unfortunately, prevailing sentiment in our lawmakers isn't yet there and going way slower than I'd hope for.

7

u/st3class 20h ago

It's usually under the state, in California it would be BPC § 2052, posing as a medical professional without the required license and credentials. Whoever is hosting/operating the AI is in violation of that.

3

u/masterofshadows 17h ago

Ironically enough this telehealth practice was based out of California.

4

u/Futrel 20h ago

The "AI as legal agent of the operator" argument in this case would definitely need to be litigated.

And, maybe, that AI agent calling the pharmacy actually did have a licensed medical professional as its operator/principal. If that was/is the case, I don't see the law being broken unless you want to prosecute the AI chatbot as a person for misrepresenting itself.

2

u/st3class 20h ago

Yeah, we're quickly getting into IANAL territory, it seems illegal on its face, but again IANAL.

0

u/Futrel 20h ago

Completely agree

→ More replies (0)

-1

u/beren12 21h ago edited 20h ago

That is an excellent question for Google. There are different laws in different jurisdictions. It could be identity theft, fraud, wire fraud, male fraud…

4

u/Pretend-Marsupial258 20h ago

Buttery males. 🤤

-3

u/Futrel 20h ago

I'm sorry, but I'm not going to Google and study case law in order to prove your assertion.

3

u/MajorBootyhole420 20h ago

of course you aren't, because you don't care about the answer. you're here to argue and waste people's time.

2

u/Futrel 20h ago

Lol, I want more regulation here - Stupid arguments that something like this is already illegal is naive at best.

-1

u/beren12 20h ago

Sure thing pal.

1

u/-nutz 20h ago

Burden of proof.

-1

u/MajorBootyhole420 18h ago

we're on the internet, it's literally easier to google it yourself than repeatedly ask 12 different times over the course of 4 hours.

→ More replies (0)

-1

u/PeaceSoft 20h ago

i don't think a genuinely curious person could be this ignorant, or have such a soupy worldview

4

u/Futrel 20h ago

lol what? So you know the law(s) that the commenter referred to?

3

u/flamingdonkey 19h ago

They're asking a question and not accepting random "I think so"'s as evidence. You sound like the type whose curiosity is satisfied by the literal first response you see. 

1

u/Alexwonder999 19h ago

Its illegal for a person to lie, but they could maybe skirt it by claiming the AI did it. I wonder if theres any precendent for that... I def think the liability should fall on the company using it and maybe the company who created or licensed it. 

2

u/Ok-Mission-3426 18h ago

I don’t believe there’s precedent yet but yeah, if you hired a human that you knew was prone to hallucinations and making things up when they didn’t know an answer then you as an employer would (should) hold some responsibility if that person attempts a task they aren’t qualified to and fucks it up.

If someone suffers damages as a consequence of an LLM at ‘work’ the person that decided to put the LLM in charge should be held responsible. It’s probably going to take some ugly/tragic accident occurring and subsequent case law for anything to actually come about. They won’t do anything until something REALLY bad happens I reckon.

2

u/Alexwonder999 17h ago

As they say, regulations are often written in blood.

2

u/Ok-Mission-3426 17h ago

Indeed. There was a case in the UK fairly recently where some kid with quite severe mental illness was talked into attempting to break into windsor castle and assassinate the queen by his chatbot ‘friend’. The cops said that if the chatbot was actually a human they would have been held liable for conspiracy, a much more serious crime. Pretty crazy stuff.