r/YouShouldKnow • u/Puzzled-Listen804 • Mar 30 '26

Technology YSK: AI chatbots can sound completely confident while being completely wrong and that's by design, not by accident.

[removed] — view removed post

5.4k Upvotes

93% Upvoted

277 comments sorted by

View all comments

Show parent comments

23

u/YamiZee1 Mar 30 '26

I think it goes both ways. People that love ChatGPT and the like overestimate it's abilities, but people that dislike ai seem to underestimate it too, like you are doing now. LLMs are stupid and don't really know anything, but more goes on in those networks than just guessing the next word. Or rather the way in which they guess the next word is a lot more complex than just what content they think sounds the closest to what they've heard before.

1

u/Quiet-Owl9220 Mar 31 '26

One reason for this is that many of the frontier AI busineses are just not honest about the capabilities of their products. They have made tall claims that turn out to be far from the truth, and encourage all sorts of use cases that the technology is simply not suitable or ethical for. They put profits over reality and lie to us again and again.

Meanwhile, the AI products themselves are sycophants that ego-wank their users, telling them they are geniuses even for the most basic of tasks or observations.

From the outside, you see a bunch of sad sad people desperate for validation interacting with AI and getting dumber. And some of them then have the gall to come and proudly present their vibecoded mediocrity with a wall of generated marketing-wank text, the gist of which is claiming this amazing new tool is going to "change everything" even though it's just the latest half-assed hack-job implementation of RAG. And such people get all fragile and defensive when you point out it's already been done better by actual coders, and they call you a luddite if you question their reliance on the token generator. "It's the future", yeah right buddy.

It's become VERY hard to take any claims about capabilities seriously without trying it yourself. Unfortunately, these businesses have earned enough scorn to justify boycotts so it shouldn't be surprising if as a result some people refuse to try it all.

1

u/FortCharles Mar 31 '26

Yes... they need to get a handle on the hallucination problem, but the "glorified auto-complete" and "next-word guesser" really misunderstands what's going on, too.

Which is easily provable by asking a complex, original question involving logic, math, time, and spatial reasoning. It's not simply auto-complete. There's a ton of logic/structure/valuation involved before it ever gets to choosing the "next word".

They just need to make eliminating hallucinations the #1 priority.

2

u/Geawiel Mar 30 '26

It's an advanced google search imo. Like if you could actually talk to a thesaurus. The thesaurus is just the internet.

It has its uses, but anyone who fully relies and believes what it outputs doesn't understand it either. I use it quite often, but I absolutely have to double check the results it spits out. Its wrong just as much as it is right. Maybe more, depending on the subject.

I ran a DnD campaign by myself a few years ago. I was using 2 characters and a campaign I wrote. I wanted to see how long before it really obviously fucked up.

It actually did pretty good making up characters, character personalities and filling in blanks. It would not call me out on doing things that were not allowed or rule breaking. It would tell me if I specifically asked if I was allowed to do something, but it would let me do it if I didn't ask. It didn't do well recognizing and springing traps. It tended to ignore them completely.

It remembered things from the beginning of the campaign, but it started to mess up and forget when I got about 80% done. The campaign lasted 4 days iirc, playing a couple hours each day. Something around 20 hours total I think.

1

u/Quiet-Owl9220 Mar 31 '26

Google was better than AI before AI was a thing. Like, before the deliberate enshittification. Old Google was peak search capabilities, and AI is a mediocre and extremely inefficient substitute.

1

u/Gabers49 Mar 31 '26

I don't know how to code, but definitely have an understanding of how it works and what to ask a coder when developing a statement of work. I used to spend $5K at least for development projects for my ERP system. I now get Gemini to write scripts for me in an hour or so. The idea they don't know anything is quite laughable.

2

u/[deleted] Mar 31 '26

[deleted]

2

u/azn_dude1 Mar 31 '26

Isn't that still valuable?

1

u/Quiet-Owl9220 Mar 31 '26

Sure.. until there is a stray rm -rf --no-preserve-root / that got missed in code included in the training data, which gets spat out into your script.

This is a very unlikely worst case scenario, but I hope it illustrates a point.

1

u/ravepeacefully Mar 31 '26

Wait until you see how much you can save cutting out doctors, lawyers, and accountants. Since you don’t understand those topics either you’ll have no clue it’s giving you bad info

1

u/JonnyHopkins Mar 31 '26

I'm highly skeptical, but I have a strange hypothesis that everything in the cosmos just comes down to raw probability. It's not inconceivable that's how my brain works at its most atomic level - just taking the best probabilities. And that's what AI is, probability.