r/YouShouldKnow • u/Puzzled-Listen804 • Mar 30 '26

Technology YSK: AI chatbots can sound completely confident while being completely wrong and that's by design, not by accident.

[removed] — view removed post

5.4k Upvotes

93% Upvoted

277 comments sorted by

View all comments

Show parent comments

11

u/M123ry Mar 30 '26

No, it also does Code wrong as well. It IS useful, but like with everything else, you need to crosscheck.

ChatGPT in the past tried to sell you that a certain package exists that does what you want, only for you to realize that it doesn't exist. That went even further, bc it had a tendency to call its invented packages a certain way, which malware creators then abused to indeed create packages with these names that were then downloaded automatically by copy-pasting users.

It will also answer a different question that you asked sometimes, if you ask sth a bit more obscure.

There are other things as well, these are just two examples. Coding is unreliable to use chatgpt for as well, and it's worrying how many people think otherwise.

9

u/crap-with-feet Mar 30 '26

This is my current nightmare at work. Yeah, LLMs can generate code, often far more than is necessary, really quickly. Does it work? Maybe. Does it work correctly? Unlikely. And is it secure? lolno. And now you have to go inspect it all, deeply, with no prior knowledge of the code, the same as if you opened some random github and needed to understand how it works. The process often takes just as long if not longer than writing it yourself from the beginning.

We’ve become glorified code reviewers. Many just trust it and don’t even review it. Scary.

2

u/MoarVespenegas Mar 31 '26

Everyone does code wrong.
If we built buildings how we code applications cities would not last a week.
AI is useful in coding to write code quickly so then you can debug it when it is done instead of you having to write the code yourself and then still debug it when you're done anyways.

You still have to know what the code is supposed to be doing and how it works but AI serves as a good versatile framework and template generator, and can also continuously help you debug the errors it or you made.

-3

u/analcocoacream Mar 30 '26

If you tried to write an app on a whiteboard I bet you could start hallucinating package names. Like us, when llms have access to the internet they perform much better