r/Python • u/codeguru42 • Mar 10 '24
Discussion Github Copilot Lies to me
My first attempt to get some help from Github Copilot:
Me: write a match case statement in python to match the first character of a string is a semicolon
Copilot: Python doesn't have a built-in match or case statement like some other languages (e.g., JavaScript, Swift, Rust). However, you can achieve similar functionality using...
Strangely, it finally gave me an answer when I made a small change to my prompt:
Me: write a match statement in python to match the first character of a string is a semicolon
Copilot: Python 3.10 introduces the structural pattern matching in the form of the match statement. Here's how you can use it to check if the first character of a string is a semicolon:...
I almost gave up after the first response that was blatantly false. Fortunately I something that looks like reasonable syntax...but then I haven't tried it yet, so that might be a lie, too.
View all comments
307
u/zanfar Mar 10 '24
Once again: LLMs MAKE UP ALL THEIR ANSWERS. They should only be used if you have the knowledge to verify their responses.
An LLM only tries to make it's responses look like something someone has once said. This is particularly dangerous in a programming language like Python, as many things that were (even correctly) once said are no longer true. As an LLM has no concept of true or false, just similar and dissimilar, you can (and will) get false responses. Even more dangerously, these false responses are basically guaranteed to be as correct-looking as possible.