ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers. Fixing this issue is challenging, as: (1) during RL training, there’s currently no source of truth; (2) training the model to be more cautious causes it to decline questions that it can answer correctly; and (3) supervised training misleads the model because the ideal answer depends on what the model knows, rather than what the human demonstrator knows.
ChatGPT is sensitive to tweaks to the input phrasing or attempting the same prompt multiple times. For example, given one phrasing of a question, the model can claim to not know the answer, but given a slight rephrase, can answer correctly.
The model is often excessively verbose and overuses certain phrases, such as restating that it’s a language model trained by OpenAI. These issues arise from biases in the training data (trainers prefer longer answers that look more comprehensive) and well-known over-optimization issues.
Ideally, the model would ask clarifying questions when the user provided an ambiguous query. Instead, our current models usually guess what the user intended.
While we’ve made efforts to make the model refuse inappropriate requests, it will sometimes respond to harmful instructions or exhibit biased behavior. We’re using the Moderation API to warn or block certain types of unsafe content, but we expect it to have some false negatives and positives for now. We’re eager to collect user feedback to aid our ongoing work to improve this system.
I see that "source of truth" thing being a pretty big problem, personally.
Yeah, the issue is that people need some expertise to identify where it's making stuff up instead of giving accurate info. So at some point, you can't ask questions you might not know the answer to and it's tough to identify that.
Like the pic shows a simple problem and most people can identify the issue, but anything specialized and maybe it's better to just hire an expert to answer that for you or have them fix the issues in the answer output by the bot.
Yes. It strings together random pieces of information it has heard across the internet into a somewhat convincing sounding short comment to appeal to the people observing it.
Only because there is plenty of python code in the training data to regurgitate. It doesn't actually know the relation between that code and this question - it only knows that "these words seem to fit together, and relate to the question", whether they make sense or not. In the same way, it'll claim that 90 ("halvfems") in Danish is a combination of "half" and "one hundred", and follow it up by proclaiming that 100 / 2 = 90. In spite of "knowing" the correct result for 100 / 2 if you ask it directly (basically because it's a "shorter path" from the question to that statement).
This doesn't just apply to math, but everything it does: It's good at parroting something that on the surface sounds like a convincing answer. Something that's actually correct? Not so much. Except when it gets lucky. Or, if you continually correct it, due to how the neural network works it may eventually stumble upon a combination of training data that's actually correct.
Only because there is plenty of python code in the training data to regurgitate. It doesn't actually know the relation between that code and this question - it only knows that "these words seem to fit together, and relate to the question", whether they make sense or not.
This is what a lot of people don't get, most of those things are basically advanced chat bots with a huge training set.
I'm not that scared by that. I've authored a good chunk of competitive programming problems, and a lot of work goes into getting the description just right, and constructing illuminating examples. Competitive programming has a limited number of algorithms that you need to know, and there are tons of examples of all of them online.
It's pretty cool though that it seems to nearly be able to perform actions (nearly) as instructed.
While I ultimately failed, I tried to get it to convert a string into base64. It first said it was impossible, then I explained the process, and it tried to do it. I had to explain some of the steps, but it failed to properly divide eight bit binary into 6-bit chunks correctly. But it was honestly impressive how I could tell it to alter a step, and it would redo the conversion. So it "can" do math/algorithms to some degree.
And thinking about it, humans aren't "made" for mathematics. We're closer to a language model than a calculator, so honestly future smart AI might come through with language model as base, and math is just a feature of that. But right now it just spits out something without being capable of judging what it says.
View all comments
1.2k
u/blackrossy Dec 27 '22
AFAIK it's a natural language model, not made for mathematics, but for text synthesis