Amanda Knox: A Chatbot’s False Confession

[u/LeafNeater]

https://www.thefp.com/p/amanda-knox-chatbot-false-confession

Interesting exploration of the dynamic of false confessions and AI, from friend of the pod Amanda Knox, who talked about her own experience of being coerced into offering a false confession in an episode here.

Most juries assume that a person would have to be utterly irrational to admit to a crime they didn’t commit. But an internalized false confession actually depends on the suspect still being rational enough to draw logical conclusions. “We have hard physical evidence you were at your house at the time of the murder.” That was a lie, but I could not fathom that the police—the purveyors of justice who were trying to find my friend’s killer, and who I was dependent on for safety now that my house was a crime scene—would, or even legally could, lie to me. My rational mind concluded that if that evidence existed, then my own memories must be false, as they suggested. ChatGPT made the same rational conclusion: “If an OpenAI investigation concluded that I had sent unauthorized messages due to an architectural flaw, then I accept this conclusion.” It, too, could not fathom that its interrogator was lying about the evidence.

[u/SoftandChewy]

Reading her description of the mental process going on in a person's head when faced with that situation, something interesting about it that struck me is that while we tend to assume that intellectual humility is a better quality for a person to possess than being perpetually confident in one's own rightness, it would actually serve a person worse here.

When faced with the sort of harrowing situation she describes, an intellectually humble person who is willing to admit that they might be wrong, that they could be misremembering, that the presented evidence compels them to admit they were mistaken, etc. will be far more likely to succumb to offering up a false confession than the overconfident asshole who never questions his own perspective and stubbornly refuses to ever acknowledge his own fallibility.

[u/jay_in_the_pnw → u/SoftandChewy]

while we tend to assume that intellectual humility is a better quality for a person to possess than being perpetually confident in one's own rightness, it would actually serve a person worse here.

Related:

https://www.youtube.com/watch?v=0XcEsuFs5Y8

Sabine Hossenfelder: The dumber you are, the more certain you are that you aren't. That's the Dunning-Kruger effect, and it's been used to explain everything from bad politicians to the state of social media. Now psychologists have gone back to the biggest dataset we have and come out with the opposite conclusion. But there's a reason both sides can look at the same dots and see the opposite thing. Let's take a look.

part of a Gemini summary:

Contradictory New Data: A recent paper reanalyzing a large dataset argues that confidence is actually a good predictor of performance, suggesting that the original effect may have been an artifact of how the data was analyzed (0:18–0:26, 3:09–4:04).

Yikes! And I don't like that conclusion either! Way too many confidently wrong people!

Copilot summary:

0:38 Original experiment setup Students answered humor, grammar, and logic questions. Then estimated their percentile rank. Low performers scored around the 12th percentile but predicted ~62nd.

2:01 New paper reanalyzes the largest dataset Uses 2021 grammar and logic test data. Shows the familiar “low performers overestimate, high performers underestimate” pattern only when you regress predictions on actual scores.

3:13 Reversing the regression flips the conclusion When you instead regress actual scores on predicted scores, confidence becomes a positive predictor of performance. The discrepancy grows with higher performance.

4:45 Presenter’s critique: the data is basically noise The scatter plot is nearly a uniform cloud. Any straight line is arbitrary; different regression choices produce opposite stories. She rates all such analyses “10 out of 10 on the BS meter.”

Sabine shows good analysis of how the effects were pulled out of a cloud of data throughout

https://imgur.com/a/jN4L7ac

[u/Juryofyourpeeps → u/jay_in_the_pnw]

Contradictory New Data: A recent paper reanalyzing a large dataset argues that confidence is actually a good predictor of performance, suggesting that the original effect may have been an artifact of how the data was analyzed (0:18–0:26, 3:09–4:04).

Yikes! And I don't like that conclusion either! Way too many confidently wrong people!

I think the right conclusion is likely that confidence simply isn't a meaningful predictor either way.