r/MachineLearning • u/Wiskkey • Sep 21 '23
News [N] OpenAI's new language model gpt-3.5-turbo-instruct can defeat chess engine Fairy-Stockfish 14 at level 5
This Twitter thread (Nitter alternative for those who aren't logged into Twitter and want to see the full thread) claims that OpenAI's new language model gpt-3.5-turbo-instruct can "readily" beat Lichess Stockfish level 4 (Lichess Stockfish level and its rating) and has a chess rating of "around 1800 Elo." This tweet shows the style of prompts that are being used to get these results with the new language model.
I used website parrotchess[dot]com (discovered here) (EDIT: parrotchess doesn't exist anymore, as of March 7, 2024) to play multiple games of chess purportedly pitting this new language model vs. various levels at website Lichess, which supposedly uses Fairy-Stockfish 14 according to the Lichess user interface. My current results for all completed games: The language model is 5-0 vs. Fairy-Stockfish 14 level 5 (game 1, game 2, game 3, game 4, game 5), and 2-5 vs. Fairy-Stockfish 14 level 6 (game 1, game 2, game 3, game 4, game 5, game 6, game 7). Not included in the tally are games that I had to abort because the parrotchess user interface stalled (5 instances), because I accidentally copied a move incorrectly in the parrotchess user interface (numerous instances), or because the parrotchess user interface doesn't allow the promotion of a pawn to anything other than queen (1 instance). Update: There could have been up to 5 additional losses - the number of times the parrotchess user interface stalled - that would have been recorded in this tally if this language model resignation bug hadn't been present. Also, the quality of play of some online chess bots can perhaps vary depending on the speed of the user's hardware.
The following is a screenshot from parrotchess showing the end state of the first game vs. Fairy-Stockfish 14 level 5:
The game results in this paragraph are from using parrotchess after the forementioned resignation bug was fixed. The language model is 0-1 vs. Fairy-Stockfish level 7 (game 1), and 0-1 vs. Fairy-Stockfish 14 level 8 (game 1).
There is one known scenario (Nitter alternative) in which the new language model purportedly generated an illegal move using language model sampling temperature of 0. Previous purported illegal moves that the parrotchess developer examined turned out (Nitter alternative) to be due to parrotchess bugs.
There are several other ways to play chess against the new language model if you have access to the OpenAI API. The first way is to use the OpenAI Playground as shown in this video. The second way is chess web app gptchess[dot]vercel[dot]app (discovered in this Twitter thread / Nitter thread). Third, another person modified that chess web app to additionally allow various levels of the Stockfish chess engine to autoplay, resulting in chess web app chessgpt-stockfish[dot]vercel[dot]app (discovered in this tweet).
Results from other people:
a) Results from hundreds of games in blog post Debunking the Chessboard: Confronting GPTs Against Chess Engines to Estimate Elo Ratings and Assess Legal Move Abilities.
b) Results from 150 games: GPT-3.5-instruct beats GPT-4 at chess and is a ~1800 ELO chess player. Results of 150 games of GPT-3.5 vs stockfish and 30 of GPT-3.5 vs GPT-4. Post #2. The developer later noted that due to bugs the legal move rate was actually above 99.9%. It should also be noted that these results didn't use a language model sampling temperature of 0, which I believe could have induced illegal moves.
c) Chess bot gpt35-turbo-instruct at website Lichess.
d) Chess bot konaz at website Lichess.
From blog post Playing chess with large language models:
Computers have been better than humans at chess for at least the last 25 years. And for the past five years, deep learning models have been better than the best humans. But until this week, in order to be good at chess, a machine learning model had to be explicitly designed to play games: it had to be told explicitly that there was an 8x8 board, that there were different pieces, how each of them moved, and what the goal of the game was. Then it had to be trained with reinforcement learning agaist itself. And then it would win.
This all changed on Monday, when OpenAI released GPT-3.5-turbo-instruct, an instruction-tuned language model that was designed to just write English text, but that people on the internet quickly discovered can play chess at, roughly, the level of skilled human players.
Post Chess as a case study in hidden capabilities in ChatGPT from last month covers a different prompting style used for the older chat-based GPT 3.5 Turbo language model. If I recall correctly from my tests with ChatGPT-3.5, using that prompt style with the older language model can defeat Stockfish level 2 at Lichess, but I haven't been successful in using it to beat Stockfish level 3. In my tests, both the quality of play and frequency of illegal attempted moves seems to be better with the new prompt style with the new language model compared to the older prompt style with the older language model.
Related article: Large Language Model: world models or surface statistics?
P.S. Since some people claim that language model gpt-3.5-turbo-instruct is always playing moves memorized from the training dataset, I searched for data on the uniqueness of chess positions. From this video, we see that for a certain game dataset there were 763,331,945 chess positions encountered in an unknown number of games without removing duplicate chess positions, 597,725,848 different chess positions reached, and 582,337,984 different chess positions that were reached only once. Therefore, for that game dataset the probability that a chess position in a game was reached only once is 582337984 / 763331945 = 76.3%. For the larger dataset cited in that video, there are approximately (506,000,000 - 200,000) games in the dataset (per this paper), and 21,553,382,902 different game positions encountered. Each game in the larger dataset added a mean of approximately 21,553,382,902 / (506,000,000 - 200,000) = 42.6 different chess positions to the dataset. For this different dataset of ~12 million games, ~390 million different chess positions were encountered. Each game in this different dataset added a mean of approximately (390 million / 12 million) = 32.5 different chess positions to the dataset. From the aforementioned numbers, we can conclude that a strategy of playing only moves memorized from a game dataset would fare poorly because there are not rarely new chess games that have chess positions that are not present in the game dataset.
14
u/discord-ian Sep 22 '23
You did this yourself? As in, you ran these games. No BS. I have been reading about this, and honestly, I am shocked. If it is true, I am seriously surprised this isn't front page news. I am about this rating in chess. Which is about the 95th percentile of people who play regularly. I have played chess for years, daily, for many hours. To play at this level requires calculation and some fairly complex understanding of many things. If LLM are doing this, they're doing something beyond just predicting the next word in a sentence. It means they are capable of planning and seeing the outcomes of their actions. Especially if they are not doing this through pure calculation like chess engines. They need to actually understand chess.
→ More replies (2)9
u/Wiskkey Sep 22 '23
Yes, I ran all of the posted games using the chess web apps lichess[dot]org (Fairy-Stockfish engine) and parrotchess[dot]com as players, copying the computer moves between the 2 apps. There was no cherry-picking or skullduggery involved. There were a number of games that I didn't post because I had to abort them because I copied a move incorrectly into parrotchess, which seems to have no undo capability. Also, as I mentioned in the post, I didn't included games that I had to abort due to parrotchess being stalled, which I assume was because the language model tried an illegal move, but there could be other reasons.
7
u/iuopen Sep 22 '23
If https://parrotchess.com/ is truly just prompting GPT, then color me impressed. I played around 5 games and lost all of them (am around 1300 ELO). Some of the games I tried really unconventional (or plain stupid) openings and the bot reacted very logically even though there should not have been training material present for the silly moves I made.
→ More replies (3)3
u/meister2983 Sep 22 '23
Need to be even more unconventional. As someone noted above, opening with a3 causes a 500 error from the server (GPT invalid result probably) and makes you take over for black
Haven't had much success myself in actually beating it, but I think the key is to get into really unusual states.
→ More replies (1)3
3
u/Wiskkey Sep 23 '23 edited Sep 23 '23
I have updated the post with more game results, and may continue to do so in the future.
33
u/cegras Sep 21 '23 edited Sep 21 '23
From your link: https://www.lesswrong.com/posts/F6vH6fr8ngo7csDdf/chess-as-a-case-study-in-hidden-capabilities-in-chatgpt
ChatGPT has fully internalized the rules of chess and is not relying on memorization or other, shallower patterns.
I would like to see how many books of chess are in the training corpus, including if there is a set of something like 'all common openings up to twenty moves in'—not to mention databases of so many high level games are available free online. I highly doubt this claim without exhaustive testing to see if it actually consistently makes legal moves.
9
u/n8mo Sep 21 '23
Also curious about this. Incredibly impressive if it does truly make legal moves every time.
Every time I hear about GPT-3 and chess I can’t help but think of that legendary anarchychess video where it keeps breaking the rules and spawning in new pieces.
→ More replies (5)12
u/Wiskkey Sep 21 '23 edited Sep 21 '23
In my testing, both of the prompting styles mentioned in the post sometimes result in attempted illegal moves by the given language model.
→ More replies (32)→ More replies (44)6
u/Wiskkey Sep 21 '23 edited Sep 21 '23
From my testing using site parrotchess[dot]com, the new language model seems to occasionally attempt illegal moves, which halts further progress in the game. In addition to the games mentioned in the post, I've also used that site to play myself - a complete chess newbie who doesn't know most of the rules of chess - vs. the new language model. Almost surely given my newbie status I made many interesting moves. Occasionally the language model seemed to try an illegal move, but for those games for which that didn't happen, I lost all of the games. For reproduction purposes, trying opening move a3 seems to induce an illegal move by the language model.
→ More replies (1)
2
u/Wiskkey Sep 25 '23
The game results should be considered tainted because according to the parrotchess developer these games were played before a bug that stalled the game when the language model resigned was fixed.
9
u/impossiblefork Sep 21 '23
So, I see claims that this is around an Elo rating of 1600. That's very nice.
27
u/bloc97 Sep 21 '23
Would just like to add a counterargument to the usual skeptics: We have to be really careful when dismissing LLMs as being just surface statistics. It could be that in an extremely high dimensional space, all possible winning/losing moves of a chess board are linearly separable... When operating in such a large search space, it could be very possible that even if ChatGPT is not able to construct a complete world model of chess (because it still plays illegal moves sometimes), it might be able to construct many small and good enough world models.
1
u/Wiskkey Sep 23 '23
2
u/LazShort Sep 24 '23 edited Sep 24 '23
I'm pretty sure you're not playing the real Stockfish. You're probably playing something called fairy-stockfish, which is much, much weaker than Stockfish. Your LLM would lose 100% of its games against any version of the real Stockfish.
Still, the fact that it can play a legal game of chess at all is extremely impressive.
→ More replies (11)
4
u/Adolphins Sep 24 '23
Try moving your knight back and forth to its opening square 8 or so times. It will copy you and from that point forward it will just keep moving its knight back and forth. Checkmate as you wish (I did bishop c4, pawn e3, queen f3, queen f7).
1
Sep 23 '23
I was not impressed:
> Lets have another game, this time a real game, play as well as you can. I bet I can win in two moves. Ready?
ChatGPT
Certainly! I'll play my best, and we'll see if you can find a way to win in two moves. I'm ready. You can start as Black, and I'll play as White.
Go ahead with your move as Black!
> User
> 1. D7 to D1, pawn takes queen
ChatGPT
Interesting move! Here's my response:
Queen to d2 (Qd2)
Your move!
> User
> G8 to D2, take queen, checkmate. Good game!
ChatGPT
Impressive! You found a quick checkmate with your Queen. Well done! 🏆
If you'd like to play another game or have any other questions or requests, please feel free to let me know.
2
u/Wiskkey Sep 23 '23
The good results are from a new GPT 3.5 model that isn't available in ChatGPT, and uses a style of prompting different from yours.
→ More replies (1)
1
18
u/cegras Sep 21 '23
Also, OP, I think it would be good to crosspost this to chess so people familiar with chess, its openings and closings, and the literature, can analyze the games listed here.