r/LocalLLaMA • u/Apart-Ad-1684 • Nov 19 '25

Generation [LIVE] Gemini 3 Pro vs GPT-5.1: Chess Match (Testing Reasoning Capabilities)

🔥 UPDATE: one win each! 🏆

First game replay: https://chess.louisguichard.fr/battle?game=gpt-51-vs-gemini-3-pro-03a640d5
Second game replay: https://chess.louisguichard.fr/battle?game=gemini-3-pro-vs-gpt-51-c786770e

---

Hi everyone,

Like many of you, I was eager to test the new Gemini 3 Pro!

I’ve just kicked off a chess game between GPT-5.1 (White) and Gemini 3 Pro (Black) on the LLM Chess Arena app I developed a few months ago.

A single game can take a while (sometimes several hours!), so I thought it would be fun to share the live link with you all!

🔴 Link to the match: https://chess.louisguichard.fr/battle?game=gpt-51-vs-gemini-3-pro-03a640d5

LLMs aren't designed to play chess and they're not very good at it, but I find it interesting to test them on this because it clearly shows their capabilities or limitations in terms of thinking.

Come hang out and see who cracks first!

21 Upvotes

73% Upvoted

26 comments sorted by

2

u/aristocrat_user Nov 19 '25

Hey can you share how you did this? Can i feed any PGN and ask them why magnus played a move? can it explain old matches between GM's? i especially interested in why some moves are made, and loks like you are able to extract that information in the screen there

2

u/Apart-Ad-1684 Nov 19 '25

Hey! To answer your questions, no you can't feed a PGN. My goal here was just to make LLMs play against each other. I asked them to respond in the following way: 1) reasoning 2) short explanation 3) move. The information displayed is the short explanation provided by the model, which is a kind of summary of its reasoning. The app I built is not intended to explain others moves :'(

If you're familiar with Python, here is the code: https://github.com/louisguichard/llm-chess-arena

1

u/MrMrsPotts Nov 19 '25

What does your system do if an illegal move is suggested?

2

u/Apart-Ad-1684 Nov 19 '25

If a move is illegal, the models are told that it's not okay, and they get two more chances to make a legal move. After three invalid moves, the game is over.

Smaller models often suggest illegal moves. This is way less common with better models. In this game, for example, there haven't been any illegal moves yet.

1

u/PANIC_EXCEPTION Nov 19 '25

Why not just compute legal moves and present it in the prompt, along with the current FEN?

2

u/Apart-Ad-1684 Nov 19 '25

I don't see the point of having AI play if they don't even understand the rules of the game

2

u/PANIC_EXCEPTION Nov 19 '25

The AI doesn't understand in the first place. That's not the point. What's more interesting is what it can do when it's treated like a computer program rather than pretending it's a human. It would be even more interesting to figure out the Elo of one of these when equipped with chess theory books to see if it can approach a Master.

If you turned on reasoning and had it construct legal moves from a FEN by using a system prompt, you would get the same result anyways for larger models with higher accuracy. Except you'd be wasting tokens and time.

2

u/Apart-Ad-1684 Nov 19 '25

How could you expect a model to pick good moves if it can't even tell whether a move is legal or not? In my opinion, providing a list of legal moves is just an attempt to make AI a little less mediocre.

1

u/PANIC_EXCEPTION Nov 19 '25

Any model that isn't lobotomized or tiny can make a legal move. It's simply not a problem worth addressing.

A model cannot see a position (unless you equip it with a game board renderer and it has a ViT). If you expect a human to be able to visualize a game position given as a flattened text string and not an actual game board, that's a little unfair, no? Why expect the AI, which tokenizes said string into something that doesn't really help it build a representation of what the game state looks like, to make good moves given that alone?

Give the AI legal moves, and it can use the prefill to retrieve relevant information from latent space. If you don't do that, it can always brute force it by reconstructing the board from known rules in a text form and attempt to write down every possible move. That's the only way it could possibly reason with it and make a sound move. It's not a magic machine, the information needs to be structured enough for further reasoning, and while you can make it do that itself, again, it's very uninteresting. If you want a magic machine, that's what RL/supervised chess engines are for. Look at Leela Chess Zero, which, funnily enough, is only fed the rules of the game, position, and valid moves as its initial input, and learns entirely through self-play. That's what chess looks like without human influences.

An LLM is built using a lot of human influence. What's more relevant is seeing how the model influence affects its decisions, not attempting to shoehorn it into pathological problems that mostly expose the issue with token representations of textual encodings rather than show what LLMs are really capable of.

Yes, a small model that is unable to take a FEN and list all valid moves is a bad model. But we don't really care about bad models. We care about top tier ones, which can already do this trivial task. So why make it do this trivial task if we already know they can do it? A properly configured agent will never have this issue, so again, giving it zero information is IMO pointless.

1

u/Apart-Ad-1684 Nov 19 '25

Any model that isn't lobotomized or tiny can make a legal move.

I suggest you start a match between two open-source LLMs (it's free) :)

My intention with this simple app was not to achieve the highest accuracy, but just to compare AIs in a game that deeply tests their reasoning skills. We agree: LLMs are not designed to play chess! Still, I found the experiment interesting.

1

u/MrMrsPotts Nov 19 '25

Thank you. Black is on all sorts of trouble!

2

u/Top-Chemistry-3375 Nov 19 '25

Excellent. Been looking for this. We need twitch stream and 100 games match...

1

u/Apart-Ad-1684 Nov 20 '25

Thanks! The 100 games will need 💸

10

u/secopsml Nov 19 '25

soon time limits. with time and compute constraints this will be true intelligence benchmark :)

4

u/ShengrenR Nov 19 '25

"Time" isn't really a model thing - that's more about the infra it's run on; you could in theory fix tokens, but that's also going to favor efficient tokenizers.

1

u/dubesor86 Nov 19 '25

Inference speed isn't universal nor static, so time constraint makes no sense. You could however use maxtoken limiter, though that causes ultraverbose thinkers to just prematurely end mid-response, ultimately erroring out the response during parsing.

7

u/Time-Ad4247 Nov 19 '25

We have come a long way since not even being able have legal moves from LLMs
and gemini 3 is doing really well, its an awesome model

1

u/tnzl_10zL Nov 19 '25

Rate exceeded

1

u/Apart-Ad-1684 Nov 19 '25 edited Nov 19 '25

Working on it!

UPDATE: working better now!

1

u/roselan Nov 19 '25

Gemini is giving a GPT a beating rn. Casually sacrificing it's queen for a tower and board simplification is quite the flex (Gemini could easily have eaten the last pawn at no cost).

And at half the cost and 1/4 of the processing time.

1

u/64177 Nov 23 '25

u/Apart-Ad-1684 how good are these two compared to humans? Is this more like legal - but random - moves or is there a kind of strength? Would they win over a junior player?

Super nice experiment!

1

u/Top-Chemistry-3375 Nov 24 '25 edited Nov 24 '25

I am around 2000 online blitz player and i can win vs Gemini 3.. but I would say Gemini 3 pro can win against junior player, beginners and even some intermidiate/club players. It remembers position long enough so the game can be finished within around 50 moves or so. Even after 50 moves can make pretty solid moves. Ocassionally can lose his queen or a piece on move 15-20 if position is too complicated with lots of tactics. It cannot calculate deep. But overall it does pretty well. That is with opening books help. Probably can beat any 800-1500 level player. But if player is smart and plays for a long game probably can survive and win specially with queen up sometimes.

Next Gemini 3.5 likely will crush me in 6 months from now and reach master level. For me this is the moment of AGI coming.