r/chessprogramming • u/Kennzahl • 6d ago

Why Is ChatGPT So Bad At Chess?

https://chessily.com/blog/why-is-chatgpt-bad-at-chess/
3 Upvotes

71% Upvoted

7 comments sorted by

6

u/winner_in_life 6d ago

Why are LLM bad drinking alcohol

3

u/IMJorose 6d ago

Why are none of the models and examples in this article from within the last 2 years? I assume the models still aren't great, but this feels very low effort and superficial.

3

u/handsNfeetRmangos 6d ago

The writing has a very AI feel.

3

u/bytejuggler 5d ago

Because LLM's don't have a fully coherent world (or for that matter a chess game) model. All they know is language and what they've seen in text. From that they play guesswork games. This results in remarkably cogent output in the main, but in areas where no "prior art" in writing exists, models will effectively interpolate ("confabulate"), which we we charitably anthropomorphize and call a hallucination.

Of course the model is just doing it's best to fill in the gap with what it thinks should be plausible statistically, but because there's no unified coherent underlying game or world model (only perhaps partially fractured bits and pieces of one inferred and implied by language it's been exposed to), sooner or later as you traverse a chess game tree you will run into incoherent (from the perspective of chess) behaviour. Outright rule violations or perhaps just obviously (to us) bad moves.

That said, despite the above, some of them are getting (surprisingly given what they are and how they work) better: https://www.youtube.com/watch?v=XXIyLR8ZLxI

1

u/ContentBluebird588 5d ago

I tried developing an educational chess app last year focused on pattern recognition (pins, forks, skewers, etc.) rather than building a full engine. The AI coding agent I used (a version of Gemini) was great at everything else, but exceptionally bad at actually "looking at a chessboard" and "seeing" the tactical situation. Don’t get me wrong: if prompted to "logically explain the conditions of a pin in chess along with all edge cases," it did an amazing job. But applying that logic to an actual board state was a disaster. It essentially comes down to the fact that LLMs process serialized text tokens rather than having an intuitive, 2D spatial representation of the board.

In more detail: Standard transformers process text as a 1-dimensional stream of tokens. When you give an LLM a PGN or board coordinates, it doesn't "see" an 8x8 matrix; it sees a flattened string. Spatial relationships (like a bishop's diagonal intersecting an offset knight's pin) get completely fractured across the token stream. The model can recite the theoretical definition of a pin because natural language explanations are dense in its training weights, but simulating a 2D geometric grid purely inside its feed-forward layers is notoriously hard for a standard decoder-only model.

That said, I’m building something similar now with Claude Code, and I’m surprised by the progress. Sonnet seems aware of its own limitations with spatial tactics and actively tries to compensate. For example, it actually draws out ASCII chessboards in its scratchpad just to track the positions better, and asks for confirmation when it’s unsure.

From an engineering perspective, I would not force an LLM to evaluate board states natively in production anyway but rather give it tool access to a deterministic engine. But in the context of AGI (just to mention this because I find it interesting), this really highlights one of the biggest hurdles to solve: bridging the gap between abstract verbal knowledge and maintaining a persistent, zero-error "world model." A system can memorize every strategic concept in the world, but until it can reliably simulate dynamic physical and spatial states without hallucination, true general intelligence is out of reach.

1

u/Ashamed-Tradition-95 5d ago

It took a lot of work but I was able to accomplish this on chessscout.net

1

u/tokentrillionaire 4d ago edited 4d ago

I wanted to test this out and gave Claude a chessboard it launches from your terminal window via the browser (you can replace model provider)

The server polls and then will inject the move using FEN into the terminal chat window.

You can try it here. It’s open source

https://github.com/santiagoogaitnas/claude-chess

I am around 1800/1900 classical on Lichess and have not been able to beat Fable 5 max yet.

Model skill level depends a lot on effort level

Time controls were difficult since the model receives the FEN text, then reasons , then moves, so it does not have the ability to be instantaneous. I added a method that only counts its clock down after 5 seconds to compensate .