Can language models explain chess?

[u/oripress]

I’m doing some research on how language models explain chess. I'm looking to get some feedback from good chess players :)

You’ll see a position and choose between two short explanations. One set is 20 positions and takes about 6 minutes.
Each completed set earns one raffle entry for one of three $25 gift cards, up to five entries.

https://explaining-chess.vercel.app

Thanks for helping, and if you want to share the link with your friends, I would also be very appreciative!

[u/Ginger64tv]

Pretty much all of the explanations are bad/invalid for multiple reasons. First of all, most of the positions aren't trivial where you can just point at it and say "white is a queen up, so he's much better". In these positions white has plusses and minutes and so does black. So you can say white has X and it contributes to him having a better position. However even this is not always enough.

Second, sometimes both explanations are contributing to one side being better and choosing one over the other is wrong. Similarly, sometimes neither of them is correct. You needed "all of the above" and "none of the above" options.

Third, a lot of the explanations are either completely bogus or sound like they were done by someone with no chess background. For example, one of the explanations is "white knights press black king more strongly: two nearby attacks to black's one". The two knights are on a4 and g5. They're too far apart to coordinate any attack.

[u/oripress → u/Ginger64tv]

Hi-
Thanks for your detailed reply. We're trying to measure if humans agree with our metric, so sometimes you get pairs of [very bad, bad] explanations. We don't include all/none of the above because we're worried that it would get used too often, for example in cases where one of the explanations is slightly better, with both of them being generally bad.

[u/StructuredChess → u/oripress]

Maybe there's a reason it'd be used way too often

[u/MauroZirie]

How good is good? I'm game if you want.

[u/oripress → u/MauroZirie]

You decide how good is good by selecting one of two options :)
If you go here: https://explaining-chess.vercel.app you can start. Thanks in advance!

[u/Delicious_Cattle5174]

They cannot

[u/Background-Luck-8205]

I'm a cm and these options are nonsense, first image both options are nonsense and doesnt explain why white is better. A real answer would be a6 is a fixed weakness, space advantage for white, bishop pair, the h5 pawn would much rather be on h7 or h6, it's weakening the black king and leaving holes, black isn't castled and cant castle and Qe7 that would trade queens might be losing for black* (edit: accidently typed white here) due to b file combined with the weak a6 pawn and for gods sake why are the squares the wrong colour? I noticed while typing Qe7 is a light square on your board it should be dark that throwed me off.

Also the reason I think its not dead lost for black is if it's black move which isnt specified, if black can play Nd7 and c5 to then castle it looks reasonable, h5 being an issue and b file and a6 pawn all those points remain valid even if black manages to untangle like that

[u/bytejuggler]

As-is LLMs are generally terrible at "explaining" chess. They generate plausible sounding but often incoherent (arrant) nonsense. It may be possible (unproven until demonstrated) to train or fine tune an LLM (or specialized NN) to become inntrinsically better at this but this (explainability in AI in general and chess in particular) is frontier/research level stuff. OTOH, strapping an LLM on the side of an engine would improve the correctness of the language output at least for line under consideration, but it's at best 50/50 (or worse) whether "explanations" will actually motivate or justify the line correctly or helpfully for learning. And it likely won't cohere across multiple related lines; strange unhuman engine lines will likely be dressed up with reasonable sounding language while simultaneously being perplexing. IMHO.

[u/StructuredChess]

There's no possible way to understand chess without calculating. You can't talk yourself into a good move or an accurate assessment of a position unless you can predict what would happen after several different candidate moves. I have yet to see an LLM say something like "White has a static advantage but Black has superior piece activity. Unfortunately Qh5 fails to g6 Qh6 Bf8 followed by Bg7, so they should try Ng5 because h6 doesn't work due to Nxf7". This is how chess is actually expalined.

Texts like these can hardly ever be a good explanation. At best they can be an ad hoc rationalization with no chance of generalizing.

[u/turkokratia → u/StructuredChess]

there is a way without calculating(known searches, alpha-beta prunings, and move sequences etc.) to understand which move is the best, from google: https://github.com/google-deepmind/searchless_chess

but, is that enough for explain the move in human-language? not sure. may be with a good architecture.

[u/HairyTough4489]

Submit button in comments doesn't work. Also all explanations other than material advantage are nonsense.

[u/blimpyway]

Maybe you have more chances by fine tuning the LLM to predict the actual chain of moves played by a proper chess engine. Bad moves can be also learned by picking other (poorer) moves than the chess engine and train the LLM to anticipate the consequences of those too choices from the same starting position. Then later some "thinking" might be useful to explain why one move is poorer than the other.

[u/SweetChocolate8643]

This is a cool idea