You could also download my free chess app that hooks up to Claude for real time game analysis or chat about the game.
https://pwalessi.itch.io/chess-buddy
Just a note that this is not novel. Automatic analysis of chess games built on traditional engines has existed for a long time. Without using stockfish as a sidekick, this wouldn’t be interesting because of the poor play of LLMs, however having an LLM enhance the commentary is interesting and I suspect has already been done by chess.com for years.
This what I think too. I also think there are much more refined approaches than the one I tried here. From a cursory look, I saw there are both scientific litterature on the subject of mixing llms and tools like stockfish and some dedicated closed platforms that put that into action.
Let's make clear that I did not spend much time on this project. Ideally I would have tried to put other models into the mix, like maybe Maia to better see the game from a "real player" eyes and pinpoint where expected move and stockfish moves differ.
Anyway, to me, it's good enough to be usable and shared.
How does it work? Does it just use stockfish under the hood?
I just remember the times when I tried to use an AI model to play chess and half of the time it either hung a piece or made an illegal move.
Was a bit to overzealous... Just read the readme. Looks pretty cool. Maybe I'll give it a try and see how it compares to the chess.com analysis (although I have to say that I never really use it).
"The fact that it reflects on my own thinking during the game makes it interesting from a teaching point of view, so I thought it was worth sharing." As AI the premium sycophantic scammer does
"but for me it is a much more pleasant and memorable experience than clicking around Stockfish branches"
And SO much less effective than doing the harder, more tedious feeling work
The jury is out for the effectiveness, it's hard to debate this subject. From my cognitive background I know very well how important the generative effect is for learning. Maybe I'll add features that leverage generative/testing effect one day. Anyway, clicking on stockfish branches can be quite a passive activity too if done badly. I don't know how to do that well to be honest. My goal is often to just to understand what I missed, full stop.
For the sycophancy, I can say I did not feel that at all. When stockfish says your move suck, claude would have a hard time saying the opposite (no "you are absolutely right" when I am not).
That was exactly my estimation when I played "raw" against Opus. But Opus with stockfish as a tool and much time available can, from what I experienced, generate good comments.
Very cool work!
Did you write the skills (text, and code) all by hand, or are those prompt outputs?
You could also download my free chess app that hooks up to Claude for real time game analysis or chat about the game. https://pwalessi.itch.io/chess-buddy
Thanks for sharing! So you gave stockfish to Claude too? Did you try other techniques?
Looks really cool, would love to try it but I'm on Mac so would appreciate Mac version.
Just a note that this is not novel. Automatic analysis of chess games built on traditional engines has existed for a long time. Without using stockfish as a sidekick, this wouldn’t be interesting because of the poor play of LLMs, however having an LLM enhance the commentary is interesting and I suspect has already been done by chess.com for years.
This what I think too. I also think there are much more refined approaches than the one I tried here. From a cursory look, I saw there are both scientific litterature on the subject of mixing llms and tools like stockfish and some dedicated closed platforms that put that into action.
Let's make clear that I did not spend much time on this project. Ideally I would have tried to put other models into the mix, like maybe Maia to better see the game from a "real player" eyes and pinpoint where expected move and stockfish moves differ.
Anyway, to me, it's good enough to be usable and shared.
How does it work? Does it just use stockfish under the hood? I just remember the times when I tried to use an AI model to play chess and half of the time it either hung a piece or made an illegal move.
Was a bit to overzealous... Just read the readme. Looks pretty cool. Maybe I'll give it a try and see how it compares to the chess.com analysis (although I have to say that I never really use it).
"The result is not perfect" as LLMs are
"The fact that it reflects on my own thinking during the game makes it interesting from a teaching point of view, so I thought it was worth sharing." As AI the premium sycophantic scammer does
"but for me it is a much more pleasant and memorable experience than clicking around Stockfish branches"
And SO much less effective than doing the harder, more tedious feeling work
The jury is out for the effectiveness, it's hard to debate this subject. From my cognitive background I know very well how important the generative effect is for learning. Maybe I'll add features that leverage generative/testing effect one day. Anyway, clicking on stockfish branches can be quite a passive activity too if done badly. I don't know how to do that well to be honest. My goal is often to just to understand what I missed, full stop.
For the sycophancy, I can say I did not feel that at all. When stockfish says your move suck, claude would have a hard time saying the opposite (no "you are absolutely right" when I am not).
Generative AI is really good at making you feel like you understand a lot without actually making you understand anything in-depth
Because lengthy difficult cognitively intensive labor is required for the brain to actually change its structure and connections
I am very interested in this, but it looks like the video link gives 404
Weird, the video is supposed to be embedded. On my chrome desktop it displays properly. EDIT: fixed
The elo of opus 5 models is 1300 or so. I wouldn't take chess lessons from a 1300.
That was exactly my estimation when I played "raw" against Opus. But Opus with stockfish as a tool and much time available can, from what I experienced, generate good comments.