This is crazy. My claude chess stuff is currently near yours on the front page and noticed your post. I have a project very close than yours that I hesitated to share. From a quick glance, we went for a similar approach. I just open sourced it so that you can compare implementation notes. https://github.com/brumar/whiteboard-agents . It's not thoroughly tested but can be interesting to check.
I've had a fairly thorough exploration of how I can give my agent a whiteboard so we can work on architectures together. To my surprise, current solutions (including Excalidraw) were not good enough and didn't deliver what I wanted. I ended up finding Mermaid to be the most agent-friendly medium and coded an Obsidian plugin for it. It works okay; we can work on the same doc while I bring my own AI agent, and we can brainstorm together.
https://community.obsidian.md/plugins/mermaid-relay
YMMV, but the value I get from producing a diagram is derived from the thinking. Thinking about what I'm trying to draw leads to understanding about where my/my team's knowledge is poorer, what assumptions we're making, etc.
I don't remember who said it, but someone mentioned that mathematics papers don't read well because the models don't know which parts to emphasize and which parts to gloss over. I would imagine similar tendencies here.
Yeah I tend to make my diagrams either as I go, as exploration, or after I'm done figuring things out as a summarizing artifact (and agents are great at using Mermaid to do that second one).
What frustrates me about many projects is it's never been easier to just make a quick video or image or gif and put it in the README to show what the value is. This was at least somewhat complex before AI but now you can just paste an image and have the AI put it in your README.
I guess the frustrating bit is that the readme is 100s of lines long. AI is already saving you so much time on a project like this but you still make me clone and spin it up if i want to see it for real.
I get that open source is a gift and you’re not obliged to do anything I say but please consider taking the extra few minutes on your next project.
Wow, I just can't agree with this take at all. The README is only 161 lines (very short for a git repo README!) and took me about 15 seconds to skim and decide whether or not it was interesting enough to keep the tab open.
Most of the project pages I have seen with videos either animate things at hyper-speed, meaning you can't actually follow is what happening, or drag on for minutes just to communicate something that I could have read in seconds. My time is more valuable than that.
Images are cool, however. We should do more of those.
To that end, I've found it super easy to tell $LLM to make me a video showcasing the main features of whatever I'm building, have it write subtitles and give it a voice over. Obviously for a README a simple gif with burned in subtitles/annotations would be more useful.
Especially if the project is web-based it can just spin up playwright and record its actions.
Posted this somewhere else aswell but I've had good success with https://whiteboard-mcp.com - it seems to work better than other free alternatives for things like architecture diagram creation
I’ve been trying to find the best practice to connect a person’s existing ChatGPT/Codex / Claude accounts so they can be used in an app for the agent in the app.
Any advice? Everything seems clunky, even the best MCP efforts
best approach is to create an webapp that uses WebMCP and then use ChatGPT Desktop. you open the app using it and it and it can use all the tools. Then you implement navigation in the tools and you will see ChatGPT using your app pairing with you.
tldraw has a lot of programmatic access I have not compared it to excalidraw but there is still no equivalence between the model capabilities to build amazingly detailed infographics and diagraming tool use which is a gap I hope will narrow
This is crazy. My claude chess stuff is currently near yours on the front page and noticed your post. I have a project very close than yours that I hesitated to share. From a quick glance, we went for a similar approach. I just open sourced it so that you can compare implementation notes. https://github.com/brumar/whiteboard-agents . It's not thoroughly tested but can be interesting to check.
Excalidraw offers their own open source first party MCP endpoint[0] and server[1]:
[0]https://mcp.excalidraw.com
[1]https://github.com/excalidraw/excalidraw-mcp
I've had a fairly thorough exploration of how I can give my agent a whiteboard so we can work on architectures together. To my surprise, current solutions (including Excalidraw) were not good enough and didn't deliver what I wanted. I ended up finding Mermaid to be the most agent-friendly medium and coded an Obsidian plugin for it. It works okay; we can work on the same doc while I bring my own AI agent, and we can brainstorm together. https://community.obsidian.md/plugins/mermaid-relay
YMMV, but the value I get from producing a diagram is derived from the thinking. Thinking about what I'm trying to draw leads to understanding about where my/my team's knowledge is poorer, what assumptions we're making, etc.
I don't remember who said it, but someone mentioned that mathematics papers don't read well because the models don't know which parts to emphasize and which parts to gloss over. I would imagine similar tendencies here.
Yeah I tend to make my diagrams either as I go, as exploration, or after I'm done figuring things out as a summarizing artifact (and agents are great at using Mermaid to do that second one).
Yes I agree! Drawing helps me think.
What frustrates me about many projects is it's never been easier to just make a quick video or image or gif and put it in the README to show what the value is. This was at least somewhat complex before AI but now you can just paste an image and have the AI put it in your README.
I guess the frustrating bit is that the readme is 100s of lines long. AI is already saving you so much time on a project like this but you still make me clone and spin it up if i want to see it for real.
I get that open source is a gift and you’re not obliged to do anything I say but please consider taking the extra few minutes on your next project.
Wow, I just can't agree with this take at all. The README is only 161 lines (very short for a git repo README!) and took me about 15 seconds to skim and decide whether or not it was interesting enough to keep the tab open.
Most of the project pages I have seen with videos either animate things at hyper-speed, meaning you can't actually follow is what happening, or drag on for minutes just to communicate something that I could have read in seconds. My time is more valuable than that.
Images are cool, however. We should do more of those.
To that end, I've found it super easy to tell $LLM to make me a video showcasing the main features of whatever I'm building, have it write subtitles and give it a voice over. Obviously for a README a simple gif with burned in subtitles/annotations would be more useful.
Especially if the project is web-based it can just spin up playwright and record its actions.
Posted this somewhere else aswell but I've had good success with https://whiteboard-mcp.com - it seems to work better than other free alternatives for things like architecture diagram creation
I’ve been trying to find the best practice to connect a person’s existing ChatGPT/Codex / Claude accounts so they can be used in an app for the agent in the app.
Any advice? Everything seems clunky, even the best MCP efforts
MCP is the answer but it's only really worth it if you product is reasonably complex e.g. n8n has a community MCP fully for this reason.
best approach is to create an webapp that uses WebMCP and then use ChatGPT Desktop. you open the app using it and it and it can use all the tools. Then you implement navigation in the tools and you will see ChatGPT using your app pairing with you.
video https://lnkd.in/p/ez4EZede
tldraw has a lot of programmatic access I have not compared it to excalidraw but there is still no equivalence between the model capabilities to build amazingly detailed infographics and diagraming tool use which is a gap I hope will narrow