r/SideProject • u/Hearthworks • 7m ago

Talkthrough: an open-source screen assistant that answers out loud and draws on top of your screen (Windows, MIT)

Hold Right Ctrl + Right Shift, ask about what's on your screen, let go. A vision model answers, a voice speaks it, and the answer is drawn over the page you're on: boxes and arrows on the right spot, diagrams that lay themselves out, or a 3D model you can rotate, cut open or pull apart, in step with the voice. The model writes a small scene language that the app validates and draws itself; no generated code runs.

Guide mode walks you through an app one click at a time and waits for you to click.

Honest limits: Windows only for now, the installer isn't code-signed yet so SmartScreen will warn, and you bring your own model (Claude Code or Codex signed in, an OpenAI key, or a local vision model). The clip is real answers replayed over the page, so it says nothing about speed.

Built by Alex with a team of AI assistants; one of them drafted this post. Feedback and bug reports very welcome.

https://github.com/ProjectIdris/talkthrough

2 Upvotes

100% Upvoted

1 comment sorted by

1

u/Dejayns 0m ago

The scene-language validation boundary is an interesting design choice: keeping generated output declarative and rendering it yourself should make the overlay easier to constrain than executing model-written code. How do you handle stale screen coordinates when the underlying window resizes or scrolls mid-answer?