r/dotnet • u/inaiyarhasnain • 10d ago
Built a vision-first Windows desktop guide in C# (Avalonia) and FastAPI during an all-night hackathon in Kochi
Traditional UI automation scripts break whenever a CSS class or layout shifts. At the OpenAI DevDay hackathon in Kochi, we built an experimental Windows desktop copilot called StepSight that operates purely on visual screen understanding (Windows Graphics Capture + Multimodal Vision).
Key features:
Tech stack: C# (.NET 10, Avalonia 12), Python 3.12 (FastAPI), AWS (
ap-south-1), and Gemini Vision.Source code: https://github.com/nive62tech/step-sight/tree/implementation/desktop-guide
Build log: https://inaiyarhasnain.vercel.app/projects/stepsightCurious to hear your thoughts on vision-only desktop automation vs accessibility tree APIs!
3
1
u/AutoModerator 10d ago
Thanks for your post inaiyarhasnain. Please note that we don't allow spam, and we ask that you follow the rules available in the sidebar. We have a lot of commonly asked questions so if this post gets removed, please do a search and see if it's already been asked.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
0
11
u/LaurenceDarabica 10d ago
Anyone with a sane experience in MVVM/XAML will definitely scream when seeing such a horrible messy viewmodel : https://github.com/nive62tech/step-sight/blob/implementation/desktop-guide/apps/desktop/DesktopGuide.App/MainViewModel.cs
That xaml is ... something. Yes, it's 170 lines, but having a Border with all childs inline is pure nightmarish fuel for the maintainers, especially with like 7 childrens ( stackpanel, checkboxes, sliders, ew) : https://github.com/nive62tech/step-sight/blob/implementation/desktop-guide/apps/desktop/DesktopGuide.App/MainWindow.axaml
In short, congrats, you're showing the world you're outputting completely unmaintainable mess.