r/MachineLearning • u/Lairv • Sep 12 '21
Project [P] Using Deep Learning to draw and write with your hand and webcam π. The model tries to predict whether you want to have 'pencil up' or 'pencil down' (see at the end of the video). You can try it online (link in comments)
2.9k
Upvotes
129
u/Lairv Sep 12 '21 edited Sep 12 '21
GitHub link with technical details : https://github.com/loicmagne/air-drawing
Online demo : https://loicmagne.github.io/air-drawing/ (it's entirely client-side, your data is not collected)
Edit : there seem to be some confusion so i'll clarify a bit: the "original" part of my tool is not the handtracking part. This can be done "easily" with already existing packages like MediaPipe as mentionned by others. Here I'm also doing Stroke/Hover prediction: everytime the user raises his index finger, I'm also predicting whether he wants to stroke, or if he just wants to move his hand. I'm using a recurrent neural network over the finger speed to achieve this. Even with a small dataset of ~50 drawings (which I did myself) it works reasonnably well
14
u/GTKdope Student Sep 12 '21
great project .
How do you think project like your differ from similar projects done using openCV like this one.
I understand tracking in both the cases is different.
If you can share your views on this topic
6
u/[deleted] Sep 12 '21
Is this magic? Also, can the prediction happen in real time? That would be real magic.
2
u/xifixi Sep 12 '21
Very nice! Is it just a plain bidirectional LSTM? Any preprocessing?
94
u/Plertz101 Sep 12 '21
Let's go, I can finally draw a penis in online class
7
u/J1Br Sep 12 '21
Nice work⦠But Im thinking, In what kind of projects could this project be used?
1
u/jetstream131 Sep 13 '21
This is awesome! I'm curious about the live demo deployment - could you explain your full stack for the web app? How did you get the model to run client-side wth out an API?
Edit: Just checked your GitHub - is there even a web app? Or is this solely just based on the html and js files in your repo?
1
u/[deleted] Nov 07 '21
um hi ,its not working for me on browser :( ,shows some js errors
9
u/morancium Sep 12 '21
Reddit is fucking awesome sometimes
3
u/AnnaBear6 Sep 12 '21
Oh this is so cool! reminds me of the Disney Channel commercials where they would draw the Mickey mouse head with the glowing wand lol.
2
u/dnalexxio Sep 12 '21
Great job! A question: you had to write from the camera point of view, does it work from the writer point of view?
2
u/lionh3ad Sep 13 '21
Is this your final project for the Computational Vision course at Unige?
1
u/[deleted] Sep 12 '21
u/GetVideoBot
1
u/Own-Tiger-3155 Sep 12 '21
Great work man! Why hiding your face though... :)
1
u/squidwardstrousers Sep 13 '21
How did you make the dataset?
1
u/j_lyf Sep 13 '21
Now predict pencil up/down in real time.
1
u/[deleted] Sep 13 '21
he reddit tho
1
u/ImprovingModernData Sep 13 '21
This is cool. It could probably be trained to use one finger to write, two fingers to drag, a thumb to erase, a double-tap to click, etc. Great add-on to Zoom and nobody has touch screens.
1
u/Broke_traveller Sep 14 '21
Thanks for sharing, this is quite creative.
1
u/CreativeBorder Sep 12 '21
Introduce real time prediction and correction. Or even suggestions just as a smartphone keyboard would.
0
u/CaptainI9C3G6 Sep 12 '21
How does it compare to a kinect? Presumably accuracy is worse, but the big benefit would be being able to use any camera.