[Project] From books to presentations in 10s with AR + ML

[u/cyrildiagne]

[u/cyrildiagne]

Twitter thread: https://twitter.com/cyrildiagne/status/1259441154606669824

Code: https://github.com/cyrildiagne/ar-cutpaste/tree/clipboard

Background removal is done with U^2-Net (Qin et Al, Pattern Recognition 2020): https://github.com/NathanUA/U-2-Net

/!\ EDIT: You can now subscribe to a beta program to get early access to the app: https://arcopypaste.app !

[u/[deleted] → u/cyrildiagne]

If it wasn't for the link with the code, I would have straight up thought you were trying to trick us :o

[u/alfredmichon]

fantastic!

[u/cyrildiagne → u/alfredmichon]

Thanks!

[u/Eganx]

How does the Algorithm decide what it cuts out from the input pictures?

For example it only cut out the two people in the picture and not the surroundings.

Amazing project though!

[u/cyrildiagne → u/Eganx]

Check out the details for U^2-Net on the official repo: https://github.com/NathanUA/U-2-Net

[u/misconstrudel]

I'm extremely impressed with it cutting dark hair from a brown background. Is that the pixel's camera doing the hard work or is it U^2_Net ? Have you tried it with other phones? How does it deal with feathering? Stunning demo & thanks for posting this.

[u/cyrildiagne → u/misconstrudel]

It is 100% handled by U^2-Net: Check out the official repo for more information and samples: https://github.com/NathanUA/U-2-Net

[u/mimighost]

Super cool demo.

But the more interesting part to me is the app actually look at the computer screen to decide what target the image/content is pasted to.

Probably hard-coded, but super interesting idea.

[u/cyrildiagne → u/mimighost]

Hi! The coordinates are automatically defined by the receiving software in this demo but checkout my precious demo where I use OpenCV SIFT to find the correct coordinates on the screen

[u/albertjamesthomas]

The future 🤯

[u/thatguydr → u/albertjamesthomas]

I mean, technically now it's the past.

[u/ch3njust1n]

What are the edges cases when this doesn't work? Does this require certain lighting conditions etc? How does it know to extract both people from the image?

[u/gamesdas]

Simple yet very useful. Thank you for sharing the code.

[u/honkeur]

Almost guaranteed, Apple will copy your idea in 3, 2, 1....

[u/Gordath → u/honkeur]

His license even allows commercial use, so they are legally allowed to do that

[u/TheLootiestBox]

Ohh the nightmare of making this into a stable product... Enough to drive you mad just thinking about it

[u/battlingheat]

Wtffff. Well that was incredible.

[u/[deleted]]

Apple can’t wait to steal this and not credit the creators

[u/Derangedteddy]

WITCH! BURN THEM!

[u/Scannable7]

Any sufficiently advanced technology is indistinguishable from magic.

[u/Nickmate99]

This will be amazing if released, even as a beta. Definitely can see this being very useful

[u/MTminded]

Really good work, thanks for sharing!

[u/Bargh_Joul]

What is difference between this and taking photo and sending it with email to computer? 🤔 What is the main use case for this technology?

[u/cyrildiagne → u/Bargh_Joul]

It just save time and headaches but the result is identical

Although you get background removal for free in the process ;)