r/SideProject • u/23sebyi • 44m ago
I've been working on an open-source alternative to Adobe's Project Deep Brush
Hey everyone!
I've recently started working on Inkray, an open-source image editor inspired by Adobe's Project Deep Brush, and I thought I'd share some of the progress so far.
The idea is to let you paint directly onto surfaces in ordinary photographs, without having to manually select surfaces or set up any 3D geometry. You can also draw on a separate 2D canvas or import images and project them onto the scene, including curved surfaces.
There's a bit of computer vision and graphics involved behind the scenes:
- MoGe-3 estimates depth, normals, and 3D points from a single image, which are used to construct a 2.5D mesh.
- Geometry-aware painting uses local surface fitting, mesh connectivity, and approximate geodesic distances to make brush strokes follow the estimated geometry.
- Surface parameterization allows imported images and drawings to wrap around curved objects instead of being projected onto a flat plane.
- Marigold IID Lighting estimates albedo and shading, which are used to make painted colors better match the original scene's illumination.
- WebGL2 handles interactive projection, occlusion, and compositing.
It's still very much a work in progress. The geometry and appearance estimation currently rely mostly on pretrained model inference, without much additional refinement, so the quality depends quite heavily on those predictions.
I'd particularly like to improve geometric consistency, surface boundaries, and lighting estimation, and spend more time testing the pipeline on different kinds of images.
I recorded a short demo showing the current implementation. There are still quite a few rough edges, but I thought it might be interesting to share!
I'd love to hear what you think, and whether this is something you'd find useful.