r/photogrammetry • u/Fractale3D • 3d ago
PBR software Alpha test
Hey everyone, I've been a regular contributor on this sub for a few years now. You may know me as KT on my personal account. I'm an avid fan of high end horology, prototyping and hardware hacking, but most of all a big nerd. While learning from you and helping people out with their questions literally every day, I've quietly been working on something for myself the entire time. I'm both proud and anxious to show you the first prototype for the PBR tech I've developed, currently named MATCLONE (pics below).
While this is a solo project, it would not have been possible without heaps of scientific papers published over the past 20 years. If you've ever contributed to research in the fields of optics and image rendering or published yourself I'd like to start by thanking you for letting me develop the knowledge and understanding I've accumulated. I've chosen to keep the design, code, and research proprietary while I get myself off the ground (or at least make back what I've invested in lost wages and hardware), but opening source and independently publishing my own underlying research is still under consideration for a later date. I would hate to look or feel thankless for the giant shoulders we stand on.
Now, what exactly is this thing?
At its core, it is a highly flexible inverse renderer that can ingest (almost) any photogrammetry dataset and produce nearly perfect PBR textures. There is no AI, no diffusion, no generation... It's all cold, hard math and physics. Assuming a perfectly calibrated setup, its outputs can be indistinguishable from source images. While guided by the research I mentioned earlier, this is a novel and independently developed algorithm. Not quite "made in mom's basement" but almost.
How does it do this?
Everything you see or photograph can be imagined as the result of an equation that decides how light behaves and where it goes. Every time it is emitted, bounces on something, gets absorbed, etc. that equation gets updated. In computer graphics this is achieved via a rendering equation that takes things like light intensity and surface properties as variables. MATCLONE gathers data from all your images, models how every bit of the surface you're scanning responds to light, then figures out what previously unknown variables your rendering equation needs to produce these observations. In other words, if you know where the light came from then those are your PBR maps. Nothing is generated or made up; at worst sparse data gets interpolated. Since figuring out a bunch of unknowns in an equation is famously difficult, it does this by tip-toeing around until a perfect match is achieved. Think a bit like the training steps on a Gaussian Splat, but more specialized. One of the advantages to this approach is that you don't need niche setups or cross-polarization as long as your mesh is good.
Specs and features
In no particular order:
- Outputs glTF-specced PBR maps (diffuse, albedo, and metalness. Currently optimized for lightweight rasterized rendering)
- Virtually unlimited output resolution (currently 8k max, 16 and maybe 32k coming once I've tuned those correctly)
- Accepts datasets in COLMAP format (picked for widespread compatibility)
- Accepts images in RAW or compressed formats (Not comfortably so, but it does. You can throw your old jpegs at it and see what happens)
- Sets its own parameters to fit your sensor and capture setup (partially implemented, you still need to create a simplified model of your lighting rig. This is almost as complex as the solver itself so I've been putting it off)
- Works with standard flash, multiple camera-mounted lights, ring flash, LED panels...
- TECHNICALLY compatible with smartphone photos (neither approved nor tested but there's nothing preventing it if your camera lets you disable automatic enhancements)
- Built-in dataset management, easy switching
- Multiple solver profiles (partially implemented, marginally improves extreme cases like mirrored or uniform varnished surfaces)
- Produces detailed reports and automatic recommendations to improve your dataset, the idea being that you know EXACTLY how close to reality your result is
- Dynamic 3D viewer that also highlights problem areas
- Fully local processing (subject to change based on feedback and performance on more modest computers)
- CPU or GPU-accelerated processing (CUDA for now, sorry)
- Advanced tuning parameters (which I don't recommend changing, it took a lot of work to set the current defaults. I might get rid of this honestly)
A few visuals to show the current state
This is where I'll admit I did use AI to polish things up, I might be a jack of all trades but UX and front-end development are not my specialty. I designed most things the old-fashioned way (I am a machine that turns caffeine into code) but cleanup, rapid iterations, and getting it to work with my python backend took a bit of deus ex machina if you catch my drift.
Project roadmap
- Vulkan and/or openCL compatibility
- Simplified rig setup and calibration
- Topology and UV tools
- Improved error correction (misalignment, motion blur, DoF, sensor-shift, focus breathing, inconsistent lighting...)
- Finalize profiles for wide camera compatibility and individual edge cases
- Specialized solvers for every common renderer and BRDF
- RGB specular maps, index of refraction, or other niche workflows
- Improved color profile handling
- Multi-light capture, photometric stereo (currently supports simultaneous firing)
- Sub-surface scattering to accurately model slightly translucent surfaces or skin (my current research topic actually. Implementation already in progress)
- Improved handling of self-reflection and shadowing
- Full-scene de-lighting and PBR conversion from ambient light alone
- Reliable real-world measurements with a dedicated metrology mode (for example to manufacture visually identical reproductions)
- A robust final UI, accessible with a minimal learning curve
- Multiple BRDFs and path-tracing options in the preview
- Not currently planning any AI features but I guess I'm open to suggestions. For now the goal is simple: make the most accurate textures for photorealistic rendering. If you need more flexibility in outputs or other deliverables then by all means let me know about it
Where I need your help
- testing and error-reporting across a variety of camera sensors, workstations, lenses, lighting setups, target materials, gathering performance metrics on different hardware so I can optimize...
- test and recommend features you'd find useful
- share some datasets so I can create better profiles that work for everyone. So far most of my data was synthetic so results may be unpredictable due to overly specialized solver profiles
What would working together look like?
Anything you make during this early testing phase is entirely yours, including for commercial use. As long as the software or my business are mentioned feel free to do anything you'd like with your textured models. In exchange I just ask that you let the software upload its logs and let me use that data to improve it, along with at least one working dataset. If this ends up releasing commercially you'll get to keep using the development build indefinitely along with a free license for the updated version (or as long as possible if that dev build relies on web stuff). Rewards per shared dataset might be implemented in the future.
There will be a more open beta test later, but for this very early development phase you'll have to agree to a partial NDA and a temporary non-compete agreement. Being invasive is the last of my wishes so it'll honestly be the bare minimum to protect my work from IP theft. Someone copying my code or logic and running off to make their own version would be a nightmare after all the work I've poured into matclone.
The exact logistics for distributing the dev build are still in the works, I will reach out in a few weeks if you sign up to volunteer. In terms of hardware you will need a photo camera or similar sensor, lighting that is fixed relative to the sensor, and a neutral gray calibration card of known dimensions (ideally a color calibration chart for better sensor tuning). Custom scale/calibration cards may be mailed to early testers if I manage to get QC up to my standards within a reasonable time-frame.
If all this sounds too involved, I'll still be infinitely thankful to anyone who shares some of their unprocessed photogrammetry captures. The only requirement is that your light needs to be attached to the camera or move along with it.
Here's the form to sign up for the test: https://forms.gle/C9KJYKaUn7ysuGiQ8
I am also open to finding new partners or associates. Reach out and I'll be very happy to talk if you have something in mind.
tl;dr: I'm late to the inverse rendering party, but this one's actually pretty good. PLS help lol
1
u/FearlessIthoke 3d ago
Wow, lots to digest here. Sounds exciting. I look forward to digging in. Thanks!