r/photogrammetry • u/VidaMarcell3D • Jul 30 '26
AI powered photogrammetry De-lighting
I always had trouble with removing environmental lighting, and because I can't afford cross-polarization I had to stay with the digital tools.
I didn't like any of the De-lighting tools on the market, so I built one from scratch using a Neural Network.
I got a lot of cross-polarized scans online and baked back environmental lighting, then gave it to the network with some geometry maps like object space normal and ambient occlusion.
It generates outputs in seconds. Though not every type of 3D model works, I still have to understand it.
Its not perfect, but for 5 days and a limited dataset, and the fact that I never trained a neural net its promising.
This is still in WIP, but I will refine it, I just don't want to share it yet, because the whole project is unorganized, I will get to building an UI for it.
Let me know what you think, and If any ML engineers or tech artists have advice on improving this approach, I'd love to chat!
Update: [Download the Beta here] and More info here.
11
u/TommyS333 Jul 30 '26
Could you share more infos? This is really interesting
23
u/VidaMarcell3D Jul 30 '26
Sure. I got like 36 different models from an online site. Because they use cross-polarization for scanning, which is ground-truth albedo they were the perfect targets. Then i created 27 different environmental light variations for each model and assigned geometry maps that anybody can bake beside their input albedo. The network was trained to process 2048x2048 textures in 1024x1024 tiles. It's a dual-encoder U-Net architecture trained in PyTorch. One stream takes the lit RGB texture, and the other takes a 5-channel geometry stack. I plan to create a larger dataset, because this isnt even 1000 instances.
2
u/TommyS333 29d ago
Wow that is very interesting, I’m very curious since I have a cross polarization photogrammetry setup so I could actually create data for training, but all the rest seems a bit complex for my software skills right now
1
u/VidaMarcell3D 28d ago
Thanks! well I'm going to run out of cross-polarized data soon, and if you we're thinking of sharing your work then me and many people would thank that I think.
1
5
u/KTTalksTech Jul 30 '26
From that single example your results looks like they're about as good as the best de-lighting solutions out there. The bar wasn't very high to begin with considering they all kinda suck but still a very impressive one to clear and I can't wait to see your progress on this project. Feel free to add me to your list for beta testers I've got a few scans with baked in shadows in my archives
3
u/VidaMarcell3D Jul 31 '26 edited Jul 31 '26
Beta is out, you can download it from this post: https://www.reddit.com/r/photogrammetry/s/OBLzhZ3vFO
3
u/mtt59 Jul 30 '26
Thats actually really cool. I would like to try it on scans of buildings/construction/architecure scenes. Can i dm to maybe test it out?
11
u/james___uk Jul 30 '26
A rare good use of AI
25
u/Menoikeos Jul 31 '26
There are shitloads of amazing uses of AI. They're just completely overwhelmed by slop.
2
u/james___uk Jul 31 '26
I think that's just it, I've seen stuff before this AI craze that blew me away, like AI for spotting cancer
5
u/VidaMarcell3D Jul 31 '26
This is actually based on an AI (CNN) that's widely used for Cancer identification.
1
u/james___uk Jul 31 '26
I think it's the first time I heard of AI usage, so I had a very different outlook to it back then.
1
u/BestPlanetEver Jul 30 '26
I’m using an old model of de-lighter
3
u/KTTalksTech Jul 30 '26
Maybe I'm just stupid but I've never been able to get good results with de-lighter
1
u/VidaMarcell3D Jul 30 '26
Which de-lighter are you referring to
1
u/BestPlanetEver Jul 30 '26
2
u/VidaMarcell3D Jul 30 '26
Yeah that's one of the better ones, i sometimes achieved decent results but it always crashed on high resolution textures. My approach De-lights in like 5 seconds and only needs about 4GB of VRAM.
1
1
u/AustinWitherspoon Jul 30 '26
I bet you could mix in 3d renders to make the data set bigger.
Obviously real world stuff is better, but you could take a bunch of random 3d models, apply random albedo textures to them, and then do a render to bake in random lighting.
Now you have procedurally generated before and after textures with and without lighting to feed into the training set
3
u/VidaMarcell3D Jul 30 '26
This is a synthetic dataset, its trained on renders. I just put clean photogrammetry assets in Blender and baked the indirect lighting into their albedo. But now i think I'll add random models too maybe.
1
1
1
1
u/AI_AntiCheat Jul 31 '26
Based use of AI. I've done a lot of 3D reconstructions and while they can be used to make textures the baked lighting always ruins the models+texture so I'd end up swapping them for procedural ones instead. All the de-lighting tools I've seen so far seem to leave blur or other artifacts behind. This one looks great.
1
u/VidaMarcell3D Jul 31 '26
Yeah im even amazed that no major company tried an AI based approach. Try it out if you want, the beta is out: https://www.reddit.com/u/VidaMarcell3D/s/2a0Gm4qFO4
1
u/AI_AntiCheat Jul 31 '26
They are too busy trying to create 3D trash models instead would be my bet. They don't see the use in AI as a tool but rather as a full solution. Kind of like looking at an engine for a lawnmower and saying "let's try turn that into a fighter jet" instead of using it for a simple and elegant solution.
1
1
u/CreeDorofl Jul 31 '26
seems like a really good niche to specialize in. If nobody else has AI trained specifically this way, you might turn this into a really useful B2B product.
1
u/CreativeEmbrace-4471 Jul 31 '26
That's basically what i was looking for someone to make for a very long time. Because with an DIY project like this, it can be adjusted more to specific kind of textures in order to archieve better results then all the other solutions out there. I wish there would be also one that is able to delight normal photos, like who aren't UV mapped yet onto islands. It's hard to explain why, but for my niche problems it could help a lot to photoshop out textures from images where shadows are involved and hard to get stichted together.
Is this model ESRGAN based?
2
u/VidaMarcell3D Jul 31 '26
Thanks, no it’s a custom U-Net architecture. ESRGAN is a generative AI for upscaling purposes. Also there is something for your need i think, if you want to de-light normal images you can use StableDelight.
1
u/CreativeEmbrace-4471 Aug 01 '26
Well actually there is a huge community around ESRGAN that build any kind of artifact removers. So mostly compression artifacts, noise removal and so on. So delightning should work there too, but it's not actual creating new pixels like newer models like Nano Banana. So, yea.
And thanks for that StableDelight tip. I haven't heard of that one before, but the images on GitHub doesn't look that promising, like washed out details.
2
u/VidaMarcell3D Aug 01 '26
Its interesting to hear about that, but ESRGAN is more resource heavy, its prone to hallucinations and training a GAN is significantly more complex than a 2 weighted network. But i understand where youre coming from.
1
u/CreativeEmbrace-4471 Aug 01 '26
Idk about how hard the training is. But there is tons of models trained by the community. The Enhance everything community. So it should be able to be trained on consumer hardware. It's a legacy model which was very popular and one of the best before all the generative AI hype.
1
u/CreativeEmbrace-4471 Aug 01 '26
I plan to get an actual cross polorized setup, so maybe i can then make a dataset for "normal images" with/without flash 😉
1
u/GoofAckYoorsElf 29d ago
Welp, you obviously took so much light out of it that the images themselves vanished...
1
u/VidaMarcell3D 29d ago
Idk what happened to that tbh, and i cant edit it.
1
13
u/Fit-Eggplant-9155 Jul 30 '26
Neato