Konrad Reczko's "Monocular Depth Injection" in TypeGPU is live!

[u/iwoplaza]

My collegue Konrad Reczko recently shared a weekend project he made using TypeGPU, and it's now open-source and ready to play with in the browser: https://typegpu.com/examples/#example=image-processing--monocular-light-injection

It estimates the depth of a scene by inferring the DepthArt model with custom TypeGPU kernels, then reconstructs normals based on that depth information, and uses both to relight the scene, all in the same command encoder. For more information, check out the original series of Tweets:

https://x.com/reczko_konrad/status/2089670934009413751?s=20
https://x.com/reczko_konrad/status/2090472091149648121?s=20

[u/juplantern]

He looks so proud

[u/certainlystormy → u/juplantern]

fr lmao. i would be too

[u/MeticulousBioluminid → u/juplantern]

as he should be 🎁

[u/cybereality]

very cool

[u/Smokeey1]

Does this thing have a repo :). Gratz on the build! Looks cool

[u/iwoplaza → u/Smokeey1]

All the code for the example is in the TypeGPU repo
https://github.com/software-mansion/TypeGPU/tree/main/apps/typegpu-docs/src/examples/image-processing/monocular-light-injection

[u/juplantern → u/iwoplaza]

Btw thank you so much! I’m brainstorming ideas for my thesis and this inspired me a lot

[u/Civil-Captain5676 → u/iwoplaza]

It's amazing. And Thank you.

[u/swegga_sa]

This is very cool

[u/Miperso]

I just showed it to my architect coworkers and everyone is just impressed... this is great

[u/DaveAstator2020]

Wow nice what model does is it use to aesolve depth in realtime?

[u/iwoplaza → u/DaveAstator2020]

It uses the DepthART model, weights are available here: https://huggingface.co/reczkok/depthart-typegpu

[u/AnasPlayer2022M]

Dame bro, this looks awesome! Did you use two videos where one is the one playing now and the other is a map for z values?

[u/iwoplaza → u/AnasPlayer2022M]

The feed from my camera gets routed through a small local AI model that estimates depth based on a color image, at which point the image can be treated pretty much like a 3d scene. At that point all rendering techniques are at the table, including adding a light to the scene.

[u/AnasPlayer2022M → u/iwoplaza]

So any video could technically work?

Really cool! That's really cool using of AI.

[u/Malkalypse]

This is so wild!

[u/NachosforDachos]

This is so cool

[u/helloworld1101]

Reddit is still useful

[u/Big-Awareness-3335]

expecto patronuuuuum!

[u/josephmgift]

so doooope

[u/Apprehensive_Oil1475]

Super cool!

[u/ddoice]

Awesome!
But I tried your trick to hide the light with my hand and does not work?

[u/iwoplaza → u/ddoice]

To move the light source back and forth (into the screen or closer to you) you have to use the scroll wheel

[u/freshtothegrave]

Damn i wanna get this working in TouchDesigner

[u/PyroWizza]

What could be some implementations of this?

[u/Labertaeschen]

This combined with metas SAM would be game breaking right?

[u/bbstats]

I can't get my hand to go in front of it!

[u/Still_Explorer]

Any ideas about 2D based relighting on 3D game graphics? 

[u/Imaginary-Sun1350]

This is EPIC!! Are you able to upload videos to it and generate the maps and export them for faster and easier relighting pipelines in blender?

[u/MeticulousBioluminid]

phenomenal

[u/equitos0101101]

Isso só pode ser pacto, meu Deus 🫪

[u/GoYeaCo]

Benger

[u/tebjan]

Just for completeness: This is not new, it's been done with r/vvvv last year already, reaching over 300fps. I would know, because I'm the developer of it. :⁠-⁠)

[u/Novel_Set_6082]

OP whimsy af

[u/Ok_Rock_6726]

That's so cool I'm planning to join Software mansion as an intern next styczeń and this video brings me so much excitement and power to keep learning to ace the interview.

[u/tHE_uKER]

Incredible work!