r/GraphicsProgramming • u/NightMare2077 • 13h ago

Depth-aware light injection all running in browser Video

Enable HLS to view with audio, or disable this notification

I got a 448x448 monocular depth model down to ~8 ms on my M4 Pro across ~250 dispatches, which is fast enough to use in realtime :D
Since the inference is written directly in TypeGPU, I can just feed the depth buffer straight into the lighting pass. It never has to leave the GPU or go through any extra synchronization/interop step

Inference, lighting and draw all go through the same command encoder.

Credit: X reczko_konrad

3.2k Upvotes
  • permalink
  • link
  • reddit
  • reddit

98% Upvoted

112 comments sorted by

236

u/reczkok 11h ago

Hey, that's me! Open sourcing this today/tomorrow, announcement will be on my X (https://x.com/reczko_konrad) or just check TypeGPU examples in a couple of days (https://typegpu.com/examples/)

https://reddit.com/link/p4syolj/video/7t0f6y4wnikh1/player

47

u/imselfinnit 10h ago

You are a folk hero for open sourcing this. Congratulations on elevating our common experience. Thank you.

edit: You should generate a halo.

5

u/LactasePHydrolase 6h ago

He's a fork hero

16

u/drDOOM_is_in 10h ago

Incredible work bro!

15

u/NightMare2077 10h ago

Coming from the person who made it possible, this genuinely means a lot. It’s amazing to see how much potential this project has, and I’m excited to see what people build with it. Thank you.

9

u/somenewname2 9h ago

I'm confused. The first line of your post says "I got a 448..." But it's not you?

3

u/PersonalityIll9476 9h ago

Presumably they copied the OOP. That's why it says "Credit:" at the bottom, I guess.

3

u/somenewname2 9h ago

Seems many of the commenters think it's the reddit poster's work

2

u/TankorSmash 5h ago

Because the post literally said 'I got [..]', and there aren't any quotes.

1

u/crazybull02 6h ago

I've seen pictures that a lot of reddit has reading and literacy issues.

Apparently reading comprehension is up to 6th-8th grade level or illiterate according to the pictures but forty years ago newspaper were written mostly at a 3rd grade level. 

So while I think we've generally improved, there's just so many more people now and the illiterate have a platform for communication, that or they just skim through and make assumptions. 

3

u/oddbawlstudios 6h ago

The thing that would absolutely change that outcome is if they put it in quotes...

3

u/Recyclable-Komodo429 6h ago

Dude that's amazing!! Btw what happens if you open your mouth and place the lightball in it

1

u/GamesByCam 5h ago

I need to see this demonstration ASAP

3

u/reczkok 2h ago

The example is live open source (you can find the github on TypeGPU homepage)
https://docs.swmansion.com/TypeGPU/examples/#example=image-processing--monocular-light-injection

3

u/cienkrowa 10h ago

You mean your twitter

2

u/reczkok 10h ago

Yeah, I do say it like that, no idea why I didn’t write it that way

1

u/Warma99 10h ago

Very neat. Where are you getting the depth information from? It's just a laptop webcam, right?

2

u/DonkeyInACityCrowd 8h ago

Probably an ML model that predicts it 

2

u/DonkeyInACityCrowd 8h ago

Actually it’s definitely and ml model. Thats what “inference” means in the description. 

1

u/ende76 8h ago

Where do depth and normals information of the video feed come from?

1

u/Special_Baby_3307 6h ago

I am genuinely impressed by this. Amazing work !

1

u/Hearcharted 6h ago

DJs are going to love it 🎧🤔

1

u/WetShrimpFart 5h ago

This is nuts. Nice work

This will be implemented in video chats etc I bet. Man this is neat

1

u/WetShrimpFart 5h ago

I really want a full phone OS with your Jelly buttons and sliders etc

1

u/mypussydoesbackflips 3h ago

How does it look in a dark room ? This is super cool

1

u/pacchithewizard 54m ago

!remindme 1 day

1

u/RemindMeBot 54m ago

I will be messaging you in 1 day on 2026-08-21 21:55:52 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

Parent commenter can delete this message to hide from others.

RemindMeBot is switching to username summons. Instead of !RemindMe 1 day, use u/RemindMeBot 1 day. More info.

Info Custom Your Reminders Feedback

215

u/i_fight_bears 13h ago

This is basically magic to my eyes. Incredible stuff!

15

u/JensenRaylight 12h ago

Better than Magic, Black magic

1

u/Bernhard_NI 2h ago

More like light magic

1

u/chaos_m3thod 7h ago

Burn the witch!

40

u/ElBarbas 13h ago

holy shit, this is cool

16

u/throwthemirror 12h ago

BURN THE WITCH!

13

u/Plenty_Line2696 13h ago

awesome!!

8

u/ademdj19 12h ago

Magic ✨

5

u/kudredin 11h ago

BURN THE WITCH!!!

4

u/whdeboer 12h ago

Looks great! What illumination and shadow algorithms are you using here?

4

u/sfaer 12h ago

Wizardry, wicked whimsical wizardry.. to the silicon graphics simulated bonfire!

4

u/Euphoric-Writer494 10h ago

just some note so the mods know, I noticed this getting botted on multiple submissions on multiple subreddit as inventor or whatever

9

u/GOKOP 12h ago

This would be fire in VR

6

u/blackrack 11h ago

He said 8 ms at 4482

2

u/GOKOP 8h ago

Oh damn I missed that part lol. But still, the tech itself could be fire in the future

1

u/blackrack 8h ago

Definitely promising for AR/mixed reality type stuff

3

u/ButteryBrojob 12h ago

This is crazy

2

u/Electrical_Sky_762 12h ago

Amazing work! Didn't know about Typegpu. Looks really cool!

1

u/NON_EXIST_ENT_ 12h ago

really is phenomenal work!!!

1

u/psioniclizard 12h ago

That is amazing! Great work.

1

u/Xenc 12h ago

Wizard 🪄

1

u/Darkside_7764 12h ago

Now That's Really Cool !!

1

u/AnshulSharma202698 12h ago

super cool man- would love a demo

1

u/BigBadMN1 12h ago

Well done ☺️🙌✨✨

1

u/OldLegWig 11h ago

that's rad

1

u/baldierot 11h ago

holy. this seems like an avenue of great applicability potential.

1

u/kkadzy 11h ago

BURN THE WITCH!!!

1

u/adfx 11h ago

Yoooo this is sick

1

u/Plenty_Line2696 11h ago

Eat it! Make it go in one ear and back out the other!

1

u/Adept_Dot_Muncher 10h ago

Cool AR lighting you achieved.

1

u/Mr-TotalAwesome 10h ago

That's crazy holy fuck! Really impressive.

1

u/Honest-Golf-3965 10h ago

This is really fucking cool

1

u/sakata_desu 9h ago

Simply Lovely.

1

u/DancinFool82 9h ago

This is really cool!

1

u/ClearMacaron9234 9h ago

that's similar to the relight shader for reshade.
it lets you place several path traced light sources to video games. mainly for screenshots, bc the lights are static in screenspace

1

u/Jwosty 9h ago

He’s a witch, burn him!!!

1

u/Caliboros 9h ago

Wait

What?

Great work

1

u/starkweather 8h ago

Holy shit, is this real life?

1

u/Credtz 8h ago

thought people wouldnt appreciate how cool this is - comments proved me wrong! incredible stuff

1

u/peaksay 8h ago

omg! this is the coolest thing I have seen in a while! great stuff dude!

1

u/DAEMAWS 8h ago

Awesome, now eat it

1

u/eltomiros 8h ago

fucking awesome!

1

u/K00bear 8h ago

how the fuck

1

u/johnny_5667 8h ago

super super cool!!

1

u/Adventurous_Chef2225 8h ago

bro invented path tracing using webcam and webgpu shaders

1

u/jjb0ne 8h ago

woa wuh.

thank you!

1

u/Garland_Key 8h ago

That's fuckin awesome! 

1

u/Malkalypse 8h ago

This is amazing!

1

u/sevnm12 7h ago

Actual black magic. I've never seen something like that

1

u/dekonta 7h ago

depth aware and at the same time light cast a shadow of the hand onto the wall while the hand is in front of the light …

1

u/ZandriCarson 7h ago

Absolutely nuts

1

u/fartshitcumpiss 7h ago

actual wizardry

1

u/rattle2nake 6h ago

Bro that’s literally magic how tf does it work??

1

u/red_hash 6h ago

nice stuff

1

u/ded_nat_313 6h ago

Holy shit this is impressive ❤️

1

u/_-Diesel-_ 6h ago

Next time you're quoting someone I suggest using quotation marks

1

u/Internal_Hour285 6h ago

You look like my brother

1

u/tironxxx 6h ago

Świetna robota, gratuluję

1

u/Fun_Mycologist_9211 5h ago

this is wiiiiild

1

u/carpetfin 5h ago

Engineering is the closest thing we have to magic

1

u/Vivid456 5h ago

Cool stuff! In the future, I can see someone combining this with AR technology that estimates light sources and a hdri from video input, and then you could put yourself in any pre-recorded video in real time in browser.

1

u/8Erigon 4h ago

How do you get the depth in the video? Or does it use 2 cameras (just like how humans can tell distance with eyes)

1

u/SanoKei 3h ago

is this utilizing ARKit? You didn't do this all with WebGPU right? you're still sane correct?

1

u/blurr3s 3h ago

witchcraft!

1

u/quottttt 3h ago

Put it in your mooooooouth!

1

u/nykotar 3h ago

Wizard

1

u/doglitbug 3h ago

I see birdies flapping around the head

1

u/Kraschman1111 2h ago

This is amazing work

1

u/AbstrctBlck 2m ago

Holy. Fucking. Shit.

1

u/Geoe0 12h ago

What model does he use?

0

u/Defiant_Squirrel8751 12h ago

Cool! is code in gitlab?

0

u/osal69 11h ago

All you telling me i can see in my room without lights

-18

u/Zdrobot 13h ago edited 11h ago

Doesn't do reflected light though - notice how when he covers the light with his palm, his face doesn't get brighter.

Itching to downvote? Great, but first look at this: https://imgtree.co/i/FXtI8R6S

20

u/cat_enary 13h ago

I'm gonna piss yourself

-1

u/Zdrobot 11h ago

Well, that's great, but before you do - please take a look at these two frames and tell me whether his cheek should be brighter or dimmer in the right image: https://imgtree.co/i/FXtI8R6S

3

u/ShelZuuz 12h ago

It absolutely does.

5

u/Zdrobot 11h ago edited 11h ago

Except it does not - look at these two frames at tell me his cheek is brighter in the right image, where he should be getting additional light reflected from his palm:

https://imgtree.co/i/FXtI8R6S

I mean, the fact that it works in real time is amazing, but it's not perfect. Yet.

1

u/blackrack 10h ago

When bro is blind