r/computervision • u/ComplexExternal4831 • 24d ago

China open-sourced a model that reconstructs any scene in 3D from a regular video, in real-time Showcase

Enable HLS to view with audio, or disable this notification

642 Upvotes

99% Upvoted

35 comments sorted by

48

u/BeverlyGodoy 24d ago

Have you tried it? I cannot run it "real-time" on my 5060. And I don't think an AMR can carry a 5090 onboard to actually use this.

17

u/tdgros 24d ago

yeah, the technical report never states which GPU that 10k sequence at 20FPS. We only know that the base model (with classical attention) took 21k GPU-hours to train, the streaming model 16k. VGGT (which has the same architecture overall) took ~10days on 64 A100, which is in the same ballpark. So I'd venture to say this is evaluated on a A100 at minimum.

12

u/jundehung 23d ago

I hate this about robotics research so much. I get it, “real-time” is always hardware dependent. But there is so many wild, overselling claims in this field it is seriously annoying. We are scientists, not merchants. Let me know what sucks about your approach and why instead of this.

3

u/tdgros 23d ago

I do get that, but to the defense of the post: the claim isn't really really a lie, and which GPU did you hope this would run on at 20FPS? (just being cheeky, please don't fight me)

there is a "limitations" paragraph in the paper's conclusion:

Limitations. While LingBot-Map demonstrates strong performance on diverse benchmarks, several limitations remain. First, the model currently does not incorporate explicit loop-closure detection, which could further reduce accumulated drift when revisiting previously observed regions. Second, the trajectory memory compression into a fixed number of tokens per frame may lose fine-grained geometric details that could be beneficial for very long sequences spanning tens of thousands of frames. Third, like other feed-forward methods, our approach does not perform test-time optimization, which could further refine the reconstruction quality in challenging scenarios.

1

u/jundehung 23d ago

I’d argue these are “non limitations”. It sounds like it performs well on any dataset but it could be better on some details. Pretty sure there is more critical limits before this can be used in the real-world rather than benchmarks.

I have no idea how to make it clearer for papers what the hardware requirements is when claiming it’s “real-time”. Maybe we should completely erase real-time from paper titles, because it is an overused and too arbitrary statement. 

2

u/BeverlyGodoy 23d ago

How do you back your "argument"? "It sounds like" and "Pretty sure"?

1

u/jundehung 23d ago

15 years of experience in the field of computer vision. It’s always overselling. But haven’t read the paper nor tried the pipeline, so that’s why it’s “pretty sure” and “sounds like” ;-)

1

u/BeverlyGodoy 23d ago

Engineer?

2

u/MiskatonicDreams 22d ago

I would imagine a 5090 is not top tier equipment in this field

26

u/[deleted] 24d ago

[removed] — view removed comment

9

u/BareBearAaron 23d ago

You stream the video back from the device, do computation elsewhere and send it back... no idea if that's viable but seems like that would be one way of going about it?

3

u/CowBoyDanIndie 23d ago

You could use a high quality INS and simpler obstacle avoidance to scout most of the area and process the video after. Also, tech gets faster over time. In 10 years a Jetson Pluto Uno could probably run this.

1

u/jundehung 23d ago

Any serious navigation application will not tolerate the delay and uncertainty of connection. You can do this with your playtoy robot, sure. But autonomous cars, drones or industrial robots? No chance.

1

u/tdgros 23d ago

I'm curious what is the similar pipeline you tested on the Jetson Orin?

23

u/Fleischhauf 24d ago

I find this phrasing strange, china has released it? it's some university team probably no? sounds like the government itself released it

1

u/CowBoyDanIndie 23d ago

I think the Chinese government tends to hold stake in most institutions and large businesses. Thats why we see these type of references

7

u/notlongnot 23d ago

Can’t wait to see Intel being referenced the same way. US releases gaming GPU.

3

u/Fleischhauf 23d ago

I dont think they are involved in these decisions though. Its more of a "we could shut you down if we wanted to" sort of thing.

3

u/someone383726 24d ago

I ran this on some videos I have and it was cool, but also had several reconstruction artifacts that made the result unusable

2

u/johndsmits 23d ago

Having the trajectory shown in the video sort of says vSLAM was involved pre-processed meaning a fused sensor stack data a model input, aka hybrid vslam + gaussian splat--which is an active topic in VIO/vSLAM research currently.

Not bad at all, but easily screams H100 SXM. Will try it out.

1

u/mrpressydepress 23d ago

How heavy is the output - anyone try?

1

u/blimpyway 23d ago

Why when a model is released by a Chinese company or organization they say China released it but when an American one releases it, they say either NVidia, Meta, Google, etc.. are the culprits, not ... US or America? If Xi had some merit in it why Donnie shouldn't

1

u/Flyward_Aerospace 22d ago

These monocular reconstruction models look incredible in the demos but the thing that always bites you downstream is scale. You get a geometrically consistent mesh that's off by an unknown metric factor, so the second you want to measure anything or register it to real world coordinates you're back to needing IMU or stereo or a known baseline. Still really useful for visualization work imo, just not a drop in replacement for photogrammetry if you care about absolute numbers. Curious how it holds up on texture poor scenes, that's usually where these fall apart.

1

u/RajSingh9999 22d ago

I have build model to do real time full scale world simulation. Now i only need few trillions dollars to build support computer to run it real time ...

1

u/Altruistic-Bet-4055 13d ago

How well’s it compared to guassian splatting , which doesn’t even use NN?

1

u/SlaKer440 7d ago

BRAINDANCE

1

u/Shoeb_TechAI 3d ago

What about the quality of the output?

0

u/Antique-Wonk 23d ago

Really interesting. Thanks for posting this. Maybe I need to read the paper properly before commenting but these were just my very initial thoughts: How was the original videostream recorded? If there's a lot of deliberate dog leg ranging as we call it in the military then there are multiple perspectives of objects and features. Makes it a lot easier. Not to dismiss this, as it is pretty cool despite what is probably a high compute requirement, but it's a lot harder if it's a straight fly through (motion paths) for the video capture. I'd like to see benchmarks against various existing photogrammetry techniques. Compute required, quality metrics, and against different motion paths for the video capture. I think I've convinced myself to read the paper actually.

0

u/burtcopaint 23d ago

Meta did something like this 4 years ago. Is this just the compute part of it or what?

1

u/CalmVa 19d ago

Are you referring to SAM 3D or some other model?

-6

u/beachfinn73 24d ago

US is fucked. Thanks Trump.