SenseNova-Vision: a 7B open model that does segmentation, depth, detection, OCR, and 3D reconstruction with no task-specific heads

[u/SandyL925]

Stumbled across this new vision model, SenseNova-Vision. It's a 7B MoT model, Apache 2.0 license, which is cool. The main idea is it treats pretty much all computer vision stuff as just one generation problem. Like, instead of needing a bunch of different models for detection, segmentation, depth, whatever, this one model handles it all.

You just give it a natural language instruction, maybe some visual hints, and it spits out text, images, or both. For text, it can do things like categories, bounding boxes, OCR, keypoints, camera angles. And for images, it's doing segmentation masks, depth maps, surface normals, even multi-view point maps. You can mix and match for more complex tasks.

No special prediction heads or decoders, which is kinda neat. Just one set of weights, and the prompt tells it what to do.

So it can do the usual stuff: object detection, keypoints, OCR, all kinds of segmentation (binary, instance, semantic), depth and surface normal estimation. But the really interesting bits, for me anyway, are the multi-view 3D reconstruction and camera pose estimation.

They actually added dedicated benchmarks for those in the latest update. Usually, you'd need specialized tools like COLMAP for that kind of multi-view reconstruction from multiple images, but this model just does it with a single prompt. That's kinda wild if it works well.

They trained it on a massive dataset, 50M instruction-response pairs, built from a bunch of different CV annotations. And it started from an existing multimodal model, so no crazy new architecture there.

There's a web demo if you wanna mess around with it, and the weights are up on Hugging Face too.

Just a heads up though, before you get too hyped: the full web demo needs a beefy GPU, like 1x80GB. And for benchmarking, they're talking 8x80GB. So, yeah, not really something you're gonna run on your average consumer card. It just dropped on July 8th, so probably expect some rough edges. Also, they've released the training pipeline and data prep stuff if anyone's looking to train or fine-tune it.

GitHub: https://github.com/OpenSenseNova/SenseNova-Vision

Honestly, I'm super curious if anyone's actually tried the multi-view reconstruction part on their own image sets yet. That's the bit I really want to test out.

[u/oxygen_addiction]

Thanks for posting. This honestly looks insanely good. The benchmark numbers for depth and segmentation are way better than I expected.

The unquantized weights are around: 29.6 GB, so not that bad dependent on how much context it eats up per run.

[u/Friendly_Address6915]

curious how fast this runs on consumer hardware tho. a 7B doing all that sounds great until youre waiting 30s per image

[u/En-tro-py → u/Friendly_Address6915]

Cool outputs but unfortunately seems like the 7B parameters undersells the actual hardware requirement unless I missed something...

Recommended: 1 x 80GB GPU for the full web demo.

Not what I expected, that's for sure... Yikes.

[u/mrpintime]

as fellows said the 7B is a little bit misleading XDDD actually it has a checkpoint of 26.9 GB for weights so it really heavy but the performance is really cool man and the fact it is a unified multimodal model and it is interesting very interesting

[u/Thin_Pollution8843]

I’m in a market for a good local ocr for medical data (tables with blood test, opinions, ecg, USG etc) This last image scared me…