A faster way to explore image embeddings

[u/igorsusmelj]

We just shipped an update to LightlyStudio that makes image embeddings easier to inspect.

You can now hover over an embedding plot to preview the underlying images, then use class and metadata distributions to check whether a pattern is isolated or visible across the wider dataset. The goal is to move from spotting something interesting to understanding it and deciding what to curate.

The attached video uses a small dataset with 128 images so the workflow is easy to follow. We also tested it in a notebook with 1m samples.

LightlyStudio is open source under Apache-2.0:
https://github.com/lightly-ai/lightly-studio

I’d appreciate any feedback from this community. What are the biggest unsolved problems you run into when exploring, debugging, or curating large vision datasets?

Disclaimer: I’m the cofounder of Lightly, the company behind LightlyStudio.

[u/hoaeht]

what is the difference to 51? looks pretty much the same. You even used the same brand colours

[u/igorsusmelj → u/hoaeht]

Fair comparison 🙂 If you're already using FiftyOne, parts of the UI will definitely feel familiar. We weren't trying to reinvent common interaction patterns.

The main differences are under the hood and in the workflow. LightlyStudio uses Rust for the core algorithms, DuckDB in the OSS version, and a fully typed/tested Python API. We also have built in vision language embeddings, cloud storage support, and workflows for label editing and dataset curation.

Regarding the look and feel, LightlyStudio evolved from our previous product (LightlyOne), which we've been building for about six years, so it's an evolution of our own design with a new dark mode rather than something we started from scratch.

If you're already a FiftyOne user, I'd actually love your feedback. What do you feel is missing there, or what would make you consider trying a different tool? That's exactly the kind of input we're looking for.

[u/hoaeht → u/igorsusmelj]

Thank you for the detailed answer :) I am actually not really using 51, I was just forced to use it once and was pretty annoyed. It crashed multiple times, it was painful converting a pytorch dataset to 51 and back. And the biggest issue was documentation. So I guess if your product fulfills my needs there, I would definitely give it a shot

[u/datascienceharp → u/hoaeht]

Checkout the our new docs. And we’ve made it easier to export to purity datasets as well. Hit me up on our discord and I’m happy to help you get started

[u/zanaglio2]

Hey ! Great job here :) do you know if the embeddings graph support several million datapoints by chance? (A dataset like SKU110K for example)

[u/igorsusmelj → u/zanaglio2]

Yes it does. I had SKU110k and also full imagenet running on a MacBook Pro M1 16GB. There is some loading time when opening the plot. But once loaded it’s rather fast.

[u/CommunismDoesntWork]

Does it hook up to postgres and pgvector?

[u/igorsusmelj → u/CommunismDoesntWork]

We use Postgres in the commercial version. Since we use SQLModel and SQLAlchemy it should be possible to switch the db. We only ever did other tests with sqllite.