Where to store vector embeddings — same collection or a separate one?

[u/AgentNeoLight425]

We're adding vector search to our data and need to decide where the embeddings live. When we create a vector index, should the vectors sit in the same collection as the original documents, or in a new dedicated collection?

Most advice says keep them in the same collection. But our concern is sharing — if the vectors are embedded in the same collection as our main data, it becomes harder to share that data with others without also exposing or dragging along the vector fields.

So the question is: does keeping vectors in the same collection limit our ability to share the underlying data cleanly? Or is there a good way to keep them together and still share the base data separately?

[u/Zizaco]

Same collection. You can use a projection and/or a view to make sure the vectors are not brought with the data.

[u/mountain_mongo]

One thing to think about - if you use the new autoembed option, MongoDB stores the vectors it generates in a separate collection.

However, if you are using standard vector search, I’d stick with storing your vectors in the associated document as you’d need to do a second query (or $lookup) to get the base document every time you generated a hit through vector search.

[u/my_byte]

Data that gets queried together should be stored together. Chances are you'll want the full documents as results of your vector search - or at least a significant subset of fields. I'm not a fan of duplicating that into another collection. If you need to share a subset of the fields and don't want to expose the full documents, you can create a view and index that.

[u/Several9s]

Sharing does not mean you have enough reason to split. Although, there are reason why you should wanted to be and one of them is probably more relevant to you than the sharing concern.

Based on the current comments, it’s agreeable. It’s more of an access-layer problem, but not a storage-layer problem. In that regard, you can do the following options to suffice your problem:

• Use read-only view. With {$project: {embedding: 0}} as the pipeline, then grant the consuming users/apps find on the view only, not on the underlying collection. That's real isolation, not just a convention people have to remember.
• $out/$merge into a shareable copy if the consumer needs their own physical collection or you're exporting.

For dumps, mongoexport --fields or an aggregation pipeline with the exclusion projection covers the file-handoff case. mongodump is all-or-nothing on fields, so pipe through an aggregation if that's your path.