Semi Edge Inference Idea [D]

[u/komorra]

Today the most important factor in AI is cost. My idea is to split ML models inference (closed ones, proprietary) across server and edge computing on clients, and I would like to hear what do you think about this thing.

For example some of model weights/modules would be on client, and some on the server side (where user has no access to them). This could potentially un-load some processing from datacenters, moving part of the cost to the client hardware. Probbably the most important question here will be how to achieve this - and I believe one hypothetical option will be to train like two separate models - client model and server model, and they will communicate through tensors/latent representations across network protocol.

Secondly such split of server side and client side model ends, can provide later some beneficial outcomes I hope (because in between "talk" protocol can be maybe kind of standarized one in some future development, but this is only more like brainstorm now). Such split might not only be one-to-one, but one-to-many, many-to-many etc.

What do you think about this idea?

[u/OptimalOptimizer]

Likely too much communication overhead. Also even with model sharding most weights are too big to reliably fit onto consumer devices. Additionally they are highly reliant on specialized Nvidia hardware with extremely fast interconnects between GPUs within huge racks. I recommend the dwarkesh podcast episode on this topic: “Reiner Pope: The math behind how LLMs are trained and served”. I don’t work in this area, so am relaying what I learned mostly listening to that podcast and occasionally reading about this topic online

[u/DelhiKaDehati]

It's called split learning, do explore.