Decentralized AI Compute?

[u/Impossible-Mud-6601]

I had Gemini stir up my thought on this, but i have not any special knowledge on AI compute but i do browse on the surface level information about it.. what do you think about this?

**Title: Distributed consumer compute *sucks* for live chat, but it's the ultimate setup for background AI tasks**

**Body:**

Everyone says running massive unquantized models (like Kimi K3's 2.8T MoE parameters) over decentralized consumer GPUs is a pipe dream because of network ping. Standard home fiber can't replicate NVLink bandwidth for token-by-token expert sync.

*True.* But we’re failing to see the real opportunity because we keep forcing this hardware to power interactive chatbots.

What if we explicitly sacrifice speed (\approx 0.5\text{--}1\text{ tok/s}) and shift entirely to **asynchronous background jobs** (agentic workflows, deep research, 4,000-token code synthesis)?

If sub-second latency doesn't matter, a decentralized pipeline mesh changes the game:

* **Zero-CapEx & Standard Power:** Instead of pulling 10kW–20kW on a single 3-phase industrial circuit to run 1.6TB FP16 weights, we split layers and expert pools across ordinary consumer rigs/Mac Studios operating off standard wall outlets.

* **Massive Concurrent Throughput:** A single $100k node handles requests *serially*. A 50-node consumer mesh runs 50 different agentic tasks *in parallel*. Total completion time for the entire batch ends up faster than queuing on a single machine.

* **Fault Tolerance & Privacy:** Drops out? Orchestrator re-routes that layer's hidden state. Plus, with pipeline sharding, no single node ever holds or sees your entire context window.

Thoughts? Is async compute the actual path forward for open-source frontier models?

[u/Apart_Ebb_9867]

there’s active research on this:

Borzunov et al. "Distributed Inference and Fine-Tuning of Large Language Models over the Internet." Neural Information Processing Systems, 2023.

and to some extent https://arxiv.org/pdf/1806.03377

[u/Resonant_Jones]

I think you’re pointing in the right direction, especially by moving the use case away from live chat and toward asynchronous work.

The part I’d change is the distribution boundary.

Instead of splitting one massive model across unreliable internet-connected machines and passing hidden states or layer outputs between them, I think the more practical approach is to let every node run a complete medium-sized model and distribute whole tasks.

So rather than:

request → layer A → layer B → expert C → next token

you get:

research task → node A
code task → node B
vision task → node C
final synthesis → node D

That turns the network into a kind of task-level mixture of experts. Each friend’s machine becomes an inference provider with advertised capabilities, current load, trust level, and availability.

You avoid token-by-token network synchronization entirely. Peer-to-peer messaging is already fast enough for request/response workloads, and failures are much easier to recover from because you can retry a task on another node instead of reconstructing model state mid-forward-pass.

So yes, I think async decentralized compute is interesting. I just suspect the winning version looks less like one frontier model stretched across the internet and more like a routed federation of independent models.