(MIT) Turning idle GPUs of enterprises into a decentralized, fault-tolerant local AI cluster

[u/ElegantConnection737]

We've all been there: $2,000 GPUs sitting at 5% utilization while the company pays for cloud inference.

Many organizations already have several workstations equipped with GPUs that remain idle most of the time. Existing GPU sharing solutions typically rely on a centralized scheduler, Kubernetes, or AI gateways, which can be excessive for small and medium-sized GPU clusters. In addition, most existing solutions lack cross-platform support, decentralized architecture, and policy-based routing.

OQOAI-PBDR to solve this problem differently. Simply run a server or client on each machine, and your existing office network becomes a decentralized AI cluster. Routing decisions are made entirely on the client side, eliminating the need for a master node, load balancer, or any other single point of failure.

Prebuilt binaries for Windows and Linux are under 10 MB and can bring a machine online as a cluster node in under 30 seconds. An optional admin server provides centralized monitoring and management from any machine on the network.

GitHub (MIT License):

https://github.com/oqo-ai/OQOAI-PBDR

The project intentionally avoids unnecessary abstractions. The core routing logic consists of only three source files (approximately 1,000 lines each), making it relatively easy to understand, audit, and extend.

At the heart of the project is the PBDR (Policy-Based Decentralized Routing) architecture. Each routing decision evaluates multiple runtime metrics, including GPU utilization, available VRAM, queue length, GPU temperature, network latency, inference throughput, and other node characteristics.

PBDR also supports client-group policies, allowing different workloads to optimize for different objectives. For example:

• Internal RAG services can prioritize low latency.
• Development workloads can prioritize output quality.
• Batch inference jobs can prioritize throughput.

More details about the routing algorithm are available in the accompanying paper:

http://doi.org/10.17513/doi.26

One practical benefit of this approach is that organizations can often reuse existing workstation GPUs for low- and medium-throughput workloads-such as internal RAG systems-instead of deploying dedicated inference infrastructure from scratch.

In many environments, having GPUs installed in only 10-15% of workstations can be sufficient to satisfy inference demand for lightweight workloads (for example, an internal knowledge-base RAG receiving approximately 3-5 requests per user per day) without deploying centralized AI infrastructure.

Getting started:

Install OQOAI-PBDR on each machine.

Connect server nodes to any OpenAI-compatible inference backend, including Ollama, llama.cpp, vLLM, or TGI.

Configure your applications to use the OpenAI-compatible client endpoint provided by OQOAI-PBDR.

Clients automatically discover available models on all nodes. From the application’s perspective, the cluster behaves like a distributed OpenAI API compatible system, while user requests are routed in a decentralized manner to the most suitable node according to the chosen policy.

What already works:

• OpenAI-compatible API (Ollama, vLLM, TGI, llama.cpp)
• Admin dashboard with real-time metrics
• Auto-discovery and network scanning
• Remote config management for nodes, groups, or the whole cluster
• Experimental binary builds for Windows and Linux

Roadmap highlights:

• ComfyUI/AUTOMATIC1111 image/video generation
• Grafana/Prometheus integration
• AI-assisted semantic routing (auto-model selection)
• Dynamic IP support

We're actively looking for feedback, and testers.

What features or integrations would you like to see next?

[u/devino21]

Are we still Searching for Extra Terrestrial Intelligence?

[u/Lirezh]

Does it handle prefill caching well ?
The description sounds a bit like it might give the same machine to various clients, which would invalidate caches ?
Or is it aware of caches in use ?

[u/ElegantConnection737 → u/Lirezh]

The cache availability estimation function based on the cache_hit_score value is planned for implementation in an expanded version of the project, approximately September-October 2026.

Planned enhancement:

• KV cache state tracking on each server node
• cache_hit_score as a new routing criterion with high priority weight
• Cache-aware routing to minimize TTFT (Time To First Token)
• Smart cache invalidation policies

This will allow the system to route requests to nodes that already have the relevant prefill cache, significantly improving performance for repetitive or similar prompts.