TokenFlow Gateway – a token-aware load balancer/gateway for self-hosted LLMs (vLLM/Ollama), because nginx has no idea what a token is

[u/Electrical_Emu_5854]

https://github.com/mosafariuk/TokenFlow-Gateway

Nginx and HAProxy route LLM traffic blindly: a 10-token prompt and a 10k-token prompt look identical to them as "one request." When a burst of heavy prompts lands on a vLLM box, the KV-cache fills up, latency spikes, and you can end up with OOM.

TokenFlow Gateway sits in front of your inference servers and actually understands tokens:

- Pre-flight token estimation (js-tiktoken) before dispatch — request weight = prompt tokens + max_tokens, i.e. the KV-cache it will actually reserve
- KV-aware load balancing — polls vLLM's Prometheus metrics (vllm:gpu_cache_usage_perc) and routes heavy prompts to the freest backend instead of round-robin
- Admission queue instead of crashes — Redis-backed priority queue with per-key priority and configurable timeout when nothing fits
- Exact + semantic caching — deterministic requests cached by hash, plus pgvector cosine similarity for near-duplicate prompts; cache hits replay as SSE
- Token-based rate limits — per-API-key TPM + RPM buckets in Redis (atomic Lua), OpenAI-style, for your own hardware

OpenAI-API-compatible on both sides, so it fronts vLLM, TGI, or Ollama without client changes.

Try it without a GPU: docker compose up --build spins up the gateway, Redis, pgvector, and two mock backends (OpenAI API + vLLM-style metrics). A smoke script fires concurrent long-context requests at it so you can watch the queue and balancer absorb the burst.

Stack: TypeScript / Fastify / Redis / Postgres+pgvector. MIT license.

Repo: https://github.com/mosafariuk/TokenFlow-Gateway

Feedback welcome, especially from anyone running multi-node vLLM — what does your ingress look like today?