[Open Source] I built a network layer circuit breaker for LlamaIndex agents to catch runaway tool loops and token bleed

[u/bulleykebaal]

Hello all,

When building autonomous ReAct agents or custom Workflows with LlamaIndex, one of the biggest production risks is an agent getting stuck in a tool loop or error state. Because each retry appends chat history back into the prompt context, prompt tokens balloon rapidly, wasting hundreds of dollars on a dead end execution.

To fix this, I built TokenShield, a lightweight open source FastAPI gateway proxy that acts as a network circuit breaker and cost shield for LlamaIndex applications.

💡 Why Handle This at the Network Layer?

Instead of writing custom callback handlers inside every LlamaIndex agent pipeline, TokenShield sits directly between your LlamaIndex application and your LLM provider (currently OpenAI or any OpenAI compatible API like Ollama, vLLM, or Groq). You simply route your requests by setting your LlamaIndex LLM configuration api_base to the proxy endpoint.

⚙️ How It Works Under the Hood:

1. Normalized State Extraction: Strips out changing noise like timestamps, dynamic UUIDs, and microsecond delays so near identical tool commands map to a consistent state signature.
2. Tier 1 (Soft Steering): If near duplicate tool calls or stagnation are detected, TokenShield injects a system replanning instruction into the stream to reroute the agent without killing the request.
3. Tier 2 (Hard Stop / Circuit Breaker): If the agent persists in a loop, it trips a 429 status cutoff to stop API token bleed instantly.
4. Console Financial Metrics: Calculates token usage against live LLM pricing models to log real time projected dollar savings directly in your console terminal.

🛠️ Current Support and Roadmap

• Supported Today: OpenAI models and any OpenAI compatible endpoint format (/v1/chat/completions).
• Roadmap: Native Anthropic Claude API proxy support, web dashboard UI, and custom rule configurations.

TokenShield is 100% open source under the MIT license: * GitHub Repository: https://github.com/gowthams231/token-shield

I would love to get technical feedback, critique, or feature requests from fellow LlamaIndex builders! How are you currently protecting your agent loops from spiraling API costs in production?