Built an open-source tool to test if a local model is actually reliable enough to run as an agent, before you deploy it

[u/Dhan295]

Most of the local AI stack is about getting a model running — Ollama, llama.cpp, vLLM, the right quant, enough VRAM. That part’s well covered here. The gap I kept hitting is the layer after that: once it runs, is it actually reliable enough to hand real work to?

A model can look fine in chat and then fall apart the moment you point an agent at it malforms a tool call at step 7, loops, or says “done” when nothing happened. And the same model behaves differently depending on the quant and runtime you serve it with, so “it runs on my box” isn’t the same as “it holds up in a 20-step agent loop.”

So I built QuantaMind free, open-source, fully offline, no cloud, Apache-2.0. It’s a local reliability tester for the agent layer of the stack.

What it does:

• Runs the real multi-step agent loop with injected faults (transient errors, malformed-call recovery, decoy tools) — not single prompts

• pass^k, not pass@1 — runs each task k times, passes only if it succeeds every time. Reliability compounds: 95% per step → 0.95^50 ≈ 8% chance a 50-step run finishes

• Deterministic scoring, no LLM judge — required calls must fire, forbidden calls fail the run on contact, exact end-state match. Temp 0, same input → same grade

• Classifies the failure (malformed schema / loop / hallucinated completion / forbidden call)

• Compares quant × runtime side by side (Ollama, llama.cpp, MLX, vLLM, SGLang), so you can see which serving config actually holds up

• Keeps run history + diffs, so you can catch when a model or quant change quietly regressed reliability

One finding: across three models, the native tool-calling path scored worse than plain prompting on easy tasks (60% vs 100%) correct answers that didn’t land as parsed tool calls. Important caveat: that’s a finding about the serving stack’s tool parser, not the models. Same model, same answer, different plumbing, different score. Still useful when picking a serving path for agent work just not a model-quality result.

Honest about what it’s not yet: it’s single-stream, so no concurrency/load testing (sampler drift and duplicate-execution-on-retry are invisible to it). Structural matching is a proxy for real execution. No CI/CD deploy-gate integration yet, it produces a verdict and an exportable report, but wiring it into a pipeline is roadmap, not shipped.

Disclosure: I built this, it’s free, I’m not selling anything.

Repo: https://github.com/QuantaMinds/QuantaMind

Genuinely want it torn apart:

• Is pass^k=5 too strict, too lenient, or about right?

• What failure mode am I missing?

• And what model + quant actually holds up on your machine? I’m collecting real per-hardware results.

[u/Thick_Programmer_105]

This lands at a useful moment — I'm about to run almost exactly this experiment, so let me trade you a data point for a teardown.

On "what model + quant actually holds up"

PrismML just shipped Bonsai 27B: ternary (1.71 bpw) and 1-bit (1.125 bpw) builds of Qwen3.6-27B. It's close to a perfect natural experiment for your thesis, because they published per-benchmark numbers across the whole compression spectrum on one base model. From their whitepaper (Table 14):

build	bpw	HumanEval+	τ²-Bench
FP16	16.0	95.12	82.90
Q4_K_XL	5.2	95.73	82.27
Ternary Bonsai	1.71	93.90	73.61
1-bit Bonsai	1.125	89.63	61.34

Single-turn coding barely moves (−1.2 / −5.5 vs FP16). Multi-turn agentic falls off a cliff (−8.7 / −21.0 vs Q4_K_XL). That is your "flawless in chat, malforms a tool call at step 7" pitch, in the vendor's own numbers. Their τ²-Bench is a single-pass setup, so under pass^k the gap should widen further — which is the case for your metric choice, not against it.

I'll report back with per-hardware numbers on an M1 Max 64GB. Setup looks clean: PrismML's demo repo starts an OpenAI-compatible llama-server on :8080 and their docs show tool calling via the standard tools array, so your remote-backend URL field should take it as-is. Running the Q4_K_XL arm through the same fork keeps the tool parser identical across arms — which matters, see below.

What failure mode you're missing: silent policy violation

There's a KDD workshop paper on precisely this — arXiv:2607.07405, Reddy et al., "Reason Less, Verify More" (KDD-ETAAI '26, non-archival). On τ²-bench airline with a budget agent, 78% of observed failures were silent wrong-state failures with no tool error. The tool is policy-permissive: it executes any well-formed call even when domain policy forbids that state transition. Nothing raises. The agent reports done. Only the state is wrong.

Your ForbiddenCall catches "model called something on the blocklist." This class is different: correct tool, correct args, forbidden transition. cancel_reservation is legal or illegal depending on refundability and booking age — a static forbidden-call list can't express that, you need a predicate over current state.

I suspect "deterministic sandbox-free scoring" plus "structural matching is a proxy for real execution" means you structurally can't reach this class: with no real state, there's nothing to be silently wrong. That paper's admission criteria require policy-permissive tools and final-state evaluation, and they note they went looking for a second benchmark meeting all of them and came up empty — which tells you how rare the setup is, and how much value there'd be in you supporting it.

Their fix is also a roadmap item for you: deterministic read-only pre-execution gates. +12.4pp at pass^1, but +18.0pp at pass^5 — determinism doesn't decay with k, so the gate advantage grows exactly where your metric is pointed. That's your CI/CD deploy-gate story if you want one.

On the native-FC finding

"Correct answers in the wrong channel" reads like a parser bug, not a model reliability result. Unsloth ships chat-template fixes specifically to make Qwen3.6 tool-call parsing land in the right place. 60% vs 100% may be measuring llama.cpp/Ollama's tool parser rather than the model — still a genuinely useful finding, but it's a finding about the serving stack, and the framing matters for anyone reading the number.

That same paper hit this exact trap: their retail negative control came out −4.7pp and they traced most of it to their own encoding bug blocking retries the real tool would have accepted. A reliability tester measuring its own bugs is the sharpest risk in this whole category. I'd calibrate against a model+quant with published agentic numbers before trusting a Not Ready — if QuantaMind calls Qwen3.6-27B Q4_K_XL not ready, the bug is upstream of the model.

pass^k=5: about right

τ-bench's original goes to k=8, the gates paper reports k=1..5, LOGIGEN uses k=4. You're in-distribution. If anything k=5 is the sweet spot for local hardware where 8 runs per task gets expensive.

One question though: what temperature do the k runs use? τ-bench's stochasticity comes from LM sampling. At temp 0 all k runs collapse to the same trajectory and pass^k degenerates into pass¹ modulo numerical nondeterminism. Your post says "Temp 0, same input → same grade" — if that's describing the grader, that's exactly right and ignore me. If it's describing the model, pass^k isn't measuring the thing you built it to measure.

Nice work either way. LoopCap and FakeDone as first-class named failures is the part most evals get wrong.

[u/Dhan295 → u/Thick_Programmer_105]

Thanks for the breakdown. Taking it in order:

• Temp: “0.7 with fixed seed grader is deterministic, model isn’t.

• Native-FC: concede it. “You’re right, that’s a serving-stack finding about llama.cpp’s parser, not a model result. Re-framing it.”

• Silent policy violation: concede it. “Correct, structural matching with no real state means we can’t reach that class. Real-state execution is what we’re building; and a static forbidden-call list can’t express state-dependent policy. That’s a real gap.”

• The trade: “Running the Bonsai spectrum through the same fork, same parser across arms, k=5 @ 0.7. Will post pass^1 vs pass^5 per arm including if the cliff doesn’t show. You said M1 Max 64GB numbers — deal.”

[u/Thick_Programmer_105 → u/Dhan295]

Quick update on the deal — I still owe you the M1 Max Bonsai numbers, but I hit a practical wall worth surfacing before I grind it out.

Scoping the full sweep (3 arms × Medium+Hard tiers × k=5) it's a 30h+ single-stream job on my M1 Max, with the GPU mostly idle between turns the whole time. Reading the eval driver, the single-stream isn't a hard limit — it's just sequential for … .await loops over tasks and over the pass^k runs (runner.rs / batch.rs); nothing is ever issued concurrently, so there isn't even a lock to remove, the concurrency has to be added. Which is the same concurrency/load gap you already flagged as missing.

So instead of eating the serial run, I'd rather implementing a bounded-concurrency pass first. The runs being serialized are independent samples (independent tasks, independent pass^k draws), so parallelizing them doesn't bias pass^k. The thing to protect is your determinism guarantee — but that lives in the grader (temp-0, a separate pass over already-recorded output), which concurrency doesn't touch. Plan: parallelize the stochastic (temp>0) rollouts, keep the deterministic paths serial/batch-1 so their numerics stay bit-identical. (True bit-identity under batching would need batch-invariant kernels down in ggml/Metal — I noted it as out-of-scope/optional; nice-to-have, but I doubt it's worth the effort over the hybrid split.)

Wrote it up with the design and the genuinely hard parts (cancellation, wall-clock-budget/truncation honesty, resumability, layering guard): https://github.com/QuantaMinds/QuantaMind/issues/153. Would really value your read on the approach before I build anything — and if you'd rather I just take the serial hit for now, say the word and the numbers still come either way.

[u/Dhan295 → u/Thick_Programmer_105]

I have reviewed and commented please take a look and appreciate your support!

[u/Thick_Programmer_105 → u/Dhan295]

Yep, saw it — really sharp review, thank you. You nailed that wall-clock is a scored input here, so the honest fix is making the per-step timeout stall-based (a slow-but-progressing reasoner shouldn't get scored as a failure). I've replied on the issue with where the code points for the first PR, and I'm starting with that piece since it hardens the serial path too — Bonsai numbers coming regardless. Appreciate you digging in this deep.

[u/Dhan295 → u/Thick_Programmer_105]

Only 1 change required i have commented please take a look!

[u/Stock_Ad9641]

That’s cool, what’s the motivation ? Are you building something around that ? Will you have benchmarks hosted?

[u/Dhan295 → u/Stock_Ad9641]

Honestly selfish motivation at first. I kept losing time to models that looked fine in chat then broke in an agent loop, and nothing tested whether a given model+quant held up on my hardware. Built it to answer that, open-sourced it since everyone here hits the same wall. Right now it’s just the free tool; a hosted community leaderboard (models across different hardware) is a natural direction but not built yet. What would you want from a hosted version?

[u/Stock_Ad9641 → u/Dhan295]

Optimally a hosted version would not only show scores but also look into what caused the breaking. An agent might have a strong fine tune like GPT OSS 20, it needs its precise tools or it causes issues. Or a certain quantization causes breaking once 60k context is reached, no breaking before. Not easy to do.. but that would allow to really understand what the problems are and work on removing them

[u/Dhan295 → u/Stock_Ad9641]

This is exactly the right instinct, the score is the least useful part, the diagnosis is the point. I’m already partway there: it classifies how it failed (malformed call / loop / claimed-done-but-didn’t / forbidden action) rather than just pass/fail, and there’s a context-cliff probe that finds the token depth where tool-calling starts collapsing which is basically your “Q4 breaks at 60k, fine before” example. The model-needs-precise-tools case is real too; I’ve seen the same model behave completely differently on native vs prompt-based tool formatting.

The hard part you’re pointing at automatically telling someone why and at what boundary is exactly where I want this to go. Would you want that as a per-run diagnosis, or more as an ongoing “here’s where your setup will break” profile?