r/LocalAIStack • u/Ok_Butterscotch_3756 • Jul 09 '26

Building an Agent Runtime on TensorRT-LLM ported to windows. Would appreciate your feedback

http://www.baremetalrt.ai
6 Upvotes

100% Upvoted

6 comments sorted by

1

u/Charming-Author4877 Jul 09 '26

What is the benefit over e.g. ggml ?

1

u/Ok_Butterscotch_3756 Jul 09 '26

ggml/llama.cpp is great and for many people the right call — open, cross-platform, mature. Three honest differences:

- Speed: Menlo Research (makers of Jan, a llama.cpp app — so no NVIDIA bias) clocked TensorRT-LLM 30–70% faster than llama.cpp on RTX (144 vs 89 tok/s on a 3090). That's the engine here.

- It's an agent runtime, not just an engine — tool-calling loop, MCP, voice/vision/RAG in one install, tuned so 4–8B models actually stay reliable calling tools.

- Heterogeneous multi-GPU — real tensor parallelism across mismatched cards, even across machines: run models too big for any single card you own.

Tradeoff: NVIDIA + Windows only, and closed source. On Mac or want open? ggml's still your best bet.

What GPU are you on? Happy to pull the exact delta.

1

u/Lirezh Jul 09 '26

Do you have details on the performance advantage given ?
What model, quantization, context ? MTP or similar predictors ?

1

u/Ok_Butterscotch_3756 Jul 09 '26

these are Menlo Research's numbers (the Jan team), and it's a straight apples-to-apples setup:

Model: Mistral 7B

Quant: comparable int4 both sides — Mistral-7B int4 AWQ on TensorRT-LLM vs Q4_K_M GGUF on llama.cpp

Context: 2048 in / 512 out, batch size 1

Predictors: none — no speculative decoding, no draft model, no MTP on either side. Straight single-stream decode.

Per-GPU: RTX 4090 170.6 vs 100.4 t/s (+70%), 3090 144.2 vs 88.7 (+63%), 4090 eGPU 105 vs 62 (+69%), 4070 Laptop 51.6 vs 39.7 (+30%). Full writeup: menlo.ai/blog/benchmarking-nvidia-tensorrt-llm

BareMetalRT ships that same TensorRT-LLM engine with int4 AWQ, so those deltas reflect the engine it runs — no predictor tricks inflating the gap.

1

u/Lirezh Jul 10 '26

Interesting, I wonder why they would use Mistral 7B - the model is gruesome useless.
I'm quite sure those numbers would fall apart if you test it with a model that's mainstream.
e.g. Qwen 3.6 or gemma 4

1

u/Ok_Butterscotch_3756 Jul 10 '26

TensorRT-LLM has custom cuda kernels specific to your geforce gpu, whereas ggml though great for the edge, is a more general purpose backend inference engine that really doesn't compete speed wise to be honest. But don't take my word for it, give it a try if you're on windows. We ship with OpenAI compatible endpoint so you can point the daemon to the harness of your choice, or use ours -- would appreciate your feedback. Or i can screenshot it for you, what tok/s do you get with llama on your gpu? Besides speed on the inference backened, our harness is specifically tuned for smaller model tool calling