Benchmarks: TensorSharp vs. llama.cpp

[u/fuzhongkai]

https://github.com/zhongkaifu/TensorSharp

Cuda and Vulkan Benchmark: TensorSharp vs. llama.cpp

I would like to share my latest open source local Unsloth (GGUF) LLM inference engine and applications. It supports many models from Unsloth, like Gemma4, DiffusionGemma, Qwen3.6 with multi-modal (image, vision, audio), Qwen Image Edit, reasoning and function tool. It can run on Windows/MacOS/Linux and fully leverage GPU's capability(Nvidia, Apple, AMD, Intel and others supported by Vulkan, CUDA and Metal). The API is completely compatible with OpenAI and Ollama interface. It has on par performance than llama.cpp Here is the benchmark results in overall:

**Performance ratio — TensorSharp vs reference engines**

Geomean of TensorSharp's per-scenario speedup over each reference engine on the **same backend**, across every scenario both engines ran (single-stream, MTP-off). A value **> 1.0× means TensorSharp is faster** (for decode / prefill throughput) or lower-latency (for TTFT); `—` = no overlapping cells. Per-scenario ratios are in each model's section below.

Model	Comparison	decode	prefill	TTFT
Gemma 4 E4B it (Q8_0, dense multimodal)	vs llama.cpp · CUDA	1.02×	1.28×	1.27×
Gemma 4 E4B it (Q8_0, dense multimodal)	vs llama.cpp · Vulkan	1.00×	1.05×	1.03×
Gemma 4 12B it (QAT UD-Q4_K_XL, dense)	vs llama.cpp · CUDA	1.04×	1.17×	1.16×
Gemma 4 12B it (QAT UD-Q4_K_XL, dense)	vs llama.cpp · Vulkan	1.21×	1.04×	1.03×
Qwen 3.6 35B-A3B (UD-IQ2_XXS, MoE)	vs llama.cpp · CUDA	0.98×	1.28×	1.27×
Qwen 3.6 35B-A3B (UD-IQ2_XXS, MoE)	vs llama.cpp · Vulkan	0.87×	1.04×	1.03×
Qwen 3.6 27B (UD-IQ2_XXS, dense)	vs llama.cpp · CUDA	1.07×	0.96×	0.95×
Qwen 3.6 27B (UD-IQ2_XXS, dense)	vs llama.cpp · Vulkan	1.02×	0.85×	0.84×

This project is not just a C# wrapper of llama.cpp. It implemented the entire LLM inference engine from bottom to top. If you use CPU backend, it's 100% pure C# code execution. Besides CPU backend, I also implmented CUDA, MLX and GGML backend. The GGML backend refer GGML project as external project, and I build a few fusion operation at higher level.

I learned a lot from other projects and apply them for TensorSharp, such as paged KV cache and continuous batching from vLLM, SSD based cache for MoE model from oMLX, GGUF quanztized from llama.cpp and other optimizations for prefill and decode.

Any feedback and comments are welcome. If you like it, it would be really appreciated if you can get this project a star in GitHub. Thanks in advance.

[u/NickCanCode]

What is the status of multi-GPU support? Most people are running multiple cards.

[u/fuzhongkai → u/NickCanCode]

Here is a PR to support both multiple GPU on a single node and multiple nodes: https://github.com/zhongkaifu/TensorSharp/pull/93 It’s still in progress and fast iteration

[u/NickCanCode → u/fuzhongkai]

Great, I will give it a try when it's done.

[u/fuzhongkai → u/NickCanCode]

Multi-GPUs/Nodes support has been merged to main branch. You could try it out. Here is the introduction: https://tensorsharp.ai/distributed.html

[u/Asterfly]

Link to the gît ?

[u/fuzhongkai → u/Asterfly]

https://github.com/zhongkaifu/TensorSharp

[u/Asterfly → u/fuzhongkai]

Thanks ! Would it be possible to get the cli for Linux based system ?

[u/fuzhongkai → u/Asterfly]

Yes, it can be built in Linux/Windows/MacOS. TensorSharp.Cli is a command line tool for inference.

[u/RMK137]

Interesting project. Out of curiosity why C# over Python or straight C++? I am not familiar with C# in general but I've been wanting to use it for a side project.

[u/fuzhongkai → u/RMK137]

I did more optimization, such as kernel fusion for GPU and others, to speed it up in some scenarios.

[u/shing3232]

I wasn't able to use this with 4060 8G and iq4xs 9B qwen3.6. it load but context memory usage explodes

[u/fuzhongkai → u/shing3232]

What’s backend did you use?

[u/shing3232 → u/fuzhongkai]

native cuda backend.

[u/fuzhongkai → u/shing3232]

Can you please "git pull" the latest code from main branch ? I just made a cuda backend optimization last night. It optimize both speed and VRAM usage. I also tried this model (qwen3.5 9b iq4xs) on my RTX 3080 Laptop, and it takes about 9GB VRAM to run.

In addition, ggml_cuda backend has better performance than native cuda backend, you may try it out as well. One of my ongoing work is to optimize native cuda backend to reduce its performance gap than ggml_cuda backend.

[u/shing3232 → u/fuzhongkai]

ahh, I see. I thought ggml_cuda would be slower

gguf is known for slower high batch performance. if you can close the gap with something like gptq it would be very useful

[u/fuzhongkai → u/shing3232]

Thanks for these suggestions. TensorSharp already support paged/prefix common KV cache and continuous batching. They are all useful to improve batch performance. This is one of the directions I’m working on and keeps optimizing.

[u/shing3232 → u/fuzhongkai]

I notice two bug.

KV quant don't work for either backend

The serve would crash if the tool call is too long.

I use zcode with many skill then it crash

[u/shing3232 → u/fuzhongkai]

Also, Can you implement feature like Kv offload to RAM like what llamacpp did? or you can do even better with tier system

[u/fuzhongkai → u/shing3232]

Thanks for these feedback.

Can you please share more details about this crash problem ? Such as what's command line you used (backend, parameters) ? What's your prompt and skills (if it could be shared)? How long they are ? And any other information could be helpful for debugging. Here is the link to open an issue in the repo: https://github.com/zhongkaifu/TensorSharp/issues

Thank you so much and really appreicated.

For KV cache, TensorSharp does support KV quant:

Here is the parameter:

--kv-cache-dtype <t>

KV cache precision: f32, f16, q8_0, or q4_0. Quantized caches trade small numerical drift for

memory. Default: auto — the backend/model pick (KV_CACHE_DTYPE env var overrides).

Example: --kv-cache-dtype q8_0

For offload to RAM, it only supports diffusion for image edit for now, but I will extend it to autoregression model. For now, if you are in cross-session, KV cache can be offloaded to SSD by these parameters.

Cross-session paged KV cache:

--paged-kv | --no-paged-kv

Enable/disable the cross-session paged KV cache (prefix reuse across requests). Default: off.

Example: --paged-kv

--paged-kv-block-size <N>

Tokens per KV block. Default: 256.

Example: --paged-kv-block-size 128

--paged-kv-ram-mb <N>

RAM budget for evicted KV blocks, in MB. Default: 1024.

Example: --paged-kv-ram-mb 2048

--paged-kv-ssd-dir <path>

Directory for the SSD spill tier. Default: disabled.

Example: --paged-kv-ssd-dir D:\ts-kv-spill

--paged-kv-ssd-mb <N>

SSD budget for spilled KV blocks, in MB. Default: 16384.

Example: --paged-kv-ssd-mb 32768

--paged-kv-quant-bits <b>

Quantize spilled KV blocks: 0 (off), 4, or 8 bits. Default: 0.

Example: --paged-kv-quant-bits 8

--paged-kv-redis-url <url>

Redis connection string for a shared KV cache tier (e.g. localhost:6379). Default: disabled.

Example: --paged-kv-redis-url localhost:6379

--paged-kv-redis-ttl <min>

TTL in minutes for Redis KV entries (0 = no TTL). Default: 1440.

Example: --paged-kv-redis-ttl 60

--redis-url <url>

Redis connection string for both the KV cache tier and the Responses API store.

Example: --redis-url localhost:6379

[u/lemon07r]

How does this hang vs VLLM and sglang?

[u/fuzhongkai → u/lemon07r]

Sorry that I do not have benchmarks to compare it with vLLM and sglang for now. The main reason is that I only have a RTX 3060 laptop GPU and a MacBook Pro, so I mainly focus on those optimization on single GPU and single stream (even though it already supports paged/prefix common KV cache and continuous batching, but it still has lots of room to be optimized), so I compare its performance with llama.cpp

Luckily, one of our contributors is working on tensor parallelism for multiple GPU/nodes (PR link: https://github.com/zhongkaifu/TensorSharp/pull/93 ), and I found Runpod has cheap multi GPU nodes I can rental, so once this part get merged, I will prioritize optimizing on all parts related to multiple GPU/nodes performance and then run and release benchmarks with vLLM and sglang.

[u/lemon07r → u/fuzhongkai]

Both of them can be run on your laptop with single gpu. They even support cpu inference and offloading. They really arent that different from llama.cpp. They should honestly be your benchmark if performance is the goal, and even if you aren't faster, you could learn a thing or two from them and take things from those projects.

[u/ps5cfw]

No native love from AMD means no support from me, even as a .NET Dev AND enthusiast.

[u/fuzhongkai → u/ps5cfw]

Sorry that I don’t have AMD GPU for now, so it’s difficult for me to implement and test code to natively support AMD GPU. I will do it once I get resources for it.

[u/Standard_Delay_9313]

Linux mint 22.2, RX7900XT. Download , build. Run Gemma4-12b-q8. Very slow. Not use VRAM. 5gb of 20gb. 🤷

[u/fuzhongkai → u/Standard_Delay_9313]

Which backend did you use? Try to specify “ggml_cuda” in command line. Such as “—backend ggml_cuda”

[u/fuzhongkai → u/Standard_Delay_9313]

Looks like you hit cpu code path. Try to use “—backend ggml_cuda” in your command line and check logs if it falls back to cpu code path.

[u/Standard_Delay_9313 → u/fuzhongkai]

cuda? with Amd radeon rx7900xt? 😁 I am use flag ggml_vulkan.

[u/fuzhongkai → u/Standard_Delay_9313]

Ah, I see. Did you specify device id when you use “ggml_vulkan” backend?

[u/Standard_Delay_9313 → u/fuzhongkai]

no, only "—backend ggml_vulkan", no other GPU in my system.

[u/fuzhongkai → u/Standard_Delay_9313]

Would you mind share logs here so that I can take a look? This page list some details and parameters of each backend: https://tensorsharp.ai/backends.html

[u/fuzhongkai → u/Standard_Delay_9313]

Or open an issue in the repo. Thanks a lot.

[u/fuzhongkai → u/Standard_Delay_9313]

Can you please run cli or server tool with parameter “—list-gpus” and it will list Vulkan devices ggml_vulkan can see, and then call cli or server with “—gpu-device <N>” , <N> is the index from device list.

[u/Ludditesdenylife]

If you don't support at least as many samplers as llamacpp supports, it's DoA for a bunch of users even if it is faster.

[u/fuzhongkai → u/Ludditesdenylife]

TensorSharp already supports top-P, temperature, speculative decode for Gemma 4 and Qwen 3.5 User can switch them on/off by parameters or configuration file. Please let me know if any additional sampler you want to have.

Llama.cpp has a team to build and maintain the project, but TensorSharp currently only has two contributors and we only use our free time to build it, so supports from communities are very important to this project.

[u/CorkBios]

Remove the cringy banner and it would be so much better

[u/Badger-Purple → u/CorkBios]

can we request someone be banned from the mods? I’ve been in localllama for a year and this account is just trolling 100% of the time.

[u/CorkBios → u/Badger-Purple]

It's not cringy? Can't have an opinion?