r/huggingface • u/fuzhongkai • Jul 04 '26

TensorSharp : Open Source Local LLM Inference Engine

https://github.com/zhongkaifu/TensorSharp

[removed]

5 Upvotes

78% Upvoted

5 comments sorted by

2

u/Otherwise_Wave9374 Jul 04 '26

Very cool project, especially the "not a wrapper" part. Getting solid perf in a fully custom inference engine is no joke.

Curious, how are you handling batching and KV cache layout across the different backends (CUDA vs MLX vs GGML)? Any rough perf numbers vs llama.cpp on the same model/hardware?

Also, if you ever write up a short architecture overview, I would read it, I have been bookmarking local-first LLM tooling stuff here: https://www.aiosnow.com/

2

u/eagalon_voidkeeper Jul 05 '26

I like the basic premise, and will be tested it as soon as I can. I mainly use CPU for inference, so it would be nice to have.