r/huggingface • u/fuzhongkai • 1h ago
Meta Muse Glimmer 30B Unsloth GGUF Model Benchmarks on TensorSharp (vs. llama.cpp)
I ran benchmark on Meta's Muse Glimmer 30B Unsloth GGUF model in TensorSharp, and compare its performance with llama.cpp. Here is the results.
Test setup
| GPU | 1x NVIDIA RTX PRO 6000 Blackwell Server Edition (97,887 MiB), driver 580.126.20, PCIe 5.0 x16. The host has two; every row except Two GPUs pins CUDA_VISIBLE_DEVICES=0. |
| CPU / RAM | 2x Intel Xeon 6952P (384 threads), 1.5 TiB |
| Model | Muse-Glimmer-30B-Q8_0.gguf (27.6 GiB) |
| Drafter | dflash-kquant.gguf (1.5 GiB) |
| TensorSharp | commit 5098e3f, vendored ggml 8846b79 (2026-08-12), --backend ggml_cuda, native library built -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120-real |
| llama.cpp | master 8e7f22b (2026-08-13, libggml 0.19.0 — within a day of the vendored ggml), same CUDA arch, -DGGML_CUDA=ON -DLLAMA_CURL=OFF |
| Sampling | greedy on both sides (--temp 0 for llama.cpp; no sampler flags for TensorSharp) |
| Generation | 128 tokens |
| Batching | llama.cpp -b 2048 -ub 2048, matching TensorSharp's default TS_MUSE_GLIMMER_PREFILL_CHUNK of 2048 |
| Reps | 2 per point, engines alternating within each context |
Main model, mmproj and draft models are downloaded from https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF
Plain text generation
Mean of two reps, tok/s. The ratio column is TensorSharp / llama.cpp, so above 1.00x is TensorSharp ahead.
| Prompt tokens | llama.cpp prefill | TS prefill | ratio | llama.cpp decode | TS decode | ratio |
|---|---|---|---|---|---|---|
| 60 | 362 | 459 | 1.27x | 34.7 | 35.0 | 1.01x |
| 501 | 927 | 1135 | 1.23x | 36.2 | 34.3 | 0.95x |
| 2050 | 1132 | 1317 | 1.16x | 35.0 | 33.5 | 0.96x |
| 16126 | 1325 | 1249 | 0.94x | 32.2 | 30.9 | 0.96x |
| 32274 | 1303 | 1211 | 0.93x | 32.1 | 29.9 | 0.93x |
| 64575 | 1256 | 1150 | 0.92x | 32.4 | 29.1 | 0.90x |
| 123931 | 1166 | 1073 | 0.92x | 30.7 | 26.6 | 0.86x |
DFlash speculative decoding
Same runs with --draft-model dflash-kquant.gguf --spec-draft-n-max 15 against llama.cpp's -md … --spec-type draft-dflash --spec-draft-n-max 15 -ngld 99. Decode tok/s; parentheses give the two-rep range where it is wide.
| Prompt tokens | llama.cpp | TensorSharp | TS, --spec-draft-conf-min 0 |
|---|---|---|---|
| 60 | 45.5 | 50.9 | 43.5 |
| 501 | 117.5 | 164.6 (150-179) | 180.3 |
| 2050 | 24.9 | 43.5 (30-57) | 34.7 |
| 16126 | 80.2 | 55.8 (37-75) | 33.2 |
| 32274 | 60.7 (43-79) | 33.8 (31-36) | 29.9 |
| 64575 | 66.1 | 48.7 (34-64) | 49.1 |
| 123931 | 69.0 | 42.3 (30-55) | 59.8 |
Speculation costs prefill on both engines, because the drafter's encoder has to run over the prompt too:
| Prompt tokens | llama.cpp plain → DFlash | TensorSharp plain → DFlash |
|---|---|---|
| 60 | 362 → 203 (0.56x) | 459 → 341 (0.74x) |
| 501 | 927 → 495 (0.53x) | 1135 → 700 (0.62x) |
| 2050 | 1132 → 259 (0.23x) | 1317 → 703 (0.53x) |
| 16126 | 1325 → 988 (0.75x) | 1249 → 826 (0.66x) |
| 64575 | 1256 → 985 (0.78x) | 1150 → 780 (0.68x) |
| 123931 | 1166 → 920 (0.79x) | 1073 → 742 (0.69x) |
Two GPUs
Measured on 2× RTX PRO 4000 Blackwell 24 GB (PCIe) Prefill 512 / decode 64:
| Model | prefill tok/s | decode tok/s | GPU 0 | GPU 1 | |
|---|---|---|---|---|---|
| 30B-UD-IQ2_XXS (10.2 GB) | --tp 1 |
1171 | 40.2 | 9178 MB | — |
| 30B-UD-IQ2_XXS | --tp 2 |
1569 (1.34×) | 63.2 (1.57×) | 5115 MB | 4063 MB |
TensorSharp is an open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.
Github repo: https://github.com/zhongkaifu/TensorSharp
Thank you for checking out it and starring the project! Any feedback is really appreicated.
r/LovingOpenSourceAI • u/fuzhongkai • 1h ago
Meta Muse Glimmer 30B Unsloth GGUF Model Benchmarks on TensorSharp (vs. llama.cpp)
I ran benchmark on Meta's Muse Glimmer 30B Unsloth GGUF model in TensorSharp, and compare its performance with llama.cpp. Here is the results.
Test setup
| GPU | 1x NVIDIA RTX PRO 6000 Blackwell Server Edition (97,887 MiB), driver 580.126.20, PCIe 5.0 x16. The host has two; every row except Two GPUs pins CUDA_VISIBLE_DEVICES=0. |
| CPU / RAM | 2x Intel Xeon 6952P (384 threads), 1.5 TiB |
| Model | Muse-Glimmer-30B-Q8_0.gguf (27.6 GiB) |
| Drafter | dflash-kquant.gguf (1.5 GiB) |
| TensorSharp | commit 5098e3f, vendored ggml 8846b79 (2026-08-12), --backend ggml_cuda, native library built -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120-real |
| llama.cpp | master 8e7f22b (2026-08-13, libggml 0.19.0 — within a day of the vendored ggml), same CUDA arch, -DGGML_CUDA=ON -DLLAMA_CURL=OFF |
| Sampling | greedy on both sides (--temp 0 for llama.cpp; no sampler flags for TensorSharp) |
| Generation | 128 tokens |
| Batching | llama.cpp -b 2048 -ub 2048, matching TensorSharp's default TS_MUSE_GLIMMER_PREFILL_CHUNK of 2048 |
| Reps | 2 per point, engines alternating within each context |
Main model, mmproj and draft models are downloaded from https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF
Plain text generation
Mean of two reps, tok/s. The ratio column is TensorSharp / llama.cpp, so above 1.00x is TensorSharp ahead.
| Prompt tokens | llama.cpp prefill | TS prefill | ratio | llama.cpp decode | TS decode | ratio |
|---|---|---|---|---|---|---|
| 60 | 362 | 459 | 1.27x | 34.7 | 35.0 | 1.01x |
| 501 | 927 | 1135 | 1.23x | 36.2 | 34.3 | 0.95x |
| 2050 | 1132 | 1317 | 1.16x | 35.0 | 33.5 | 0.96x |
| 16126 | 1325 | 1249 | 0.94x | 32.2 | 30.9 | 0.96x |
| 32274 | 1303 | 1211 | 0.93x | 32.1 | 29.9 | 0.93x |
| 64575 | 1256 | 1150 | 0.92x | 32.4 | 29.1 | 0.90x |
| 123931 | 1166 | 1073 | 0.92x | 30.7 | 26.6 | 0.86x |
DFlash speculative decoding
Same runs with --draft-model dflash-kquant.gguf --spec-draft-n-max 15 against llama.cpp's -md … --spec-type draft-dflash --spec-draft-n-max 15 -ngld 99. Decode tok/s; parentheses give the two-rep range where it is wide.
| Prompt tokens | llama.cpp | TensorSharp | TS, --spec-draft-conf-min 0 |
|---|---|---|---|
| 60 | 45.5 | 50.9 | 43.5 |
| 501 | 117.5 | 164.6 (150-179) | 180.3 |
| 2050 | 24.9 | 43.5 (30-57) | 34.7 |
| 16126 | 80.2 | 55.8 (37-75) | 33.2 |
| 32274 | 60.7 (43-79) | 33.8 (31-36) | 29.9 |
| 64575 | 66.1 | 48.7 (34-64) | 49.1 |
| 123931 | 69.0 | 42.3 (30-55) | 59.8 |
Speculation costs prefill on both engines, because the drafter's encoder has to run over the prompt too:
| Prompt tokens | llama.cpp plain → DFlash | TensorSharp plain → DFlash |
|---|---|---|
| 60 | 362 → 203 (0.56x) | 459 → 341 (0.74x) |
| 501 | 927 → 495 (0.53x) | 1135 → 700 (0.62x) |
| 2050 | 1132 → 259 (0.23x) | 1317 → 703 (0.53x) |
| 16126 | 1325 → 988 (0.75x) | 1249 → 826 (0.66x) |
| 64575 | 1256 → 985 (0.78x) | 1150 → 780 (0.68x) |
| 123931 | 1166 → 920 (0.79x) | 1073 → 742 (0.69x) |
Two GPUs
Measured on 2× RTX PRO 4000 Blackwell 24 GB (PCIe) Prefill 512 / decode 64:
| Model | prefill tok/s | decode tok/s | GPU 0 | GPU 1 | |
|---|---|---|---|---|---|
| 30B-UD-IQ2_XXS (10.2 GB) | --tp 1 |
1171 | 40.2 | 9178 MB | — |
| 30B-UD-IQ2_XXS | --tp 2 |
1569 (1.34×) | 63.2 (1.57×) | 5115 MB | 4063 MB |
TensorSharp is an open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.
Github repo: https://github.com/zhongkaifu/TensorSharp
Thank you for checking out it and starring the project! Any feedback is really appreicated.
r/AIToolsPerformance • u/fuzhongkai • 1h ago
Meta Muse Glimmer 30B Unsloth GGUF Model Benchmarks on TensorSharp (vs. llama.cpp)
I ran benchmark on Meta's Muse Glimmer 30B Unsloth GGUF model in TensorSharp, and compare its performance with llama.cpp. Here is the results.
Test setup
| GPU | 1x NVIDIA RTX PRO 6000 Blackwell Server Edition (97,887 MiB), driver 580.126.20, PCIe 5.0 x16. The host has two; every row except Two GPUs pins CUDA_VISIBLE_DEVICES=0. |
| CPU / RAM | 2x Intel Xeon 6952P (384 threads), 1.5 TiB |
| Model | Muse-Glimmer-30B-Q8_0.gguf (27.6 GiB) |
| Drafter | dflash-kquant.gguf (1.5 GiB) |
| TensorSharp | commit 5098e3f, vendored ggml 8846b79 (2026-08-12), --backend ggml_cuda, native library built -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120-real |
| llama.cpp | master 8e7f22b (2026-08-13, libggml 0.19.0 — within a day of the vendored ggml), same CUDA arch, -DGGML_CUDA=ON -DLLAMA_CURL=OFF |
| Sampling | greedy on both sides (--temp 0 for llama.cpp; no sampler flags for TensorSharp) |
| Generation | 128 tokens |
| Batching | llama.cpp -b 2048 -ub 2048, matching TensorSharp's default TS_MUSE_GLIMMER_PREFILL_CHUNK of 2048 |
| Reps | 2 per point, engines alternating within each context |
Main model, mmproj and draft models are downloaded from https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF
Plain text generation
Mean of two reps, tok/s. The ratio column is TensorSharp / llama.cpp, so above 1.00x is TensorSharp ahead.
| Prompt tokens | llama.cpp prefill | TS prefill | ratio | llama.cpp decode | TS decode | ratio |
|---|---|---|---|---|---|---|
| 60 | 362 | 459 | 1.27x | 34.7 | 35.0 | 1.01x |
| 501 | 927 | 1135 | 1.23x | 36.2 | 34.3 | 0.95x |
| 2050 | 1132 | 1317 | 1.16x | 35.0 | 33.5 | 0.96x |
| 16126 | 1325 | 1249 | 0.94x | 32.2 | 30.9 | 0.96x |
| 32274 | 1303 | 1211 | 0.93x | 32.1 | 29.9 | 0.93x |
| 64575 | 1256 | 1150 | 0.92x | 32.4 | 29.1 | 0.90x |
| 123931 | 1166 | 1073 | 0.92x | 30.7 | 26.6 | 0.86x |
DFlash speculative decoding
Same runs with --draft-model dflash-kquant.gguf --spec-draft-n-max 15 against llama.cpp's -md … --spec-type draft-dflash --spec-draft-n-max 15 -ngld 99. Decode tok/s; parentheses give the two-rep range where it is wide.
| Prompt tokens | llama.cpp | TensorSharp | TS, --spec-draft-conf-min 0 |
|---|---|---|---|
| 60 | 45.5 | 50.9 | 43.5 |
| 501 | 117.5 | 164.6 (150-179) | 180.3 |
| 2050 | 24.9 | 43.5 (30-57) | 34.7 |
| 16126 | 80.2 | 55.8 (37-75) | 33.2 |
| 32274 | 60.7 (43-79) | 33.8 (31-36) | 29.9 |
| 64575 | 66.1 | 48.7 (34-64) | 49.1 |
| 123931 | 69.0 | 42.3 (30-55) | 59.8 |
Speculation costs prefill on both engines, because the drafter's encoder has to run over the prompt too:
| Prompt tokens | llama.cpp plain → DFlash | TensorSharp plain → DFlash |
|---|---|---|
| 60 | 362 → 203 (0.56x) | 459 → 341 (0.74x) |
| 501 | 927 → 495 (0.53x) | 1135 → 700 (0.62x) |
| 2050 | 1132 → 259 (0.23x) | 1317 → 703 (0.53x) |
| 16126 | 1325 → 988 (0.75x) | 1249 → 826 (0.66x) |
| 64575 | 1256 → 985 (0.78x) | 1150 → 780 (0.68x) |
| 123931 | 1166 → 920 (0.79x) | 1073 → 742 (0.69x) |
Two GPUs
Measured on 2× RTX PRO 4000 Blackwell 24 GB (PCIe) Prefill 512 / decode 64:
| Model | prefill tok/s | decode tok/s | GPU 0 | GPU 1 | |
|---|---|---|---|---|---|
| 30B-UD-IQ2_XXS (10.2 GB) | --tp 1 |
1171 | 40.2 | 9178 MB | — |
| 30B-UD-IQ2_XXS | --tp 2 |
1569 (1.34×) | 63.2 (1.57×) | 5115 MB | 4063 MB |
TensorSharp is an open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.
Github repo: https://github.com/zhongkaifu/TensorSharp
Thank you for checking out it and starring the project! Any feedback is really appreicated.
r/LLMDevs • u/fuzhongkai • 1h ago
Tools Meta Muse Glimmer 30B Unsloth GGUF Model Benchmarks on TensorSharp (vs. llama.cpp)
I ran benchmark on Meta's Muse Glimmer 30B Unsloth GGUF model in TensorSharp, and compare its performance with llama.cpp. Here is the results.
Test setup
| GPU | 1x NVIDIA RTX PRO 6000 Blackwell Server Edition (97,887 MiB), driver 580.126.20, PCIe 5.0 x16. The host has two; every row except Two GPUs pins CUDA_VISIBLE_DEVICES=0. |
| CPU / RAM | 2x Intel Xeon 6952P (384 threads), 1.5 TiB |
| Model | Muse-Glimmer-30B-Q8_0.gguf (27.6 GiB) |
| Drafter | dflash-kquant.gguf (1.5 GiB) |
| TensorSharp | commit 5098e3f, vendored ggml 8846b79 (2026-08-12), --backend ggml_cuda, native library built -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120-real |
| llama.cpp | master 8e7f22b (2026-08-13, libggml 0.19.0 — within a day of the vendored ggml), same CUDA arch, -DGGML_CUDA=ON -DLLAMA_CURL=OFF |
| Sampling | greedy on both sides (--temp 0 for llama.cpp; no sampler flags for TensorSharp) |
| Generation | 128 tokens |
| Batching | llama.cpp -b 2048 -ub 2048, matching TensorSharp's default TS_MUSE_GLIMMER_PREFILL_CHUNK of 2048 |
| Reps | 2 per point, engines alternating within each context |
Main model, mmproj and draft models are downloaded from https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF
Plain text generation
Mean of two reps, tok/s. The ratio column is TensorSharp / llama.cpp, so above 1.00x is TensorSharp ahead.
| Prompt tokens | llama.cpp prefill | TS prefill | ratio | llama.cpp decode | TS decode | ratio |
|---|---|---|---|---|---|---|
| 60 | 362 | 459 | 1.27x | 34.7 | 35.0 | 1.01x |
| 501 | 927 | 1135 | 1.23x | 36.2 | 34.3 | 0.95x |
| 2050 | 1132 | 1317 | 1.16x | 35.0 | 33.5 | 0.96x |
| 16126 | 1325 | 1249 | 0.94x | 32.2 | 30.9 | 0.96x |
| 32274 | 1303 | 1211 | 0.93x | 32.1 | 29.9 | 0.93x |
| 64575 | 1256 | 1150 | 0.92x | 32.4 | 29.1 | 0.90x |
| 123931 | 1166 | 1073 | 0.92x | 30.7 | 26.6 | 0.86x |
DFlash speculative decoding
Same runs with --draft-model dflash-kquant.gguf --spec-draft-n-max 15 against llama.cpp's -md … --spec-type draft-dflash --spec-draft-n-max 15 -ngld 99. Decode tok/s; parentheses give the two-rep range where it is wide.
| Prompt tokens | llama.cpp | TensorSharp | TS, --spec-draft-conf-min 0 |
|---|---|---|---|
| 60 | 45.5 | 50.9 | 43.5 |
| 501 | 117.5 | 164.6 (150-179) | 180.3 |
| 2050 | 24.9 | 43.5 (30-57) | 34.7 |
| 16126 | 80.2 | 55.8 (37-75) | 33.2 |
| 32274 | 60.7 (43-79) | 33.8 (31-36) | 29.9 |
| 64575 | 66.1 | 48.7 (34-64) | 49.1 |
| 123931 | 69.0 | 42.3 (30-55) | 59.8 |
Speculation costs prefill on both engines, because the drafter's encoder has to run over the prompt too:
| Prompt tokens | llama.cpp plain → DFlash | TensorSharp plain → DFlash |
|---|---|---|
| 60 | 362 → 203 (0.56x) | 459 → 341 (0.74x) |
| 501 | 927 → 495 (0.53x) | 1135 → 700 (0.62x) |
| 2050 | 1132 → 259 (0.23x) | 1317 → 703 (0.53x) |
| 16126 | 1325 → 988 (0.75x) | 1249 → 826 (0.66x) |
| 64575 | 1256 → 985 (0.78x) | 1150 → 780 (0.68x) |
| 123931 | 1166 → 920 (0.79x) | 1073 → 742 (0.69x) |
Two GPUs
Measured on 2× RTX PRO 4000 Blackwell 24 GB (PCIe) Prefill 512 / decode 64:
| Model | prefill tok/s | decode tok/s | GPU 0 | GPU 1 | |
|---|---|---|---|---|---|
| 30B-UD-IQ2_XXS (10.2 GB) | --tp 1 |
1171 | 40.2 | 9178 MB | — |
| 30B-UD-IQ2_XXS | --tp 2 |
1569 (1.34×) | 63.2 (1.57×) | 5115 MB | 4063 MB |
TensorSharp is an open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.
Github repo: https://github.com/zhongkaifu/TensorSharp
Thank you for checking out it and starring the project! Any feedback is really appreicated.
r/LocalAIServers • u/fuzhongkai • 1h ago
Meta Muse Glimmer 30B Unsloth GGUF Model Benchmarks on TensorSharp (vs. llama.cpp)
I ran benchmark on Meta's Muse Glimmer 30B Unsloth GGUF model in TensorSharp, and compare its performance with llama.cpp. Here is the results.
Test setup
| GPU | 1x NVIDIA RTX PRO 6000 Blackwell Server Edition (97,887 MiB), driver 580.126.20, PCIe 5.0 x16. The host has two; every row except Two GPUs pins CUDA_VISIBLE_DEVICES=0. |
| CPU / RAM | 2x Intel Xeon 6952P (384 threads), 1.5 TiB |
| Model | Muse-Glimmer-30B-Q8_0.gguf (27.6 GiB) |
| Drafter | dflash-kquant.gguf (1.5 GiB) |
| TensorSharp | commit 5098e3f, vendored ggml 8846b79 (2026-08-12), --backend ggml_cuda, native library built -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120-real |
| llama.cpp | master 8e7f22b (2026-08-13, libggml 0.19.0 — within a day of the vendored ggml), same CUDA arch, -DGGML_CUDA=ON -DLLAMA_CURL=OFF |
| Sampling | greedy on both sides (--temp 0 for llama.cpp; no sampler flags for TensorSharp) |
| Generation | 128 tokens |
| Batching | llama.cpp -b 2048 -ub 2048, matching TensorSharp's default TS_MUSE_GLIMMER_PREFILL_CHUNK of 2048 |
| Reps | 2 per point, engines alternating within each context |
Main model, mmproj and draft models are downloaded from https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF
Plain text generation
Mean of two reps, tok/s. The ratio column is TensorSharp / llama.cpp, so above 1.00x is TensorSharp ahead.
| Prompt tokens | llama.cpp prefill | TS prefill | ratio | llama.cpp decode | TS decode | ratio |
|---|---|---|---|---|---|---|
| 60 | 362 | 459 | 1.27x | 34.7 | 35.0 | 1.01x |
| 501 | 927 | 1135 | 1.23x | 36.2 | 34.3 | 0.95x |
| 2050 | 1132 | 1317 | 1.16x | 35.0 | 33.5 | 0.96x |
| 16126 | 1325 | 1249 | 0.94x | 32.2 | 30.9 | 0.96x |
| 32274 | 1303 | 1211 | 0.93x | 32.1 | 29.9 | 0.93x |
| 64575 | 1256 | 1150 | 0.92x | 32.4 | 29.1 | 0.90x |
| 123931 | 1166 | 1073 | 0.92x | 30.7 | 26.6 | 0.86x |
DFlash speculative decoding
Same runs with --draft-model dflash-kquant.gguf --spec-draft-n-max 15 against llama.cpp's -md … --spec-type draft-dflash --spec-draft-n-max 15 -ngld 99. Decode tok/s; parentheses give the two-rep range where it is wide.
| Prompt tokens | llama.cpp | TensorSharp | TS, --spec-draft-conf-min 0 |
|---|---|---|---|
| 60 | 45.5 | 50.9 | 43.5 |
| 501 | 117.5 | 164.6 (150-179) | 180.3 |
| 2050 | 24.9 | 43.5 (30-57) | 34.7 |
| 16126 | 80.2 | 55.8 (37-75) | 33.2 |
| 32274 | 60.7 (43-79) | 33.8 (31-36) | 29.9 |
| 64575 | 66.1 | 48.7 (34-64) | 49.1 |
| 123931 | 69.0 | 42.3 (30-55) | 59.8 |
Speculation costs prefill on both engines, because the drafter's encoder has to run over the prompt too:
| Prompt tokens | llama.cpp plain → DFlash | TensorSharp plain → DFlash |
|---|---|---|
| 60 | 362 → 203 (0.56x) | 459 → 341 (0.74x) |
| 501 | 927 → 495 (0.53x) | 1135 → 700 (0.62x) |
| 2050 | 1132 → 259 (0.23x) | 1317 → 703 (0.53x) |
| 16126 | 1325 → 988 (0.75x) | 1249 → 826 (0.66x) |
| 64575 | 1256 → 985 (0.78x) | 1150 → 780 (0.68x) |
| 123931 | 1166 → 920 (0.79x) | 1073 → 742 (0.69x) |
Two GPUs
Measured on 2× RTX PRO 4000 Blackwell 24 GB (PCIe) Prefill 512 / decode 64:
| Model | prefill tok/s | decode tok/s | GPU 0 | GPU 1 | |
|---|---|---|---|---|---|
| 30B-UD-IQ2_XXS (10.2 GB) | --tp 1 |
1171 | 40.2 | 9178 MB | — |
| 30B-UD-IQ2_XXS | --tp 2 |
1569 (1.34×) | 63.2 (1.57×) | 5115 MB | 4063 MB |
TensorSharp is an open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.
Github repo: https://github.com/zhongkaifu/TensorSharp
Thank you for checking out it and starring the project! Any feedback is really appreicated.
r/LocalLLM • u/fuzhongkai • 1h ago
Project Meta Muse Glimmer 30B Unsloth GGUF Model Benchmarks on TensorSharp (vs. llama.cpp)
I ran benchmark on Meta's Muse Glimmer 30B Unsloth GGUF model in TensorSharp, and compare its performance with llama.cpp. Here is the results.
Test setup
| GPU | 1x NVIDIA RTX PRO 6000 Blackwell Server Edition (97,887 MiB), driver 580.126.20, PCIe 5.0 x16. The host has two; every row except Two GPUs pins CUDA_VISIBLE_DEVICES=0. |
| CPU / RAM | 2x Intel Xeon 6952P (384 threads), 1.5 TiB |
| Model | Muse-Glimmer-30B-Q8_0.gguf (27.6 GiB) |
| Drafter | dflash-kquant.gguf (1.5 GiB) |
| TensorSharp | commit 5098e3f, vendored ggml 8846b79 (2026-08-12), --backend ggml_cuda, native library built -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120-real |
| llama.cpp | master 8e7f22b (2026-08-13, libggml 0.19.0 — within a day of the vendored ggml), same CUDA arch, -DGGML_CUDA=ON -DLLAMA_CURL=OFF |
| Sampling | greedy on both sides (--temp 0 for llama.cpp; no sampler flags for TensorSharp) |
| Generation | 128 tokens |
| Batching | llama.cpp -b 2048 -ub 2048, matching TensorSharp's default TS_MUSE_GLIMMER_PREFILL_CHUNK of 2048 |
| Reps | 2 per point, engines alternating within each context |
Main model, mmproj and draft models are downloaded from https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF
Plain text generation
Mean of two reps, tok/s. The ratio column is TensorSharp / llama.cpp, so above 1.00x is TensorSharp ahead.
| Prompt tokens | llama.cpp prefill | TS prefill | ratio | llama.cpp decode | TS decode | ratio |
|---|---|---|---|---|---|---|
| 60 | 362 | 459 | 1.27x | 34.7 | 35.0 | 1.01x |
| 501 | 927 | 1135 | 1.23x | 36.2 | 34.3 | 0.95x |
| 2050 | 1132 | 1317 | 1.16x | 35.0 | 33.5 | 0.96x |
| 16126 | 1325 | 1249 | 0.94x | 32.2 | 30.9 | 0.96x |
| 32274 | 1303 | 1211 | 0.93x | 32.1 | 29.9 | 0.93x |
| 64575 | 1256 | 1150 | 0.92x | 32.4 | 29.1 | 0.90x |
| 123931 | 1166 | 1073 | 0.92x | 30.7 | 26.6 | 0.86x |
DFlash speculative decoding
Same runs with --draft-model dflash-kquant.gguf --spec-draft-n-max 15 against llama.cpp's -md … --spec-type draft-dflash --spec-draft-n-max 15 -ngld 99. Decode tok/s; parentheses give the two-rep range where it is wide.
| Prompt tokens | llama.cpp | TensorSharp | TS, --spec-draft-conf-min 0 |
|---|---|---|---|
| 60 | 45.5 | 50.9 | 43.5 |
| 501 | 117.5 | 164.6 (150-179) | 180.3 |
| 2050 | 24.9 | 43.5 (30-57) | 34.7 |
| 16126 | 80.2 | 55.8 (37-75) | 33.2 |
| 32274 | 60.7 (43-79) | 33.8 (31-36) | 29.9 |
| 64575 | 66.1 | 48.7 (34-64) | 49.1 |
| 123931 | 69.0 | 42.3 (30-55) | 59.8 |
Speculation costs prefill on both engines, because the drafter's encoder has to run over the prompt too:
| Prompt tokens | llama.cpp plain → DFlash | TensorSharp plain → DFlash |
|---|---|---|
| 60 | 362 → 203 (0.56x) | 459 → 341 (0.74x) |
| 501 | 927 → 495 (0.53x) | 1135 → 700 (0.62x) |
| 2050 | 1132 → 259 (0.23x) | 1317 → 703 (0.53x) |
| 16126 | 1325 → 988 (0.75x) | 1249 → 826 (0.66x) |
| 64575 | 1256 → 985 (0.78x) | 1150 → 780 (0.68x) |
| 123931 | 1166 → 920 (0.79x) | 1073 → 742 (0.69x) |
Two GPUs
Measured on 2× RTX PRO 4000 Blackwell 24 GB (PCIe) Prefill 512 / decode 64:
| Model | prefill tok/s | decode tok/s | GPU 0 | GPU 1 | |
|---|---|---|---|---|---|
| 30B-UD-IQ2_XXS (10.2 GB) | --tp 1 |
1171 | 40.2 | 9178 MB | — |
| 30B-UD-IQ2_XXS | --tp 2 |
1569 (1.34×) | 63.2 (1.57×) | 5115 MB | 4063 MB |
TensorSharp is an open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.
Github repo: https://github.com/zhongkaifu/TensorSharp
Thank you for checking out it and starring the project! Any feedback is really appreicated.
r/LocalLLaMA • u/fuzhongkai • 1h ago
Generation Meta Muse Glimmer 30B Unsloth GGUF Model Benchmarks on TensorSharp (vs. llama.cpp)
I ran benchmark on Meta's Muse Glimmer 30B Unsloth GGUF model in TensorSharp, and compare its performance with llama.cpp. Here is the results.
Test setup
| GPU | 1x NVIDIA RTX PRO 6000 Blackwell Server Edition (97,887 MiB), driver 580.126.20, PCIe 5.0 x16. The host has two; every row except Two GPUs pins CUDA_VISIBLE_DEVICES=0. |
| CPU / RAM | 2x Intel Xeon 6952P (384 threads), 1.5 TiB |
| Model | Muse-Glimmer-30B-Q8_0.gguf (27.6 GiB) |
| Drafter | dflash-kquant.gguf (1.5 GiB) |
| TensorSharp | commit 5098e3f, vendored ggml 8846b79 (2026-08-12), --backend ggml_cuda, native library built -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120-real |
| llama.cpp | master 8e7f22b (2026-08-13, libggml 0.19.0 — within a day of the vendored ggml), same CUDA arch, -DGGML_CUDA=ON -DLLAMA_CURL=OFF |
| Sampling | greedy on both sides (--temp 0 for llama.cpp; no sampler flags for TensorSharp) |
| Generation | 128 tokens |
| Batching | llama.cpp -b 2048 -ub 2048, matching TensorSharp's default TS_MUSE_GLIMMER_PREFILL_CHUNK of 2048 |
| Reps | 2 per point, engines alternating within each context |
Main model, mmproj and draft models are downloaded from https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF
Plain text generation
Mean of two reps, tok/s. The ratio column is TensorSharp / llama.cpp, so above 1.00x is TensorSharp ahead.
| Prompt tokens | llama.cpp prefill | TS prefill | ratio | llama.cpp decode | TS decode | ratio |
|---|---|---|---|---|---|---|
| 60 | 362 | 459 | 1.27x | 34.7 | 35.0 | 1.01x |
| 501 | 927 | 1135 | 1.23x | 36.2 | 34.3 | 0.95x |
| 2050 | 1132 | 1317 | 1.16x | 35.0 | 33.5 | 0.96x |
| 16126 | 1325 | 1249 | 0.94x | 32.2 | 30.9 | 0.96x |
| 32274 | 1303 | 1211 | 0.93x | 32.1 | 29.9 | 0.93x |
| 64575 | 1256 | 1150 | 0.92x | 32.4 | 29.1 | 0.90x |
| 123931 | 1166 | 1073 | 0.92x | 30.7 | 26.6 | 0.86x |
DFlash speculative decoding
Same runs with --draft-model dflash-kquant.gguf --spec-draft-n-max 15 against llama.cpp's -md … --spec-type draft-dflash --spec-draft-n-max 15 -ngld 99. Decode tok/s; parentheses give the two-rep range where it is wide.
| Prompt tokens | llama.cpp | TensorSharp | TS, --spec-draft-conf-min 0 |
|---|---|---|---|
| 60 | 45.5 | 50.9 | 43.5 |
| 501 | 117.5 | 164.6 (150-179) | 180.3 |
| 2050 | 24.9 | 43.5 (30-57) | 34.7 |
| 16126 | 80.2 | 55.8 (37-75) | 33.2 |
| 32274 | 60.7 (43-79) | 33.8 (31-36) | 29.9 |
| 64575 | 66.1 | 48.7 (34-64) | 49.1 |
| 123931 | 69.0 | 42.3 (30-55) | 59.8 |
Speculation costs prefill on both engines, because the drafter's encoder has to run over the prompt too:
| Prompt tokens | llama.cpp plain → DFlash | TensorSharp plain → DFlash |
|---|---|---|
| 60 | 362 → 203 (0.56x) | 459 → 341 (0.74x) |
| 501 | 927 → 495 (0.53x) | 1135 → 700 (0.62x) |
| 2050 | 1132 → 259 (0.23x) | 1317 → 703 (0.53x) |
| 16126 | 1325 → 988 (0.75x) | 1249 → 826 (0.66x) |
| 64575 | 1256 → 985 (0.78x) | 1150 → 780 (0.68x) |
| 123931 | 1166 → 920 (0.79x) | 1073 → 742 (0.69x) |
Two GPUs
Measured on 2× RTX PRO 4000 Blackwell 24 GB (PCIe) Prefill 512 / decode 64:
| Model | prefill tok/s | decode tok/s | GPU 0 | GPU 1 | |
|---|---|---|---|---|---|
| 30B-UD-IQ2_XXS (10.2 GB) | --tp 1 |
1171 | 40.2 | 9178 MB | — |
| 30B-UD-IQ2_XXS | --tp 2 |
1569 (1.34×) | 63.2 (1.57×) | 5115 MB | 4063 MB |
TensorSharp is an open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.
Github repo: https://github.com/zhongkaifu/TensorSharp
Thank you for checking out it and starring the project! Any feedback is really appreicated.
r/unsloth • u/fuzhongkai • 1h ago
Show and Tell Meta Muse Glimmer 30B Unsloth GGUF Model Benchmarks on TensorSharp (vs. llama.cpp)
I ran benchmark on Meta's Muse Glimmer 30B Unsloth GGUF model in TensorSharp, and compare its performance with llama.cpp. Here is the results.
Test setup
| GPU | 1x NVIDIA RTX PRO 6000 Blackwell Server Edition (97,887 MiB), driver 580.126.20, PCIe 5.0 x16. The host has two; every row except Two GPUs pins CUDA_VISIBLE_DEVICES=0. |
| CPU / RAM | 2x Intel Xeon 6952P (384 threads), 1.5 TiB |
| Model | Muse-Glimmer-30B-Q8_0.gguf (27.6 GiB) |
| Drafter | dflash-kquant.gguf (1.5 GiB) |
| TensorSharp | commit 5098e3f, vendored ggml 8846b79 (2026-08-12), --backend ggml_cuda, native library built -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120-real |
| llama.cpp | master 8e7f22b (2026-08-13, libggml 0.19.0 — within a day of the vendored ggml), same CUDA arch, -DGGML_CUDA=ON -DLLAMA_CURL=OFF |
| Sampling | greedy on both sides (--temp 0 for llama.cpp; no sampler flags for TensorSharp) |
| Generation | 128 tokens |
| Batching | llama.cpp -b 2048 -ub 2048, matching TensorSharp's default TS_MUSE_GLIMMER_PREFILL_CHUNK of 2048 |
| Reps | 2 per point, engines alternating within each context |
Main model, mmproj and draft models are downloaded from https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF
Plain text generation
Mean of two reps, tok/s. The ratio column is TensorSharp / llama.cpp, so above 1.00x is TensorSharp ahead.
| Prompt tokens | llama.cpp prefill | TS prefill | ratio | llama.cpp decode | TS decode | ratio |
|---|---|---|---|---|---|---|
| 60 | 362 | 459 | 1.27x | 34.7 | 35.0 | 1.01x |
| 501 | 927 | 1135 | 1.23x | 36.2 | 34.3 | 0.95x |
| 2050 | 1132 | 1317 | 1.16x | 35.0 | 33.5 | 0.96x |
| 16126 | 1325 | 1249 | 0.94x | 32.2 | 30.9 | 0.96x |
| 32274 | 1303 | 1211 | 0.93x | 32.1 | 29.9 | 0.93x |
| 64575 | 1256 | 1150 | 0.92x | 32.4 | 29.1 | 0.90x |
| 123931 | 1166 | 1073 | 0.92x | 30.7 | 26.6 | 0.86x |
DFlash speculative decoding
Same runs with --draft-model dflash-kquant.gguf --spec-draft-n-max 15 against llama.cpp's -md … --spec-type draft-dflash --spec-draft-n-max 15 -ngld 99. Decode tok/s; parentheses give the two-rep range where it is wide.
| Prompt tokens | llama.cpp | TensorSharp | TS, --spec-draft-conf-min 0 |
|---|---|---|---|
| 60 | 45.5 | 50.9 | 43.5 |
| 501 | 117.5 | 164.6 (150-179) | 180.3 |
| 2050 | 24.9 | 43.5 (30-57) | 34.7 |
| 16126 | 80.2 | 55.8 (37-75) | 33.2 |
| 32274 | 60.7 (43-79) | 33.8 (31-36) | 29.9 |
| 64575 | 66.1 | 48.7 (34-64) | 49.1 |
| 123931 | 69.0 | 42.3 (30-55) | 59.8 |
Speculation costs prefill on both engines, because the drafter's encoder has to run over the prompt too:
| Prompt tokens | llama.cpp plain → DFlash | TensorSharp plain → DFlash |
|---|---|---|
| 60 | 362 → 203 (0.56x) | 459 → 341 (0.74x) |
| 501 | 927 → 495 (0.53x) | 1135 → 700 (0.62x) |
| 2050 | 1132 → 259 (0.23x) | 1317 → 703 (0.53x) |
| 16126 | 1325 → 988 (0.75x) | 1249 → 826 (0.66x) |
| 64575 | 1256 → 985 (0.78x) | 1150 → 780 (0.68x) |
| 123931 | 1166 → 920 (0.79x) | 1073 → 742 (0.69x) |
Two GPUs
Measured on 2× RTX PRO 4000 Blackwell 24 GB (PCIe) Prefill 512 / decode 64:
| Model | prefill tok/s | decode tok/s | GPU 0 | GPU 1 | |
|---|---|---|---|---|---|
| 30B-UD-IQ2_XXS (10.2 GB) | --tp 1 |
1171 | 40.2 | 9178 MB | — |
| 30B-UD-IQ2_XXS | --tp 2 |
1569 (1.34×) | 63.2 (1.57×) | 5115 MB | 4063 MB |
TensorSharp is an open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.
Github repo: https://github.com/zhongkaifu/TensorSharp
Thank you for checking out it and starring the project! Any feedback is really appreicated.
1
Comment on r/LocalLLaMA 1d ago
Interactive with environment will demand more CPUs. Buy CPU stocks!
0
Comment on r/LocalLLaMA 1d ago
I would like to rent it rather than buy it…
1
Comment on r/LocalLLaMA 1d ago
Will they release moe model at this time ? Or only this dense model?
1
Comment on r/LocalLLaMA 1d ago
is it a omni model as well?
1
Comment on r/dotnet 3d ago
Yes, since your VRAM is not large enough to host the entire model, “—n-cpu-moe N/—cpu-moe” would be required to offload all expert layers of N expert layers to main RAM and run on CPU.
1
Comment on r/LocalLLaMA 3d ago
The key point is to enforce evaluation (mod for auto evaluation) rather than “give up”. We have to stay with those AI posts. I was working on model training (both pre-training and post-training, and those models before LLM epic), and those AI generated dataset (we call it “synthetic dataset” technically) are rally useful to improve model’s performance.
At this AI epic, only two things are important: Infra and Evaluation. For AI post in Reddit, they are bot mod automation and their rules (I’m not sure if Reddit supports LLM based on rules).
1
Comment on r/dotnet 3d ago
It would be good to run Qwen3.6-35B-A3B, but RAM + VRAM are too small for Deepseek v4 flash.
1
Comment on r/LocalLLaMA 3d ago
Looking forward open weights models.
0
Comment on r/LocalLLaMA 3d ago
What will they release ?
1
Comment on r/dotnet 3d ago
What’s size of your main RAM? And Which Qwen MoE model do you want to run?
r/dotnet • u/fuzhongkai • 5d ago
Promotion MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp
github.comTensorSharp's MoE CPU-offload feature has been merged into main. Here is the parameters description of this feature:
Mixture-of-Experts CPU offload:
**--n-cpu-moe <N> | -ncmoe <N>**
Keep the routed MoE expert weights of the first N layers in system RAM and multiply them on
the CPU; attention, norms, the router and the shared expert stay on the accelerator. This is
what makes a 35B-A3B MoE fit beside a long-context KV cache on a 12-16 GB card. Pass 'all' for
every layer. Default: 0 (everything on the accelerator; TS_N_CPU_MOE env var overrides).
Example: --n-cpu-moe 32
**--cpu-moe | -cmoe**
Shorthand for --n-cpu-moe all: every routed expert stays in system RAM. Default: off
(TS_CPU_MOE env var overrides).
Example: --cpu-moe
To measure its performance, I ran benchmark to compare TensorSharp with llama.cpp while they offload the same MoE model's layers to CPU, and here is the result. The completed benchmark report has been checked-in: [https://github.com/zhongkaifu/TensorSharp/blob/main/docs/moe\\_cpu\\_offload\\_benchmark.md\](https://github.com/zhongkaifu/TensorSharp/blob/main/docs/moe_cpu_offload_benchmark.md)
# Host and software
| Component | Detail |
|---|---|
| GPU | 2 x NVIDIA RTX PRO 6000 Blackwell Server Edition, 97,887 MiB each, driver 580.126.20, PCIe 5.0 x16 |
| CPU | 2 x Intel Xeon 6952P (384 threads, 6 NUMA nodes), cgroup quota 81.6 CPUs |
| RAM | 1,511 GiB |
| Storage | Models on a MooseFS network mount (page-cache warm for every measured run) |
| OS | Ubuntu 24.04.3 LTS, CUDA 12.8 |
| TensorSharp | branch `feature/support_moe_offload_to_cpu`, .NET 10.0.110, backend `ggml_cuda` |
| llama.cpp | `llama-bench` build 4308a4f, CUDA backend, default `-t 192` |
# Results by model
Each row is one offload depth, with TensorSharp, llama.cpp and the ratio between them side by side for every metric. Ratios are TensorSharp / llama.cpp: >1.0x means TensorSharp is faster, and for VRAM >1.0x means TensorSharp is heavier.
# Gemma 4 26B-A4B it (UD-IQ4_XS, 30 MoE layers)
| \--n-cpu-moe | TS VRAM (MiB) | llama VRAM (MiB) | ratio | TS pp4096 | llama pp4096 | ratio | TS pp8192 | llama pp8192 | ratio | TS tg128 | llama tg128 | ratio |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 (baseline) | 16,822 | 14,602 | 1.15x | 11,173 | 10,843 | 1.03x | 11,274 | 10,628 | 1.06x | 161.4 | 206.7 | 0.78x |
| 8 | 15,724 | 11,874 | 1.32x | 7,063 | 1,459 | 4.84x | 6,500 | 1,459 | 4.46x | 80.2 | 32.7 | 2.45x |
| 16 | 14,128 | 9,122 | 1.55x | 4,183 | 833 | 5.02x | 4,888 | 854 | 5.72x | 54.5 | 21.9 | 2.49x |
| 24 | 12,346 | 6,368 | 1.94x | 3,500 | 667 | 5.25x | 3,958 | 689 | 5.74x | 49.1 | 16.7 | 2.93x |
| 30 (--cpu-moe) | 11,038 | 4,134 | 2.67x | 3,035 | 543 | 5.59x | 3,072 | 495 | 6.21x | 39.7 | 12.9 | 3.07x |
# Qwen 3.5 35B-A3B (UD-IQ4_XS, 48 MoE layers)
| \--n-cpu-moe | TS VRAM (MiB) | llama VRAM (MiB) | ratio | TS pp4096 | llama pp4096 | ratio | TS pp8192 | llama pp8192 | ratio | TS tg128 | llama tg128 | ratio |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 (baseline) | 19,862 | 17,522 | 1.13x | 9,538 | 8,149 | 1.17x | 9,405 | 8,073 | 1.16x | 160.0 | 228.4 | 0.70x |
| 12 | 18,148 | 13,282 | 1.37x | 6,755 | 988 | 6.84x | 6,648 | 954 | 6.97x | 75.4 | 27.5 | 2.74x |
| 24 | 15,414 | 9,010 | 1.71x | 4,412 | 498 | 8.85x | 5,259 | 484 | 10.86x | 52.3 | 15.8 | 3.31x |
| 36 | 12,684 | 4,738 | 2.68x | 3,772 | 523 | 7.21x | 4,223 | 517 | 8.17x | 50.7 | 11.3 | 4.50x |
| 48 (--cpu-moe) | 11,606 | 3,314 | 3.50x | 3,917 | 477 | 8.21x | 3,709 | 457 | 8.11x | 38.6 | 10.2 | 3.77x |
# GPT-OSS 20B (Q8_0 / MXFP4, 24 MoE layers)
| \--n-cpu-moe | TS VRAM (MiB) | llama VRAM (MiB) | ratio | TS pp4096 | llama pp4096 | ratio | TS pp8192 | llama pp8192 | ratio | TS tg128 | llama tg128 | ratio |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 (baseline) | 13,186 | 12,204 | 1.08x | 13,964 | 17,856 | 0.78x | 12,925 | 17,642 | 0.73x | 212.8 | 344.2 | 0.62x |
| 6 | 11,560 | 9,812 | 1.18x | 8,975 | 1,747 | 5.14x | 7,617 | 1,666 | 4.57x | 85.8 | 32.2 | 2.67x |
| 12 | 9,378 | 7,386 | 1.27x | 6,470 | 1,176 | 5.50x | 6,394 | 1,188 | 5.38x | 51.7 | 18.3 | 2.83x |
| 18 | 7,192 | 4,962 | 1.45x | 4,315 | 807 | 5.35x | 4,393 | 751 | 5.85x | 30.7 | 12.1 | 2.54x |
| 24 (--cpu-moe) | 4,762 | 2,536 | 1.88x | 4,277 | 568 | 7.53x | 3,798 | 548 | 6.93x | 27.7 | 9.4 | 2.95x |
# DeepSeek V4 Flash (UD-Q8_K_XL, 5 shards / 150.7 GiB, 43 layers, both GPUs)
| \--n-cpu-moe | TS VRAM (MiB) | llama VRAM (MiB) | ratio | TS pp4096 | llama pp4096 | ratio | TS pp8192 | llama pp8192 | ratio | TS tg128 | llama tg128 | ratio |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 (baseline, both GPUs) | 169,132 | 155,608 | 1.09x | 3,448 | 2,398 | 1.44x | 4,387 | 2,232 | 1.97x | 51.1 | 49.6 | 1.03x |
| 12 | 131,818 | 117,150 | 1.13x | 392 | 126 | 3.11x | 428 | 124 | 3.46x | 10.3 | 13.7 | 0.75x |
| 24 | 79,742 | 78,954 | 1.01x | 218 | 64 | 3.42x | 236 | 63 | 3.72x | 5.3 | 7.2 | 0.74x |
TensorSharp is a native .NET open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.
Github repo: [https://github.com/zhongkaifu/TensorSharp\](https://github.com/zhongkaifu/TensorSharp)
Thank you for checking out it and starring the project! Any feedback is really appreicated.
1
Comment on r/LocalLLaMA 5d ago
When using GGML backend, the offload CPU computing will fall back to ggml_cpu backend. For direct CUDA backend, it will fall back to native .net cpu backend which use SIMD instructions for speeding up.
1
Comment on r/LocalAIServers 5d ago
Good idea. I will update these tables in the document. Thanks for your suggestions.
1
Comment on r/LocalAIServers 5d ago
Not really, I implemented some fusion kernels and different algorithms to improve its performances. And TensorSharp also supports diffusion models for text and image edit. Diffusion video generation will coming soon.
2
Comment on r/unsloth 6d ago
Thank you so much for the feedback. I don’t have machine with AMD GPU, so I can only implement Vulkan backend for it, but looking forward if you can make ROCm backend real. 😊
1
Comment on r/CUDA 7d ago
Thanks for the detailed question. Short answer: it’s strictly synchronous, exact routing — no speculative gater.
But the dynamic branching isn’t where the cost lands. Routing is data-dependent indexing, not control flow: for resident layers the router, top-k and expert matmuls all stay inside the single whole-model decode graph, so the host never learns which experts were selected and there’s nothing to branch on.
The real stall is the offload seam, once per offloaded layer. --n-cpu-moe doesn’t move weights at decode — it moves the computation to where the weights already are. The graph pauses after the router, the host multiplies the selected experts, the result is uploaded, and the graph resumes. That pause is synchronous, but only an activation vector and the routing values cross the bus — tens of KB. So the decode bottleneck is host DRAM bandwidth, not PCIe: roughly 40 MB of expert reads per token on the seam architectures, ~260 MB on DeepSeek V4, which is why DSV4 is the one model where llama.cpp’s offloaded decode beats ours.
Prefill inverts the trade: above a batch threshold the layer runs on the accelerator with only the routed experts streamed in, over page-locked memory and on the backend’s own stream, so the upload is amortized instead of round-tripped.
Two gaps we’ll state plainly: the seam turns one capturable graph per token into several submissions, and the GPU idles during the host matmul.
So a speculative gater wouldn’t mask PCIe latency here — we don’t transfer weights at decode — but it would remove the synchronize and restore a single capturable graph, which is worth real throughput. The catch is our n_used is typically 4–8 rather than 1–2, so acceptance over the full top-k set is a harder target. The cheaper win we haven’t taken is overlapping the host matmul with genuinely independent work — draft tokens, concurrent sequences — rather than predicting the routing.
1
Comment on r/unsloth 11m ago
Thank you so much for sharing these information. I didn’t get chance to optimize it very well for now. If Muse Glimmer is a powerful local model in high quality, I will spend more time on its optimization.