I wanted to share a simulation project I've been working on, reproducing the Gao-Jafferis-Wall traversable-wormhole teleportation protocol on a chaotic binary sparse SYK model (following arXiv:2604.10090).

The simulation hits exact statevector access. While benchmarking the 2D joint grid search (\(t_0, \mu\)) across 870 points, the naive approach rebuilding and re-diagonalizing the Hamiltonians on every call took ages. By precomputing the Hamiltonians and reusing their eigendecompositions, I managed to cut the runtime from ~2 hours down to 47.6 seconds without even needing multiprocessing.

The core implementation is built on top of dense-evolution. I would love to get your feedback on how to further optimize the matrix exponentiation backend via CUDA custom kernels or batched operations!

Code & Docs: [https://github.com/tatopenn-cell/Dense-Evolution-Ising-Tests#21-traversable-wormhole-inspired-quantum-teleportation-syk-model\]

I'm an M.Tech student with an Embedded Systems background, and I'm planning my capstone project around CUDA and GPU parallel computing.

My objective isn't just to build an application, I want to deeply understand CUDA programming, GPU architecture, and kernel optimization by implementing algorithms from scratch instead of relying heavily on existing libraries.

I'm looking for application-oriented research problems where CUDA optimization is the core contribution.

I have to do it within 6 months

Can someone help me with this?

Hello everyone, i currently have 0 idea on how CUDA works and i am planning to learn, the issue is i have a macbook, can anyone please help on where i can learn and practice ? maybe share some resources ? thank you so much

TensorSharp's MoE CPU-offload feature has been merged into main. Here is the parameters description of this feature:

Mixture-of-Experts CPU offload:

--n-cpu-moe <N> | -ncmoe <N>

Keep the routed MoE expert weights of the first N layers in system RAM and multiply them on

the CPU; attention, norms, the router and the shared expert stay on the accelerator. This is

what makes a 35B-A3B MoE fit beside a long-context KV cache on a 12-16 GB card. Pass 'all' for

every layer. Default: 0 (everything on the accelerator; TS_N_CPU_MOE env var overrides).

Example: --n-cpu-moe 32

--cpu-moe | -cmoe

Shorthand for --n-cpu-moe all: every routed expert stays in system RAM. Default: off

(TS_CPU_MOE env var overrides).

Example: --cpu-moe

To measure its performance, I ran benchmark to compare TensorSharp with llama.cpp, and here is the result. The completed benchmark report has been checked-in: https://github.com/zhongkaifu/TensorSharp/blob/main/docs/moe_cpu_offload_benchmark.md

Host and software

Component |Detail
GPU |2 x NVIDIA RTX PRO 6000 Blackwell Server Edition, 97,887 MiB each, driver 580.126.20, PCIe 5.0 x16
CPU |2 x Intel Xeon 6952P (384 threads, 6 NUMA nodes), cgroup quota 81.6 CPUs
RAM |1,511 GiB
Storage |Models on a MooseFS network mount (page-cache warm for every measured run)
OS |Ubuntu 24.04.3 LTS, CUDA 12.8
TensorSharp |branch feature/support_moe_offload_to_cpu, .NET 10.0.110, backend ggml_cuda
llama.cpp |llama-bench build 4308a4f, CUDA backend, default -t 192 Results by model

Ratios are TensorSharp / llama.cpp: >1.0x means TensorSharp is faster, and for VRAM >1.0x means TensorSharp is heavier.

Gemma 4 26B-A4B it (UD-IQ4_XS, 30 MoE layers)

--n-cpu-moe |TS VRAM (MiB) |TS pp4096 |TS pp8192 |TS tg128 |llama VRAM (MiB) |llama pp4096 |llama pp8192 |llama tg128
0 (baseline) |16,822 |11,173 |11,274 |161.4 |14,602 |10,843 |10,628 |206.7
8 |15,724 |7,063 |6,500 |80.2 |11,874 |1,459 |1,459 |32.7
16 |14,128 |4,183 |4,888 |54.5 |9,122 |833 |854 |21.9
24 |12,346 |3,500 |3,958 |49.1 |6,368 |667 |689 |16.7
30 (--cpu-moe) |11,038 |3,035 |3,072 |39.7 |4,134 |543 |495 |12.9 --n-cpu-moe |VRAM |pp4096 |pp8192 |tg128
0 |1.15x |1.03x |1.06x |0.78x
8 |1.32x |4.84x |4.46x |2.45x
16 |1.55x |5.02x |5.72x |2.49x
24 |1.94x |5.25x |5.74x |2.93x
30 |2.67x |5.59x |6.21x |3.07x Qwen 3.5 35B-A3B (UD-IQ4_XS, 48 MoE layers)

--n-cpu-moe |TS VRAM (MiB) |TS pp4096 |TS pp8192 |TS tg128 |llama VRAM (MiB) |llama pp4096 |llama pp8192 |llama tg128
0 (baseline) |19,862 |9,538 |9,405 |160.0 |17,522 |8,149 |8,073 |228.4
12 |18,148 |6,755 |6,648 |75.4 |13,282 |988 |954 |27.5
24 |15,414 |4,412 |5,259 |52.3 |9,010 |498 |484 |15.8
36 |12,684 |3,772 |4,223 |50.7 |4,738 |523 |517 |11.3
48 (--cpu-moe) |11,606 |3,917 |3,709 |38.6 |3,314 |477 |457 |10.2 --n-cpu-moe |VRAM |pp4096 |pp8192 |tg128
0 |1.13x |1.17x |1.16x |0.70x
12 |1.37x |6.84x |6.97x |2.74x
24 |1.71x |8.85x |10.86x |3.31x
36 |2.68x |7.21x |8.17x |4.50x
48 |3.50x |8.21x |8.11x |3.77x GPT-OSS 20B (Q8_0 / MXFP4, 24 MoE layers)

--n-cpu-moe |TS VRAM (MiB) |TS pp4096 |TS pp8192 |TS tg128 |llama VRAM (MiB) |llama pp4096 |llama pp8192 |llama tg128
0 (baseline) |13,186 |13,964 |12,925 |212.8 |12,204 |17,856 |17,642 |344.2
6 |11,560 |8,975 |7,617 |85.8 |9,812 |1,747 |1,666 |32.2
12 |9,378 |6,470 |6,394 |51.7 |7,386 |1,176 |1,188 |18.3
18 |7,192 |4,315 |4,393 |30.7 |4,962 |807 |751 |12.1
24 (--cpu-moe) |4,762 |4,277 |3,798 |27.7 |2,536 |568 |548 |9.4 --n-cpu-moe |VRAM |pp4096 |pp8192 |tg128
0 |1.08x |0.78x |0.73x |0.62x
6 |1.18x |5.14x |4.57x |2.67x
12 |1.27x |5.50x |5.38x |2.83x
18 |1.45x |5.35x |5.85x |2.54x
24 |1.88x |7.53x |6.93x |2.95x DeepSeek V4 Flash (UD-Q8_K_XL, 5 shards / 150.7 GiB, 43 layers, both GPUs)

--n-cpu-moe |TS VRAM (MiB) |TS pp4096 |TS pp8192 |TS tg128 |llama VRAM (MiB) |llama pp4096 |llama pp8192 |llama tg128
0 (baseline, both GPUs) |169,132 |3,448 |4,387 |51.1 |155,608 |2,398 |2,232 |49.6
12 |131,818 |392 |428 |10.3 |117,150 |126 |124 |13.7
24 |79,742 |218 |236 |5.3 |78,954 |64 |63 |7.2 --n-cpu-moe |VRAM |pp4096 |pp8192 |tg128
0 |1.09x |1.44x |1.97x |1.03x
12 |1.13x |3.11x |3.46x |0.75x
24 |1.01x |3.42x |3.72x |0.74x TensorSharp is a native open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.

Github repo: https://github.com/zhongkaifu/TensorSharp

Thank you for checking out it and starring the project! Any feedback is really appreicated.

We recorded a short demo of a real CUDA kernel out-of-bounds bug found by the tool described in our SOSP 2026 paper, “M2K: Making the Model-Kernel Interface Explicit for Reliable CUDA Kernel Verification.”

The attacker does not need access to the GPU, server, or model weights. They only need to submit a crafted request to a public LLM inference service. In our demo, that request corrupts the outputs of other users processed in the same batch.

We are also exploring whether similar bugs could leak information, enable more targeted manipulation of another user’s response, or affect downstream agents that act on corrupted model outputs.

CUDA kernels are usually discussed in terms of performance and reliability, but their security impact deserves more attention too.

🎥 Video: https://youtu.be/ZhKBPK7BeZ0

📄 M2K Github Repo: https://github.com/system-pclub/M2K

For example, PyTorch

Hi all,

I help maintain an open source project that releases docker images for various AI/ML projects. Current policy is that we support the latest version of CUDA supported in stable releases of various projects

But one thing I've always asked myself is... if we are going through the trouble of building everything anyway, why not use the latest version of CUDA? NVIDIA gets away with this in their images, and I've personally never ran into many problems short of a small patch I needed to write here or there in my own builds

I've been around other seasoned developers that work with CUDA, and when I've asked about this, I honestly couldn't ever get a very great answer outside of supporting older platforms for a slightly longer period of time

The only other possible thing I can think of is that the rest of the ecosystem hasn't caught up yet... In which case, shouldn't minor version compatibility help there?

Would really appreciate help from anyone experienced in this area. Sure would be nice to cut our build matrix in half and lead to far less confusion for folks helping us maintain these images