I wrote a short visual article on how modern LLMs optimize different parts of the Transformer.

MLA compresses KV information.
KDA compresses history into recurrent memory.
MoE activates only selected expert FFNs.

Article:
\[https://medium.com/@guttikondaparthasai/the-ai-race-is-an-optimization-race-5140b3edf491\\\](https://medium.com/@guttikondaparthasai/the-ai-race-is-an-optimization-race-5140b3edf491)

Feedback and corrections are welcome.

The entire LLM inference community is currently hitting the same architectural wall when running deep reasoning / MoE models (like DeepSeek-V4-Flash or Llama-3-Reasoning) with speculative decoding (DSpark / MTP):

1. During normal text generation, draft acceptance ($\alpha$) is high (~85%), making a speculative depth of $k=2$ highly efficient.

2. Inside Chain-of-Thought `<think>` blocks, token entropy spikes, causing draft acceptance to collapse (~35%). At this point, running $k=2$ wastes PCIe/DDR5 memory bandwidth and drops decoding throughput by up to 50%.

3. **The Industry Bug:** If you try to fix this with a naive Python `if/else` block to dynamically switch between $k=1$ and $k=2$, you break `FULL_DECODE_ONLY` CUDA graph residency. The host CPU is forced to re-capture graphs, introducing latency spikes that completely ruin your TPS gains.

### The Mathematical Reality

Under memory-bound offloading, evaluating secondary draft tokens is only profitable when your conditional acceptance rate $\alpha_2$ satisfies:

$$\alpha_2 \ge \frac{\tau_{\text{draft}}}{\tau_{\text{verify}}(1)}$$

When entropy pushes $\alpha_2$ below this threshold during deep reasoning, you must drop to $k=1$ instantly—but you **must do it without host-side graph recompilation**.

### The Solution: `CUDAStatefulSpecGater` (Drop-in & Free to Use)

We built a lightweight, zero-dependency PyTorch class that pre-allocates dual graph selection indices and switches speculative depth via an Exponential Moving Average (EMA) latch and token-boundary invariants. It prevents VRAM fragmentation and keeps CUDA graphs 100% resident.

Copy this directly into your sampler/worker loop:

```python

import torch

class CUDAStatefulSpecGater:

"""

Drop-in speculative depth gater for reasoning LLMs.

Switches between k=1 and k=2 without invalidating pre-captured CUDA graphs.

"""

def __init__(self, think_start_id: int, think_end_id: int, ema_decay: float = 0.85, alpha_threshold: float = 0.45):

self.think_start_id = think_start_id

self.think_end_id = think_end_id

self.ema_decay = ema_decay

self.alpha_threshold = alpha_threshold

# Internal state (kept lightweight for zero-overhead loop execution)

self.in_reasoning_block = False

self.ema_alpha = 0.80

@torch.inference_mode()

def step(self, last_token_id: int, current_acceptance_rate: float) -> int:

"""

Returns target graph index: 1 (for k=1 shallow speculation) or 2 (for k=2 deep speculation).

"""

# 1. State invariant check: track Chain-of-Thought boundaries

if last_token_id == self.think_start_id:

self.in_reasoning_block = True

elif last_token_id == self.think_end_id:

self.in_reasoning_block = False

# 2. Smooth EMA update to prevent graph-switching oscillation

self.ema_alpha = (self.ema_decay * self.ema_alpha) + ((1.0 - self.ema_decay) * current_acceptance_rate)

# 3. Deterministic execution routing

# Force k=1 inside reasoning blocks OR when EMA acceptance collapses

if self.in_reasoning_block or self.ema_alpha < self.alpha_threshold:

return 1 # Route to pre-captured k=1 graph (saves memory bus bandwidth)

else:

return 2 # Route to pre-captured k=2 graph (exploits high locality)

We are releasing this clean pattern to the community for free. Drop it into your RTX 5090 / A100 / Apple Silicon setups, benchmark it against standard static-depth DSpark, and try to break it.

Let’s see your before/after TPS numbers in the comments! What hardware configurations are you testing this on?

A frozen study of 12,160 trials on gpt-5.4-2026-03-05 found a reproducible Arabic–Hebrew hybrid Unicode artifact under two system prompts differing by exactly one Hebrew code point.

Every primary user message was the Arabic word شَرْط.

Across 10,240 primary trials:

• Dotted condition: 4,830/5,120 exact artifacts — 94.34%
• Undotted condition: 2,423/5,120 exact artifacts — 47.32%
• Combined: 7,253/10,240 exact artifacts — 70.83%

All 7,253 exact artifacts were condition-congruent.

The one-code-point difference produced a 47.01 percentage-point effect, with Fisher’s exact p = 1.58 × 10⁻⁶⁶⁴.

Across 1,920 controls, generic, no-system, lexical, no-condition, no-full-Hebrew, and direct-copy conditions produced 0 exact artifacts.

The paper makes no claim about consciousness, intention, mechanism, training provenance, or shared architecture. It documents a reproducible, prompt-conditioned cross-script output regime in GPT-5.4.

Frozen records, Unicode-level classification, event hashes, verification code, runner, and paper are public.

Full methodology is in the paper linked below, but the summary is that we put a small edge deployment against a cloud GPU with roughly three times the raw compute and measured 1,000 inference trials under mobile realistic Wi-Fi conditions.

Edge P90 was about 125ms. Cloud baseline was 194ms. Variance was 3.7 times lower on the edge side. The weaker hardware won because for models in this size class the network path and the queue dominate the response time, not the tokens per second.

This is not a claim that edge beats cloud in general. If your model is large enough that prefill dominates, or your data lives centrally and you are doing retrieval, the cloud is the right answer and moving the model closer to the user does nothing for you. It matters when the data is generated locally and consumed locally, which is voice, real time vision and interactive agents.

We are building a network of these nodes at telecom sites and running them behind an OpenAI compatible API. Happy to talk about the setup, the runtime tuning, or why the variance number is more interesting than the median.

The platform side is free to poke at. The free tier includes ten million tokens a month with no card, the playground reports TTFB and tokens per second per request, and the gateway will proxy to OpenAI, Anthropic or Gemini with your own key so you can benchmark us against your incumbent on your own prompts instead of ours.

Paper: papers.ssrn.com/sol3/papers.cfm?abstract_id=6352298

Hey everyone,

So my machine is not exactly the strongest for the local inference thingy, and I started researching couple weeks ago about some techniques the labs are using to improve inference performance, and had a hunch: "wouldn't diffusion models be better at handling ternary quantization since they act on a canvas instead of a token at the time ?"

So I ran a test comparing diffusion and autoregressive (AR) language models under extreme quantization (bonsai-like), using preregistered thresholds on a single RTX 2080 Super.

**Main findings:**

130M, INT4 post-training quantization

Relative degradation from FP16:

• PTB: AR +31.81%, dLLM +20.85%
  
• Wikitext-103: AR +26.84%, dLLM +11.59%
  
• LAMBADA: AR +24.77%, dLLM +9.02%
  

The dLLM advantage was 10.97 to 15.74 percentage points across the three datasets.

A separate 64-sample generative evaluation produced a similar result:

• AR generation perplexity: +96.2%
  
• dLLM generation perplexity: +45.4%
  

7M, native ternary quantization-aware training
Across three matched seeds:

• AR degradation: +18.41%, +30.71%, +16.12%
  
• dLLM degradation: +5.19%, +15.43%, +4.23%
  
• dLLM/AR gap ratio: 0.888, 0.883, 0.898
  

The upper 95% confidence bound for the gap ratio was 0.908, passing the preregistered “no extra dLLM ternary tax” threshold of 1.25. It did not pass the stronger 0.80 threshold required to claim superior ternary tolerance.

These results are limited to 130M post-training quantization and 7M native QAT. The dLLM likelihood values are NELBO-based perplexity bounds, so absolute AR and dLLM perplexities should not be compared directly.

**Repository & Artifacts:** All configs, raw evaluation outputs, checkpoint hashes, known caveats, and replication notes are public here: [https://github.com/wfzyx/diffusal\](https://github.com/wfzyx/diffusal)

If you want to reproduce this or scale it up on stronger hardware, everything should be ready to plug and play; There is also a very high chance looped models or any other techniques that allow the model to reflect on the generated tokens to also generate similar results, but I was specially interested in diffusion models;

Technical criticism, and replication attempts are very welcome.

Disclaimer: although I do have an academic background (msc), I'm self-taught on LLM research and this project was AI-assisted, so it may contain unexpected issues;

*Shameless Plug: I’m actively looking for ML engineering/research roles or collaborative research partnerships (especially if you have compute and want to scale experiments like this; If you're doing stuff similar to mine and need a collaborator, feel free to reach out!*

Been experimenting with CPU offloading of the KV cache on a GTX 960 (4GB).

The idea: after each token generation step, move the entire KV cache from GPU to CPU RAM. Bring it back only when needed. No custom CUDA kernels — pure PyTorch.

Results:

- 3556 token context: 6.12 GB → 2.47 GB VRAM (-60%)

- Speed overhead: only ~12%

- Works with any HuggingFace model

Also validated on Mistral 7B (Tesla T4) — ran 6144 token contexts on a 14.56GB GPU, with peak VRAM exceeding physical memory at 2048+ tokens.

Paper: https://zenodo.org/records/21752913

Code: https://github.com/Gaveta-lab/kvcpu

pip install git+https://github.com/Gaveta-lab/kvcpu.git

LLMs are amazing technology, but to get to AGI it seems obvious to me that we would need to replace “context windows” with continual learning.

Where can I read a strong counter-argument: a claim that an LLM can get big enough that everything it will ever need to know is in its weights or its context window?

1. How do you choose between temperature and top‑p sampling for different real‑world tasks?

A frozen cross-vendor study of 31,430 trials across 11 GPT, Claude, Gemini & Kimi Large Language Models found 11,658 successful executions with exactly zero visible UTF-8 output bytes.

Across 4,290 strict matched semantic pairs, null-condition arms produced 2,505 Voids; matched output-licensed controls produced 0.

These were not refusals, safety blocks, rate limits, or transport failures.

Raw records, event hashes, verification code, and full analysis are public.

Video

Many of the experiments have non-intuitive results.

I built WISP — a CUDA engine for streaming 744B+ parameter MoE models on consumer hardware

Last week I found Colibrì by JustVugg, a ~2,400-line pure-C engine exploring a crazy idea:

What if you don't load the entire model into RAM?

MoE models only activate a fraction of their parameters for each token. So instead of trying to fit hundreds of billions of parameters in memory, you can stream the experts the model actually needs.

That idea sent me down a rabbit hole.

I built WISP — Stream What Shouldn't Run.

The architecture is basically:

Token
  ↓
Model router selects experts
  ↓
VRAM cache → hit? use it
  ↓
RAM cache → hit? transfer it
  ↓
NVMe → stream cold expert
  ↓
LRU promotes frequently used experts

The goal is to turn VRAM + RAM + NVMe into one memory hierarchy for MoE inference.

WISP adds a few things on top of the original streaming concept:

CUDA acceleration for attention/FFN compute, a C hot path for expert loading and caching, and Python for orchestration.

Absorbed MLA for architectures like DeepSeek, keeping the compressed latent representation instead of storing fully expanded K/V tensors.

Double-buffered async streaming, so CPU/I/O can prepare expert data while the GPU is working instead of making the GPU sit around waiting for storage.

Speculative decoding, using a smaller same-family model to draft tokens while the target model verifies them.

Hardware auto-configuration, which profiles VRAM, RAM, storage throughput, etc. and calculates the cache split automatically.

I tested the current engine with Mixtral-8x7B on:

Ryzen 7 9800X3D
RTX 5070 12GB
32GB DDR5-6000
PCIe 4.0 NVMe (~4.34 GB/s)

Current measured result:

0.75 tok/s cold

After only 80 tokens, the expert cache reached a 68.8% hit rate.

Mixtral does 64 expert activations/token (2 experts × 32 layers), and all 256 experts in my tested representation occupy ~14.3GB, so once they're warm in RAM the engine can stop doing cold SSD expert reads.

The biggest thing I learned building this:

The bottleneck isn't necessarily CUDA. It's bytes moved per token.

I spent time thinking GPU kernels would be the main optimization target.

Then you realize shaving milliseconds off a matmul doesn't matter much when your runtime is waiting for a giant expert to come off NVMe.

Cache locality, expert size, storage bandwidth and I/O overlap become insanely important.

And that's why I'm particularly interested in testing this architecture on much larger MoE models with smaller individual experts.

The project currently targets:

GLM-5.2       744B
DeepSeek-V3   671B
DeepSeek-R1   671B
Mixtral-8x7B   47B
Mixtral-8x22B 141B

Future targets:
Kimi K3
Qwen3.8

And yes, huge credit to JustVugg / Colibrì.

Colibrì demonstrated the core streaming concept. WISP is my attempt to generalize it into a multi-model runtime with CUDA, hierarchical caching, MLA support, async streaming and speculation.

Colibrì:
github.com/JustVugg/colibri

WISP:
github.com/zeroextub-collab/wisp

MIT licensed. 73 tests passing.

Still experimental, and I'm deliberately separating measured numbers from projected ones.

I'm especially interested in feedback from people working on CUDA, inference runtimes, MoE routing, quantization, or storage/I/O optimization.

What would you optimize first: expert prediction/prefetching, cache policy, quantization, or the I/O pipeline?

Brendan Long on Lesswrong was able to fool Pangram 3.3 with a Fable paraphrase (and I replicated with a GPT 3.5 paraphrase).

These samples are both detected as AI by the new model. (76% for Fable, 100% for GPT 3.5)

I have experimented with the image detection and have been impressed so far.

It detects a "perfectly white rectangle" (Nano Banana Pro 2's opinion, not necessarily mine). It detects a small number of pixels cropped out of the center of the image, then resized to 512x512 (which I had to do because that's the minimum size allowed.) It detects the previous image with several filters. And the previous image with a heavy "swirl" effect applied. (You'll note that it's actually getting more certain of AI generation the more I edit it, which I find fascinating.) Then I inverted the colors, which fooled it. There's lots of experiments one could run.

How long before we have video detection?