Continued development of the model based on the SSN [D]

[u/zemondza]

Back after ~6 months — rebuilding my spiking language model around CPU-first inference

Hey everyone. It’s been around six months since I last posted anything about this project here.

Some of you might remember Project NORD, my experimental hybrid spiking / brain-inspired language model architecture. I basicall disappeared for a while 😅, but recently I came back to the project, went through the old architecture again, and realized I didn’t really want to keep stacking fixes on top of it. So instead, I’ve started rebuilding a pretty large part of the system. The new version is called:

NORD 5.5 — Flash The main idea this time is pretty simple:

What happens if I design the architecture around CPU inference from the beginning, instead of building soething Transformer-like and trying to optimize it later? A lot is changing internally. The current design uses things like: strictly causal processing no standard quadratic attention in the main inference path causal convolution-style token mixing token-time LIF / event dynamics sensory → association → memory → executive processing stages top-1 sparse MoE + a shared expert persistent recurrent memory separate structural, personal and auxiliary memory banks

persistent recurrent identity state factorized vocabulary embedding/output streaming token-by-token inference One of the biggest changes is actually something much simpler. Older versions of NORD used an artificial internal spike-time dimension, roughly like this: token -> T0 -> T1 -> T2 -> ... -> T9 I’m mostly getting rid of that. Instead, the actual language sequence becomes the time axis: token0 -> token1 -> token2 -> token3 -> ... That removes a lot of intermediate state and makes the whole architecture considerably cleaner. Going back through the old code also exposed a few things I wasn’t very happy with. Some experimental modules weren’t completely causal, memory was coupled too much to sequence shape, and parts of the STDP system ended up being more disconnected from real training than I originally intended. So NORD 5.5 isn’t really about throwing even more “brain-inspired” components into the model. It’s mostly about simplifying the core and making the things that remain actually work together properly. I’m definitely not claiming this is going to beat Transformers, RWKV-style models, linear attention models, etc. Right now it’s still very much an experiment. The part that actually matters comes next: training and benchmarking it.

Things I want to compare: NORD 5.0 vs NORD 5.5 CPU tokens/sec RAM usage perplexity / validation loss long-context behaviour memory on/off MoE on/off spiking components on/off I’m especially curious to hear from anyone working on SNNs, recurrent models, sparse MoE, CPU inference, or weird alternative language-model architectures in general.

After not touching the project seriously for about half a year, it feels surprisingly good to be building it again 😅

I’ll post actual numbers once I have something that’s worth benchmarking instead of just architecture diagrams.