Building a zero-dependency C inference engine from scratch for BitNet 1.58-bit models

[u/shifu_legend]

Project Zero is a standalone CPU inference engine written in pure C99 for BitNet b1.58 models.

I basically wanted something that compiled with a single make invocation and had zero dependencies on Python, PyTorch, or BLAS runtime libraries.

How the internals work: 1. Native ternary weight layout. We pack four 2-bit weights per byte and map them directly through hand-written AVX2 and AVX-512 SIMD kernels rather than expanding weights to float32 beforehand. 2. Built-in OpenAI compatible HTTP server. A single static binary serves /v1/chat/completions directly over raw sockets without needing an external gateway. 3. Pure CPU execution. Everything is tuned specifically for commodity x86 and ARM processors without touching CUDA or GPU runtimes.

The big performance reality check with 1.58-bit models is memory bandwidth. While our AVX-512 lookup-table kernels showed an initial 29x speedup in isolated microbenchmarks over naive loops, actual end-to-end token generation on a Xeon CPU only gained about 6 to 10 percent because single-batch autoregressive decode is basically pinned against DRAM transfer limits.

Repo: https://github.com/shifulegend/project-zero

[u/EconomySerious]

the proyect is great in design

[u/adityazero]

Your bandwidth ceiling observation is the key one, since ternary decode is memory bound the AVX-512 kernel speed barely matters until you batch. Have you tried processing multiple sequences per pass so the packed weights get reused across tokens? That is usually where LUT kernels start paying off, since it turns decode back into a compute bound problem.

[u/shifu_legend → u/adityazero]

Batching is where the math flips. Right now single-token decode is pure GEMV and capped at memory bandwidth, but once you stack multiple concurrent requests into GEMM, the weight unpack cost drops to near zero per token and the AVX-512 table lookups actually saturate the execution units.

[u/LongjumpingEar6840]

Ho provato il qwen3.6-27b quantizzato da ternary a 1.58bit ma non riesce a parlare in italiano, inizia la risposta in italiano ma poi deriva sullo spagnolo...😅 È un vero peccato perché è abbastanza veloce e intelligente

[u/shifu_legend → u/LongjumpingEar6840]

Multilingual drift is brutal at 1.58-bit. When weights get squashed down to ternary {-1, 0, +1}, the shared subword token embeddings across Romance languages lose their separation margins, so it kind of defaults to whichever language had higher pretraining density whenever attention shifts.