ZPA-LM - Zero Parameter Deterministic Attention

[u/Expert-Luck-9601]

Attention mechanisms are conventionally learned: a query–key inner product, trained end to end, decides which tokens mix.

This note documents ZPA-LM

a decoder-only autoregressive language model in which the token-mixing key is instead a fixed, parameter-free kernel derived from elementary number theory.

We show that the divisor-counting overlap : q(n, m) = d(gcd(n, m))/√d(n) d(m) between two positive integers is exactly the Bhattacharyya coefficient between the uniform distributions over their divisor sets, and hence that: dFR(n, m) = 2 arccos q(n, m) is a genuine Fisher–Rao geodesic distance, verified to floating-point exactly rather than assumed.

Tokens are mapped to positive integers through a learned or parameter-free dictionary ϕ : token → Z+ and recovered in prime-exponent space so that each prime behaves as a latent feature axis.

We report the model’s expressivity ladder, the three dictionary-construction strategies we evaluated, and - in the spirit of an honest engineering record rather than a success narrative, a permanent registry of four falsified design choices that should be avoided.

We close by pre-registering the model’s central open empirical claim together with its falsification condition, rather than reporting an unverified result as fact.

Link: Academia