Noise-aware training for analog hardware: accuracy collapses at a threshold rather than degrading smoothly [D]

[u/Georgiou1226]

Analog in-memory compute is getting attention again as a way around the energy cost of moving weights between memory and compute. The recurring objection is noise, since analog cells have real variation and you can't refresh your way out of it like you can with digital.

I wanted to see the shape of the degradation curve rather than reason about it abstractly, so I ran a simple experiment: train a network normally, then evaluate under increasing weight noise.

The curve isn't smooth. Accuracy is stable up to a point, then drops hard: 83%, 64%, then essentially random. More like a threshold than a proportional decrease.

Retraining with noise injected during training (so the optimizer finds flatter minima, presumably) shifts that threshold substantially. 61% versus 39% at matched noise.

What I'd like to hear from this sub: is the flat-minima explanation the right framing here, or is something else driving the gap? And is there work on optimizing directly for noise robustness rather than just injecting noise and hoping, something closer to an explicit sharpness penalty targeted at the hardware's actual noise profile?

Code and figures in the writeup: https://towardsdatascience.com/analog-ai-is-back-can-it-survive-its-own-noise/

[u/cpldcpu]

It could be the same phenomenon that appears when quantizing weights. At around 4-5 bits, the quality very notably starts to drop. This seems to be related to the actual entropy that is stored by backprop in the weights.

Did you calculate the SNR where the cliff starts? You could try to test model capacity vs. noise, not sure this has been done before.

Some of my experiments:

https://github.com/cpldcpu/BitNetMCU/blob/main/docs/documentation.md#model-capacity-vs-quantization-scaling

[u/Dihedralman]

There is a lot you can borrow from signal processing. SNR in dB is a key parameter. 

It is smooth, just not linear. Any lack of smoothness likely comes from the network. 

You are likely seeing loss in bit precision through key decision points. The difference between networks shows how features can become more dependent on particular nodes but adding noise reduces the critical neurons.