Stabilization of recurrent neural networks through divisive normalization

Flaviano Morone, Shivang Rawat, David J. Heeger, Stefano Martiniani

ABSTRACT Stability is a fundamental requirement for both biological and engineered neural circuits, yet it is surprisingly difficult to guarantee in the presence of recurrent interactions. Standard linear dynamical models of recurrent networks are sensitive to the precise values of the synaptic weights, since stability requires all eigenvalues of the recurrent matrix to lie within the unit circle. Here we demonstrate, both theoretically and numerically, that an arbitrary recurrent neural network can remain stable even when its spectral radius exceeds 1, provided it incorporates divisive normalization, a dynamical neural operation that suppresses the responses of individual neurons. Sufficiently strong recurrent weights lead to instability, but the approach to the unstable phase is preceded by a regime of critical slowing down, a well-known early warning signal for loss of stability. Remarkably, the onset of critical slowing down coincides with the breakdown of normalization, which we predict analytically as a function of the synaptic strength and the magnitude of the external input. Our findings suggest that the widespread implementation of normalization across neural systems may derive not only from its computational role, but also to enhance dynamical stability. I. SIGNIFICANCE STATEMENT Neural circuits must remain stable to function correctly, yet strong recurrent connections, which are essential for complex computations, often hinder this stability. We demonstrate that divisive normalization, a canonical neural operation found across many sensory systems, can actively stabilize neural networks. By analyzing a biologically plausible model (ORGaNICs), we find that normalization allows circuits to remain stable even when recurrent interactions are strong enough to otherwise cause exploding neural dynamics. Furthermore, we identify a theoretical link between the breakdown of normalization and critical slowing down, a state where the brain recovers slowly from perturbations. This suggests that loss of normalization may serve as an early warning signal for the onset of pathological instabilities, such as seizures.