I have been working for a while on the most tractable Recurrent Neural Net that I’ve been able to devise: a convolutional unitary RNN (cuRNN), which has a convolution kernel with a unitary structure, i.e. all of whose eigenvalues have absolute value equal to 1.
Three manuscripts on this subject have been uploaded to arXiv:
(1) Convolutional unitary or orthogonal recurrent neural networks .arXiv:2302.07396
(2) On the dynamics of convolutional recurrent neural networks near their critical point arXiv:2405.13854
(3) Input-driven circuit reconfiguration in critical recurrent neural networks arXiv:2405.15036 . I have placed the Supplementary Movies in this page.
I will use a single-layer recurrent convolutional neural network withunitary (critical) coupling kernel. The family of such networks was derived as a Poincaré map from critical ODEs, and their basic dynamical analysis is outlined in (1) and (2). The state is given by a complex-valued neural layer , henceforth a lattice of complex numbers in either 1 or 2 dimensions with periodic boundary conditions. Its evolution in time is denoted as where is (integer) time, through the recursion
where is the convolution operation natural to ; is a (unitary) convolution kernel; is a sequence of external inputs of the same shape as ; and is a smooth, scalar
activation function operating element-wise on
A natural choice of for this construction is , a complex-valued phase-preserving sigmoid (2). We choose to be a \emph{unitary} kernel, i.e. its associated linear operator’s eigenvalues lie on the unit circle: . We generate unitary convolution kernels by convolutional exponentiation of an anti-Hermitic kernel
where the exponential is element-wise, and the direct and inverse Fourier transforms in the dimension appropriate to and , and , i.e. antiHermitic.