Recurrent Neural Networks

I have been working for a while on the most tractable Recurrent Neural Net that I’ve been able to devise: a convolutional unitary RNN (cuRNN), which has a convolution kernel with a unitary structure, i.e. all of whose eigenvalues have absolute value equal to 1.

Three manuscripts on this subject have been uploaded to arXiv:

(1) Convolutional unitary or orthogonal recurrent neural networks .arXiv:2302.07396

(2) On the dynamics of convolutional recurrent neural networks near their critical point arXiv:2405.13854

(3) Input-driven circuit reconfiguration in critical recurrent neural networks arXiv:2405.15036 . I have placed the Supplementary Movies in this page.

I will use a single-layer recurrent convolutional neural network withunitary (critical) coupling kernel. The family of such networks was derived as a Poincaré map from critical ODEs, and their basic dynamical analysis is outlined in (1) and (2). The state is given by a complex-valued neural layer $Z$ , henceforth a lattice of complex numbers in either 1 or 2 dimensions with periodic boundary conditions. Its evolution in time is denoted as $Z_{n}$ where $n$ is (integer) time, through the recursion

$Z_{n+1}=\phi\left(U\otimes Z_{n}+I_{n}\right)$

where $\otimes$ is the convolution operation natural to $Z$ ; $U$ is a (unitary) convolution kernel; $I_{n}$ is a sequence of external inputs of the same shape as $Z$ ; and $\phi$ is a smooth, scalar activation function operating element-wise on $Z.$

A natural choice of $\phi$ for this construction is $\phi(z)=z/\sqrt{1+|z|^{2}}$ , a complex-valued phase-preserving sigmoid (2). We choose $U$ to be a \emph{unitary} kernel, i.e. its associated linear operator’s eigenvalues $\lambda$ lie on the unit circle: $|\lambda|=1$ . We generate unitary convolution kernels by convolutional exponentiation of an anti-Hermitic kernel

$U=e_{\otimes}^{A}\equiv\mathscr{F}^{-1}\left{ \exp\left(\mathscr{F}\left{ A\right} \right)\right}$

where the exponential is element-wise, $\mathscr{F}$ and $\mathscr{F}^{-1}$ the direct and inverse Fourier transforms in the dimension $D$ appropriate to $Z$ and $K$ , and $A^{\dagger}=-A$ , i.e. antiHermitic.