500 IGPM500.pdf        January 2020
TITLE Kinetic Theory for Residual Neural Networks
AUTHORS Michael Herty, Torsten Trimborn, Giuseppe Visconti
ABSTRACT Deep residual neural networks (ResNet) are performing very well for many data science applications. We use kinetic theory to improve understanding and existing methods. A microscopic simplified residual neural network (SimResNet) model is studied as the limit of infinitely many inputs. This leads to kinetic formulations of the SimResNet and we analyze those with respect to sensitivities and steady states. Aggregation phenomena in the case of a linear activation function are also studied. In addition the analysis is validated by numerics. In particular, results on a clustering and regression problem are presented.
KEYWORDS Keywords Residual neural network, continuous limit, mean field equation, kinetic equation, machine learning application