Euler-Flow Activations in Neural ODEs

Abstract

This thesis investigates whether using velocity fields derived from incompressible Euler equations as activation functions in residual neural networks offers advantages over using ReLUs for spatially structured classification tasks. The motivation for this study comes from two specific theoretical observations. Firstly, recent work has shown that approximating measure-preserving maps with ReLU perceptrons requires a large number of compositions that grows with the desired precision. Secondly, Arnold’s theorem identifies the geodesics of the group of volume-preserving diffeomorphisms as flows of the incompressible Euler equations. To test this hypothesis, three explicit families of solutions to the Euler equations in two dimensions are compared against ReLU and Siren on three two-dimensional classification tasks, with the number of parameters matched across all architectures. The results support the hypothesis for the main architecture tested: every non-ReLU activation outperforms ReLU by between ten and twenty-five percentage points across all tasks and seeds. However, further work is required to extend this result to deeper and more complex architectures.