Neural network activation functions

There is a whole cottage industry in showing neural networks are reasonably universal function approximators with various nonlinearities as activations, under various conditions. In practice you can take this as a given. Nonetheless, you might like to play with the precise form of the nonlinearities, even making them themselves directly learnable, because some function shapes might have better approximation properties with respect to various assumptions on the learning problems, in a sense which I will not attempt to make rigorous now, vague hand-waving arguments being the whole point of deep learning.

I think a large part of this field has been subsumed into the stability-of-dynamical-systems setting?

Nonetheless, here are a some handy references.

The current default activation function is ReLU, i.e. \(x\mapsto \max\{0,x\}\), which has many nice properties. However, it does lead to piecewise linear spline approximators which makes it hard to solve differential equations. Other classic approximators such as \(x\mapsto\tanh x\) have fallen from favour. Sitzmann et al. (2020) argues for \(x\mapsto\sin x\) which has some handy properties but requires good initialisation.

The virtues of differentiable activation functions in Implicit Neural Representations with Periodic Activation Functions.