Implicit Neural Representations with Periodic Activation Functions (2020)

>"We propose SIREN, a simple neural network architecture for implicit neural representations that uses the sine as a periodic activation function: [...]

Interestingly, any derivative of a SIREN is itself a SIREN, as the derivative of the sine is a cosine, i.e., a phase-shifted sine (see supplemental).

Therefore, the derivatives of a SIREN inherit the properties of SIRENs, enabling us to supervise any derivative of SIREN with “complicated” signals. In our experiments, we demonstrate that when a SIREN is supervised using a constraint Cm involving the derivatives of φ, the function φ remains well behaved, which is crucial in solving many problems, including boundary value problems (BVPs).

We will show that SIRENs can be initialized with some control over the distribution of activations, allowing us to create deep architectures.

Furthermore, SIRENs converge significantly faster than baseline architectures, fitting, for instance, a single image in a few hundred iterations, taking a few seconds on a modern GPU, while featuring higher image fidelity..."

SIRENs look interesting... the idea of using Sine as an Activation Function seems like a brilliant one!