Notes by Lex Toumbourou

Snake is a neural network activation function useful for modelling problems with a "periodic induction bias" - in other words, problems with regular, repeating patterns - for example, time-series data, audio signals and so on. It was described in the paper Neural networks fail to learn periodic functions and how to fix it.

In the DAC paper, High-Fidelity Audio Compression with Improved RVQGAN, they replace Leaky ReLU with Snake and significantly improve the reconstructed audio quality.

$\text{snake}(x) = x + \frac{1}{\alpha} \sin^2(\alpha x)$

The $\alpha$ is a parameter that controls the frequency of the periodic component of the signal.

In code:

import numpy as np

def snake_activation(x, alpha):
    return x + (1 / alpha) * np.sin(alpha * x) ** 2

Plot: