Gated Recurrent Neural Networks

Gated Recurrent Neural Networks are Recurrent Neural Networks that use learned gates to control how information flows through the hidden state: what to keep, what to forget and what to output at each step.

The two best-known examples are the LSTM and the Gated Recurrent Unit (GRU). The gates help with the vanishing gradient problem that makes plain RNNs struggle to learn long-range dependencies.

Before Attention Is All You Need, gated RNNs were the go-to models for sequence modelling tasks like Machine Translation. See also Were RNNs All We Needed?.