Recurrent Neural Networks

Recurrent Neural Networks (RNNs) are neural networks for sequential data that process one step at a time, passing a hidden state from each step to the next, so the network has a memory of what it's seen so far.

They're trained with Backpropagation through time, and plain RNNs struggle with long sequences because of vanishing gradients. Gated variants like the LSTM and Gated Recurrent Unit help with this.

Because each step depends on the previous one, RNNs are hard to parallelise, which is one reason the Transformer replaced them for many tasks. See also Were RNNs All We Needed?