Improving Language Understanding by Generative Pre-Training

Improving Language Understanding by Generative Pre-Training is the 2018 OpenAI paper by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever, which introduced GPT.

The approach has two stages:

1. Generative pre-training of a Transformer decoder as a Language Model on a large corpus of unlabelled text (BooksCorpus).
2. Discriminative Fine-Tuning on each labelled downstream task, with minimal changes to the model architecture.

This approach improved the state of the art on 9 of the 12 tasks studied. It was followed by GPT-2 in Language Models are Unsupervised Multitask Learners.