Pretraining

Pre-training is the first phase of the typical LLM Training Pipeline. LLMs are trained on giant collections of data from the internet, books and other sources. Autoregressive LLMs are typically trained with the next-token-prediction objective, learning language patterns and relationships represented in the dataset (Raschka, 2026).

Large pre-training runs can take weeks to months and are typically extremely expensive. The result is a pre-trained, or base, model, which can then be adapted through post-training, including Supervised Fine-Tuning and preference tuning (Raschka, 2026).