LLM Training Pipeline

Photo by Taylor Vick on Unsplash

Large Language Models training is typically structured into two phases: pre-training and post-training (Raschka, 2026).

Pre-training

In the Pre-training step, LLMs are trained on giant collections of data, often gathered from the internet, books and other sources. These collections can reach terabytes or petabytes, depending on the training run. Autoregressive LLMs are typically trained with the next-token-prediction objective. To get good at doing that, the model needs to learn patterns in language and the relationships represented in the dataset, developing a sort of "understanding" of it. Large pre-training runs can take weeks to months and are typically extremely expensive (Raschka, 2026).

Post-training

Once we have a pre-trained LLM, we fine-tune it for a task, like responding to chat messages. Techniques include Supervised Fine-Tuning, including instruction tuning and preference tuning to teach it to respond to user queries in ways people prefer. Instruction tuning is a form of supervised fine-tuning, rather than a name for all supervised fine-tuning (Raschka, 2026) (Ouyang et al., 2022).

Reinforcement Learning from Human Feedback (RLHF) is a form of preference tuning. Direct Preference Optimisation (DPO) is another approach: it learns directly from preferred and rejected responses without a separate reward-model training stage or an RL rollout loop (Ouyang et al., 2022) (Rafailov et al., 2023).