Continual Pre-Training (CPT)

Continual Pre-training is when you take an already pre-trained model and keep training it on more raw text, usually to adapt it to a new domain, language, company data, or newer information. It's really just another name for Fine-Tuning, but specifically applied to the Pre-training step within the typical LLM Training Pipeline