Tokenisation

Tokenisation is the process of splitting input into a sequence of smaller units called tokens, which are then mapped to integer IDs that a model can process. It's usually the first step in Text Preprocessing.

For text, tokens can be whole words (see Word Tokenisation), individual characters, or subword units. Modern language models mostly use subword tokenisers like Byte Pair Encoding, which keep common words as single tokens and split rare words into pieces.

The same idea applies to other modalities: see Image Tokenisation and Audio Tokenisation.