Audio Tokenisation

Audio Tokenisation is the process of converting audio into a sequence of discrete tokens, so that language modelling techniques can be applied to audio. See Tokenisation.

AudioLM: a Language Modeling Approach to Audio Generation describes two common kinds of audio tokens:

• Acoustic Tokens, produced by a neural audio codec like SoundStream or Encodec. They capture fine acoustic detail and allow high-quality reconstruction.
• Semantic tokens, produced by clustering the embeddings of a self-supervised model like w2v-BERT or HuBERT with K-Means. They capture long-term structure like phonetic content but reconstruct poorly.