Emergent Abilities

Emergent Abilities, in the context of Large Language Models, are abilities that are "not present in smaller models but [are] present in larger models" (Wei et al., 2022). Performance stays near random as models get bigger, then jumps once they pass some scale. You can't predict it by extrapolating from the smaller models.

The term comes from the idea of Emergence more generally. Wei et al. root it in physicist Philip Anderson's 1972 essay "More Is Different" (Anderson, 1972), summarising it as: "Emergence is when quantitative changes in a system result in qualitative changes in behavior". In LLMs, the quantity is scale: training compute and parameter count.

Chain-of-Thought Prompting is one of their main examples. It only beats standard prompting at around 100B parameters (Wei et al., 2022). Scratchpads and Self-Consistency decoding are also on their list.

However, not everyone agrees that these jumps are real. Schaeffer et al. argue that emergent abilities "appear due to the researcher's choice of metric rather than due to fundamental changes in model behavior with scale" (Schaeffer et al., 2023). All-or-nothing metrics like exact match make performance look like it jumps suddenly. Measure the same models with a smoother metric and the improvement is gradual and predictable.