Notes by Lex Toumbourou

Cross-entropy measures the average number of bits required to identify an event if you had a coding scheme optimised for one probability distribution $q$, where the true probability distribution was actually $p$.

It's the same as Information Entropy (Information Theory) but measuring what happens if you have are identifying messages using a different probability distribution.

Expressed as: $H(p, q)=-\sum\limits_{i=1}^{n} p_{i} \times log_2(q_{i})$

[@CrossEntropy2021]

Backlinks

• Categorical Cross-Entropy Loss