Notes by Lex Toumbourou

Entropy is a measure of uncertainty of a random variable's possible outcomes.

It's highest when there are many equally likely outcomes. As you introduce more predictability (one of the possible values of a variable has a higher probability), Entropy decreases.

It measures how many "questions" on average you need to guess a value from the distribution. Since you'd start by asking the question that is most likely to get the correct answer, distributions with low Entropy would require smaller message sizes on average to send.

The entropy of a variable from distribution $p$ is expressed as: $H=\sum\limits_{i=1}^{n} p_{i} \times log_2(\frac{1}{p})$

The expression is commonly inverted and rewritten like this: $H=-\sum\limits_{i=1}^{n} p_{i} \times log_2(p_i)$

When using log base 2, the unit of Entropy is a bit (a yes or no question).

In code:

import numpy as np

def entropy(dist):
    return -np.sum(np.array(dist) * np.log2(dist))

dist = [1/2.]*2
dist

[0.5, 0.5]

entropy(dist)

1.0

dist = [1.]
dist

[1.0]

entropy(dist)

-0.0

dist = [1/100.]*100
dist[:5]

[0.01, 0.01, 0.01, 0.01, 0.01]

entropy(dist)

6.643856189774724

dist = [99/100] + [1/100.]
dist

[0.99, 0.01]

entropy(dist)

0.08079313589591118

Claude Shannon borrowed the term Entropy from thermodynamics as part of his theory of communication.

KhanAcademyLabs (2014)

Cover from How Claude Shannon Invented the Future.

Backlinks

• Cross-Entropy