Notes by Lex Toumbourou

Categorical Cross-Entropy Loss Function, also known as Softmax Loss, is a loss function used in multiclass classification model training. It applies the Softmax Activation Function to a model's output (logits) before applying the Negative Log-Likelihood function.

Lower loss means closer to the ground truth.

In math, expressed as

$P = \text{softmax}(O)$

$-\sum\limits_{i=1}^{N} Y_{i} \times \log(P_{i})$

where $N$ is the number of classes, $Y$ is the ground truth labels, and $O$ is the model outputs. Since $Y$ is one-hot encoded, the labels that don't correspond to the ground truth will be multiplied by 0, so we effectively take the log of only the prediction for the true label.

import torch
from torch import nn, tensor

torch.set_printoptions(sci_mode=False)

dog_class_index = 0

label = tensor([dog_class_index])
logits = tensor([[3.5, -3.45, 0.23]])

nn.CrossEntropyLoss()(logits, label)

tensor(0.0382)

softmax_probs = nn.Softmax(dim=1)(logits)
softmax_probs

tensor([[    0.9625,     0.0009,     0.0366]])

-torch.log(softmax_probs[:,label])

tensor([[0.0382]])

Based on Cross-Entropy in Information Theory, which is a measure of difference between 2 probability distributions.

Howard et al. (2020) (pg. 222-231)

Backlinks

• A Discriminative Feature Learning Approach for Deep Face Recognition
• Binary Cross-Entropy Loss
• Softmax Activation Function