Categorical Cross-Entropy Loss
Categorical Cross-Entropy Loss Function, also known as Softmax Loss, is a loss function used in multiclass classification model training. It applies the Softmax Activation Function to a model's output (logits) before applying the Negative Log-Likelihood function.
Lower loss means closer to the ground truth.
In math, expressed as
where is the number of classes, is the ground truth labels, and is the model outputs. Since is one-hot encoded, the labels that don't correspond to the ground truth will be multiplied by 0, so we effectively take the log of only the prediction for the true label.
In the PyTorch implementation, the index of the ground truth label is passed instead of one-hot encoded $Y$ vector.
import torch from torch import nn, tensor torch.set_printoptions(sci_mode=False) dog_class_index = 0 label = tensor([dog_class_index]) logits = tensor([[3.5, -3.45, 0.23]]) nn.CrossEntropyLoss()(logits, label)
We can also achieve the same result by manually calling softmax and negative-log loss.
softmax_probs = nn.Softmax(dim=1)(logits) softmax_probs
tensor([[ 0.9625, 0.0009, 0.0366]])
Based on Cross-Entropy in Information Theory, which is a measure of difference between 2 probability distributions.
Howard et al. (2020) (pg. 222-231)
Jeremy Howard, Sylvain Gugger, and Soumith Chintala. Deep Learning for Coders with Fastai and PyTorch: AI Applications without a PhD. O'Reilly Media, Inc., Sebastopol, California, 2020. ISBN 978-1-4920-4552-6. ↩