fastai - Lesson 4 - Deep Learning for Coders (2020)

00:00:00 - Lesson 3 review

• Lesson 4 is going to take a deeper dive into training neural networks.
• Original MNIST implementation wasn't based on Arthur Samuel's idea to "altering the weight assignment to maximise performance"
• Stochasic Gradient Descent (SGD) review
  • Seven steps:
    • Init weights
    • Make prediction
    • Calculate loss
    • Calculate gradients
    • Take a step in the direction of the gradient
    • Stop when good enough

00:03:16 - MNIST loss function

• Concat xs into a single tensor and change from list of matrices (rank 3 tensor) to list of vectors (rank 2 tensor)
  • view method lets you change shape of tensor.
    • -1 is special param: make axis as big as needed to fill data.

• Need a label for each 1 for threes and 0 for sevens:
  • Adds an additional unit dimension to the position passed.

• A [[PyTorch Dataset]] can be indexed into: [i] which should return a tuple.

• Create out validation set:

00:07:46 - Initialise params

• Need to set initially random weight for every pixel:
  • It has a variance of 1, so we can multiply by variance.

• Weights are 28*28, 1 (a flat vector of pixels)

00:09:39 - Weights and biases

• weights*pixels isn't flexible enough, it is always equal to zero when pixels are zero (ie intercept is zeo)
  • Need the bias term in slope: y=w*x+b
  • Init to random number here:

• Weights and bias make up the parameters of a model
• Can calculate prediction for one image:
  • Need to first transpose them to line them up:

00:10:52 - calculate predictions with matrix multiplication

• Have to run that for every image.
  • Could run a for loop, but it would be slow and couldn't use GPU.
  • Doing that operation for multiple rows and cols is a matrix multiplication. In Python, matrix multiplication uses @ symbol

• That is one of the [[Two fundamental equations of a neural network]]. Activation function is the other.

00:14:28 - Why accuracy isn't a loss function

• Check accuuracy by checking if it's greater than 0 using broadcasting.

• item() returns a normal Python scalar, not a tensor.

• See what happens to accuracy with small change in weights:

• Since gradient is equal to rise over run: how much value of function goes up or down when divided by change in input.
  • Can write in maths: (y_new - y_old) / (x_new - x_old)
  • Accuracy only changes when a prediction goes from 3 to 7.
  • Small change in weights from x_old to x_new isn't likely to change prediction, so gradient is always 0.
• If gradient is 0, then step will be 0.

00:18:40 - A better loss function

• New function should give a better value when predictions are closer to ground truth, but never with zero gradient.

• Using where function:
  • For each value, if it matches the first condition, then set the 2nd argument, else 3rd.
  • Works on tensors and runs at C/Cuda speed.
• Can then take the mean to calculate results.

• If you change one of the predictions, you can see the loss improving:

00:22:46 - Sigmoid

• The last problem with mnist_loss is it assumes preds are between zero and one.
  • To ensure that use [[Sigmoid]].
    • Take big numbers and turn into something between 0 and 1.

• torch.exp(n) is math.e^n

• The definition of the function isn't that useful. The shape is useful.

• Torch already has a sigmoid function.

• Can change mnist_loss to be the same but changes to sigmoid first:

• Now we have a loss function with no zero gradients and it's between 0 and 1.

00:27:12 - Updating parameters and mini-batches

• How you would update parameters without linear algebra:
  • Loop through each image.
    • Make a prediction.
    • Calculate loss.
    • Do step.
    • Update parameters.
  • Will be really slow - a single epoch woud take a while.
• Could go faster by doing every image in the dataset with one big matrix multiplication.
  • A lot of work to update weights once.
• Comprise: do a few at a time: a [[Mini-batch]]
  • The bigger the batch size, the more "accurate" your gradient is.
  • The smaller the batch size, the more you can updated your parameters.

00:30:50

• PyTorch have a class called DataLoader that can give you data in a mini-batch.
  • Takes any Python collection and turns into iterator.

• When training model, you want collection containing independant and dependent variables (input and targets).
  • A collection that has tuples of independent and dependent variables is known as Dataset in PyTorch.

• Pass that to dataloader, see it returns tuples:

00:33:18 - Putting it all together

• The final product looks something like this:

for x, y, in dl:
    pred = model(x)
    loss = loss_func(pred, y)
    loss.backward()
    parameters -= parameters.grad * lr

• DataLoader can be created from Dataset:

• Test model with a small batch

• Create a mini-batch of size 4:

• Calculate the gradients:

• Can put into a function:

• and test it:

00:37:02 - Zeroing gradients

• If you call it twice, you get a diff value each time, which shouldn't happen with the same gradients:

• That's because backward() doesn't just calculate gradients. It adds them to stored gradients.
  • Set to zero first:
    • Note zero_ sets to matrix of 0s inplace.

00:38:05 - Train a single epoch

• Note the use of .data to avoid calculating gradient during gradient descent step

00:39:31 - SGD vs Gradient Descent

• Gradient descent doesn't have the loop part: it calculates all in one go (and is rarely used in practice):

  def train_epoch(model, lr, params):
    params = calc_grad(inputs, labels, model)
    for p in params:
        p.data -= p.grad * lr
        p.grad.zero_()
  

00:40:15 - New accuracy metric

• Since we are applying the [[Sigmoid Function]] to the predictions, the 0 position now becomes 0.5, so we decide it's a 3 if greater than 0.5

• Now a function to do for all batches in the validation set.

• Since the model hasn't been trained, a 50% accuracy is what we'd expect for picking values at random.
• Let's train for one epoch:

• And more:

• Which is about the same accuracy as pixel similarity approach.

00:43:31 - Refactoring and optimisers

• We can simplify the linear model using the PyTorch class nn.Linear which stores the weights abd bias in an object:

• You can access the parameters of the model as follows:

• Can use that information to create an optimiser:

• Can create optimiser by passing in model params:

• Now to simplify training loop:

• fastai provides the SGD class which does the same thing as BasicOptim

00:48:50 - DataLoaders

• DataLoaders a tiny class that stores dataloaders.
  • Does things like shuffling training dataloader, and not shuffling validation dataloader.

00:49:50 - Learning class

• Takes DataLoaders, model, optim function, loss function and metrics: does all the stuff we just did manually.

• Fit then performs train epochs:

00:51:09 - Adding non-linearity

• What we've done so far is created a linear function. That's not a neural network.
  • A neural network requires a non-linearity to handle more complex task.
• One thing we can add is a function that replaces negative numbers with 0.

• w1 and w2 are weight tensors, and b1 and b2 are bias tensors; parameters initially randomly initialised like the previous section.
  • Since w1 has 30 output activations, w2 must have 30 input activations.

• The res.max(tensor(0.0)) is called a [[Rectified Linear Unit [[ReLU]].
  • Also available in PyTorch as F.relu:

• ReLU is an example of a a lot of things sound complicated and "scary" in Deep Learning, but are quite simple at their core.

00:53:24 - Why non-linear activation like RelU?

• Without a non-linear activation function, 2 linear layers could simply be rewritten as a single linear layer.
  • With a non-linear, you should be able to approximate any arbitrary function according to Universal Approximation Theory.

00:55:03 - On Deep Learning's simplicity

• One of the challenges beginner's have: there isn't anything more to neural networks than the function above.
  • Some tricks to use less parameters etc
  • Most of it is just performance tweaks.

00:55:54 - Refactoring and functional composition

• [[Function Composition]] is when you take the result from one function and pass into the next.
  • A [[Neural Network]] is just [[Function Composition]] of linear layers and activation functions.
• PyTorch provides a tool for functional composition called nn.Sequential:

00:57:08 - Question: does ReLU cause many zero gradients?

• Yes, it does. However, they won't be zero for every image. And it definitely won't be for every minibatch.
  • It is still a waste of computation having "inactive neurons".
    • There is an approach that will show you how many dead neurons in the network that is shown later in the course.
  • Leaky ReLU is one alternative approach: res.max(0.0001)

00:59:27 - Training neural network

• Note that as layers get deeper, you tend to have to have smaller learning rates as they tend to be "bumpier".

• Can use recorder to peer inside learner.
  • values can be used to plot accuracy over training:

and can get final accuracy:

• We now have a function that can solve any problem to any level of accuracy, if we can find the right parameters.

01:01:53 - How can we use the learner to visual the model?

• Can use learn.model to see the model:

01:05:30 - Train models with approach from Lesson 1

• Nearly 100% accuracy! Clearly the tricks help.

01:06:20 - Jargon recap

• ReLU - function that convert negative numbers into 0.

• mini-batch - A couple of inputs and labels.

• forward pass - Applying the model to your inputs and calculting loss.

• loss - Value that tells you how model is doing.

• gradient - Derivate of loss with respect to parameters of the model.

• backward pass - Compute gradients of the loss with respect to model params.

• gradient descent - Take a step in directions opposite to gradients to make model params better.

• learning rate - Size of step we take when applying SGD to update params of model.

• parameters - numbers we're learning

• activations - numbers we're calculating

• Tensor ranks:

  • rank zero: scalar
  • rank one: vector
  • rank two: matrix

01:09:43 - Pet breeds image classification

• How do we understand the data in our data set?
  • Can use ls to see inside dataset.

• Can take a look at the images directory:

01:12:08 - L class revisited

• Methods in fastai that would return a list generally return an L class.
• L has a number of enhanced:
  • When it prints, it tells you how many items in the list and truncates the output.

01:12:50 - Generating labels with regex

• Since we know that dogs labels first character is lower case, we can use that to filter.
• Will use regex for filtering. To start, let's build the regex on one example:

• In Python, r before a string says: don't treat \ special.
• Can use that to create a DataBlock of breeds:

01:17:11 - Presizing

• These 2 lines are performance a fastai data augmentation strategy called "presizing":

  item_tfms=Resize(460),
  batch_tfms=aug_transforms(size=224, min_scale=0.75)
  

  • Resize to a reasonable size with cropping, then perform resizing for each batch.

  

• 2 advantage to this:
  1. Because 1st step creates something square, 2nd step can happen on GPU.
  2. Normally doing zoom, rotate and warp is destructive to image and is slow.
     • Fastai keeps track of coordinate transforms, then performs them all in one go.

01:21:48 - Are blocks in DataBlocks an ordered list

• Yes, it's an ordered list: independant and dependant variable.

01:22:51 - Checking and debugging a DataBlock

• Want to check DataBlocks are working correctly by visualing examples:

• Can add unique=True to show the same image with different augementations:

• Call summary on DataBlock to tell you everything that's happening:

01:24:39 - How does presize work if image is smaller than resize value

• By default uses Crop. If image is smaller than resize value then it will stretch or zoom.

01:25:46 - Data cleaning?

• A lot of people say do data cleaning before training model.
  • Fastai don't recommend that.
  • Let the model show problems with your data.
  • Once you have working datablock, then you can train model then use the ImageClassifierCleaner to clean data and retrain.

01:26:44 - fastai auto selecting loss function

• When we create a cnn_learner, we don't have to pass a loss function as fastai picks one for you.
  • Can view the loss_func picked with learn.loss_func

01:27:38 - CrossEntropyLoss

• Cross entropy loss is similar to mnist_loss from earlier, but has 2 benefits:
  • The torch.where only works when you have a [[Binary outcome]].
  • CrossEntropyLoss works when you have more than 2 categories.

01:28:48 - Why build a model before building the model?

• We train a model then clean, then retrain on a cleaner dataset.

01:29:52 - CrossEntropyLoss (cont)

• To understand CrossEntropyLoss we can first grab a single batch of data:

• The dependant variables is an integer from 0 - 36 which can be indexed into vocab:

• Now can view the predictions:

• Can see the preds add up to one:

01:32:38 - Softmax

• To make sets of predictions like this that are probability-like, we use Softmax.
• Softmax is "basically an extension of Sigmoid"
• What if we wanted more categories than just 2?

• Creating random numbers with standard deviation of 2 to resemble a model with 2 activations (is 3 or is 7)

• If we simply apply Sigmoid, we don't get something that's probability like ie sums to 1.

• What we care about is the relative confidence of being a "3 vs a "7".
  • So we can calculate the relative difference between activations and take the sigmoid of that:

• To get 2 columns, we could subtract 1-value from the second column (prob of being 7):

• How do we do this for more than 2 cols?
  • We can use the [[Softmax Function]]

• Softmax is: $$\frac{e^x}{\sum\limits_{i=1}^{N}e^{x_i}}$$
  • The Exponential function (exp) is defined as: e**x.
    • e is a special number approximately equal to 2.718. It increases rapidly.
• Can see softmax returns same value as sigmoid for the first column:

• In the multi-category case, softmax would look something like:

• Softmax tends to "amplify" a single result.
  • Which may not be what you always want.
  • An alternative is multiple binary values with Sigmoid.

01:40:38 - Log likelihood

• We could take a similar approach to the mnist_loss function by replacing torch.where with indexing:

def mnist_loss(predictions, targets):
    return torch.where(targets==1, 1-predictions, predictions).mean()

• then for each of the targ we can use that column of sm_acts using tensor indexing.
  • Here we are requesting rows between 0 and 6 and columns that match the targs:

• This works with more than 2 columns
• PyTorch provides a function that does that same thing as sm_acts[range(n), targ]
  • Note: it takes a negative, because the log taken after gives us negative numbers, called nll_loss (NLL stands for negative log likelihood):

• Confusingly, negative log likelihood doesn't take the log.

01:44:31 - Logs

• It works as a loss function, but can be better.
  • Using probabilities, it cannot be smaller than zero, or greater than one
  • Model will not care it it's predicting 0.99 or 0.999
    • But the error if there's 1000 things, the first means 10 thing is wrong, the 2nd means 1 thing are wrong.
    • Want to translate predictions as being between -infinity and infinity.
  • There is a function in Math that does it: the logarithm:

• The logarithm has this identity:

y = b**a
a = log(y, b)

• In this case, log(y, b) returns log y base b.

  • PyTorch doesn't define log this way. It uses special number $e$ (2.718...) as the base

• This relationship is one of the most important things about logarithms:

log(a*b) = log(a) + log(b)

• Because a*b can get big or small, that the precision in computers floating point math can get out of hand.
  • But, adding doesn't.

01:47:54 - Cross-Entropy Loss

• If you take the softmax, then take the log, then pass to nll_loss, it's called cross entropy loss

• In PyTorch, you can usually use nn.Class version or F.func_name. Usually the class version if preferred:

• Errata: at 01:49:56: says "camel case" name where he means "snake case".

• It returns a single number as that's required for the loss function, but the individual values are available passing reduction=none:

01:51:46 - Data ethics and mask efficiency

• Jeremy pointed at that masks are likely very effective.
  • Found himself as face of global advocacy campaign.
  • Found data around masks was misunderstood.
• Role of data scientist: to understand data and do something about it.
• What was clear: people weren't convinced by science, especially when WHO and CDC are saying the opposite.
• Put together a team to review the evidence and wrote this paper.
• Knowing the data isn't effective unless you can communicate to decision makers.