fastai - Lesson 3 - Deep Learning for Coders (2020)

Default resizing algorithm does a crop.

Can make a copy of a Datablock and change parameters as follows. I'm using it to change resize algorithm to squish

Can also add black bars for padding

Can also use reflect for pad mode

00:03:28 - Question: why squares and not rectangles

• If you know all images are rectangles - or another aspect ratio - might as well keep the that way.
• If you have tall and some wide, it's easier to square them
  • Otherwise, you have to try to get tall and wide ones in each minibatch

00:05:18 - Using RandomResizeCrop

• Can see in the pad example that some images and wide and some are tall.
• All of the methods for resizing have downsizes:
  • pad has empty space, so you are wasting computation
  • squish is most efficient but changes aspect ratio. That could be bad if that's an important part of classification
  • default cropping approach removes some information
• RandomResizeCrop:
  • Crop a random part of the image and zoom in.

• Advantage of that it's a form of [[Data Augmentation]] - each epoch, the model will be trained on a slight variation of the original image

00:08:51 - Data Augmentations and aug_transforms

• Fast.ai provides a list of other data augmentations you can use in the aug_transforms function:

• Note that these are batch tfms:
  • they happen to a minibatch at a time on the GPU.
  • That means that the augmentation can happen really fast.

00:10:42 - Shift-tab in Juypter

• Press shift-tab in function parenthesis, will pop open a list of arguments.

00:11:10 - Question: how to add different augs for train and valid sets

• Fast.ai automatically avoids add augmentations to valid set. However, it does a centre crop.

00:12:29 - Cleaning data using a trained model

• As per last lesson, I can train the model:

• We didn't do any data clearning, aside from validating each image can be opened.
  • It's much easier to clean your data after you create a model.
• Fast.ai has a tool called ImageClassifierCleaner that shows you the images that it's least confident about:

You can then iterate through all the "delete" ones and unlink them. Then move all the wrongly classified ones to the correct category.

00:15:53 - Jupyter notebooks widgets and fastbook reminder

• You can create guis/widgets inside Juypter notebooks.
• Reminder that the whole fastai book was written in Juypter notebooks.
  • Best place to learn how something was implemented is to look at notebooks, not modules

00:18:06 - Inference

• Can export to pickle file:

The 4 things returned are the probabilities of each class. You can find the order by asking for the dataloaders vocab:

• Any time you have mapping from numbers to string, mapping is stored in vocab.

00:21:24 - Creating application from Juypter notebook

• Curated approach to building web applications from notebooks.
• Key pieces of technology:
  • IPython widgets (ipywidgets)
  • Voila

• PILImage can create images from binary blob, which is what FileUpload returns

• An Output widget is something you can fill in later

I now can run another cell, to update the placeholder:

I can use a Label to display them

Then another button for classification (doesn't do anything yet)

... now build an event handler: a function that is called when it's pressed

Can now put them all together in a VBox (a vertical button widgets)

00:28:58 - Voila

• Needs to be installed:

Takes a notebooks and removes the code cells. Converting into an application that can only run the widgets.

• Mybinder can convert a notebook into a free website.

MNIST

• Images of handwritten digits

  • Collated into ML dataset by Yann Lecun

• Example of starting with something really simple and scaling up.

• untar_data is an downloads and uncompresses data idempotently.

• fastai adds an ls method to Path which returns a fastai class called L.
  • L has all the function features of lists + more.

• Example of one of the images labelled 3:

01:11:37 PIL and Images

• Python library PIL is one of the most widely used image library.
• In a computer, everything is represented as a number. To view numbers in image in Juypter, we have to convert to NumPy array using the array function.

• Taking the first 15 rows and the first 10 columns to make visualisation easier.

• The numbers are 8-bit unsigned [[Integers]]s, which means the numbers range from 0 to 255.

01:12:26 - Tensors

• Tensors are the PyTorch equivalent of NumPy array, and behaviour very similar.
  • Key difference is PyTorch tensor can be computed on GPU.

01:14:49 - Visualising image with Pandas

• Can be added to a Pandas dataframe to turn the background of a cell into a gradient.

• We can now clearly see what the computer is doing when it renders an image onto the screen.
• The image is 28x28 for a total of 784 pixels (much smaller than the 28 million (28 megapixels) on my Pixel 4 phone.

01:16:00 - Creating a baseline model without Deep Learning

Pixel similarity

• Single there's only 2 classes, we can define the problem as: how can we classify 3s?
• One approach: get the average of all the pixels of the 3s and find the "ideal" 3.
  • Then compute the distance from the "ideal 3"

01:17:33 - Baseline definition

• A simple model that you think will do okay at the task.
  • Something quick and easy to implement.
  • Any further implementation's performance then has a metric to determine success against.
• "One of the biggest mistakes is that they fail to create a baseline"

01:18:39 - List compreshension

• [[List comprehension]] to get list of all sevens and three images:

• List comprehension is a similar idea to Linq in C#.

• To perform computation across all images, we can stack them all on top of each other as a 3d cube of images to create a "rank-3 tensor"

• Taking the mean in PyTorch, requires casting [[Integer]] types to [[Float]] types, which we'll do below.
  • Note that when converting images to floats, the convention is for the pixel intensity value to be between 0 and 1.

01:22:44 - Axis of tensors

• A rank-3 tensor means it has 3 axis or dimensions.
  • Numpy calls rank value axis.
  • PyTorch calls it dimension.
• The shape attribute of a tensor returns a list with each element representing the size of an axis:

• Can get the rank of the tensor by taking the shape or calling ndim:

• The term "dimension" is used in 2 ways.
  • Sometimes used for size of an axis (space is 3 dimensional - each datapoint has an (x, y, z) value)
  • Sometimes used for the rank, or number of axes (a matrix has 2 dimensions)
• When confusing, use rank, axis and length which are unambiguous.

01:23:57 - Computing the mean

• Can calculate the mean of all image tensors, by taking mean along dimension zero, which gives us the "ideal" 3.

• Note: that if we didn't provide an axis, it would be the average across all axis:

* Same for 7s:

01:24:52 - Computing distance from mean

• First, i'll take one example of 3 at random:

• Can't simply subtract each pixel from the corresponding pixel in the mean image, because some values would be negative and would average to nothing.
• Two approaches we could take to make positive values:
  • Take the mean of absolute value of differences ([[Absolute Value]] is a function that replaces negative values with positive values).
    • Called [[Mean Absolute Difference (L1 Norm)]]
  • Take the mean of the square of differences (which also makes everything positive) then take square root (to undo the squaring).
    • Called [[Root mean squared error (RMSE) (L2 norm]]
      • Penalises bigger mistakes than L1 Norm.

• Mean absolute difference:

• Root mean squared error (RMSE)

• Since it's clearly closer to the 3 than the 7, we can guess it's 3.

• PyTorch provides these as loss functions in torch.nn.functional which is usually imported as F:

• MSE stands for mean squared error and L1 refers to math jargon for mean absolute value (in math called L1 Norm)

01:28:38 - Working with arrays and tensors

• Important to be comfortable working with arrays and tensors. Summary of key things:
  • Create array or tensor with lists of lists:

• Can index into a single row:

• Or single column...in this example, we're fetching all columns:

• Can combine with Python slice syntax ([start:end], end being excluded)

• Can add to tensor

They have a type:

Automaticaly converted from int to float if needed:

01:30:53 - Computing metrics using broadcasting

• We want to create a validation set to check how good the model is.

• Want to write a function is_3 that return True if it's a 3.
• We can first create a function that returns the mean of difference between somethings

• To do for the entire validation set, we can call the same function on the entire validation set:

• Being able to run the same function on a single image or a rank-3 tensor of images is thanks to [[Broadcasting]].

  • Broadcasting expands the tensor with the smaller rank to have same size as one with larger rank.
  • In PyTorch, the broadcasting operation is done on the GPU.

• Need to first understand [[Element-wise operations]]

  • Operations that are performed on the corresponding elements of 2 vectors:

• When you have different shapes, PyTorch "copies" the matrix the number of times required to have a shape that matches the larger rank.
  • It doesn't actually allocate any additional memory for copies of the matrix, just acts like it does.
  • Does all the operations in C or Cuda

• We can then call abs value, which calls it on each value:

• Can then create an is_3 function, which compares distance between perfect 3 and 7:

• When you convert boolean response to float, get 1.0 for true and 0.0 for false:

• Can test on full validation set of threes

• Not that you're avoiding loops. Which makes it thousands or more times faster.

• Model is over 95% accurate to recognise 3s from 7s.

01:39:50 - Stochastic Gradient Descent (SGD)

• So far solution doesn't fit Arthur Samuel's description of Machine Learning.

  • It doesn't include a way to test and update the weight assignment.

• Instead of finding an ideal image, we could instead develop a set of weights for each pixel and train it to be better at recognising images.

• Want a function that takes the pixels of an image and the weights and returns a probability of it being a certain number. For simplicity we'll turn the weights and image into a long vector:

  • 28x28 -> 1x784

• The approach to try is as follows:

1. Set the weights of the vector w to random values.
2. For each image, use weights to predict whether it appears to be a 3 or a 7.
3. Based on weights, calculate loss to see how good model is.
4. Calculate the gradient: for each weight, if I made smaller or larger, how would changing the weight change the loss?
5. Step (change) the weights based on the calculation.
6. Repeat from 2.
7. Stop training if results are good enough, or you don't want to wait any longer.

This approach is called Gradient Descent.

Each 7 steps has lots of choices that can be made. But the general approach follows basic principles:

• Initialise: set params to random values.
• Loss: Arthur Samuel's "testing the effectiveness of any current weight assignment in terms of actual performance"
  • Need a function that return a number that is small if performance of model is good.
• Step: could figure out if loss should be increased or decreased by just trying different values and seeing what happens. However, it's slow. Calculus allows us to figure out what direction to change the weights in.
• Stop: decide when to stop. Usually just a certain number of epochs.

01:46:05 - A simple first approach

• Start with a simple quadratic function - pretending it's a loss function with x as weight param of function:

Try to find the bottom point - the x value at the bottom.

Start by picking a random value of parameter and calculating value of loss

Then, see what happens if we increase or decrease our params a bit. This is slope at a particular point:

If you change weight in the direction of slope, and calculate loss and adjustment again, repeat this a few times.

Eventually, you will get to lowest point on curve:

• Idea comes from Newton and is called: "Newton's Method"

01:48:05 - Calculating the gradients

• To calculate the derivative, we need calculus.
• However, you don't have to do calculus: the computer does it for you.
• In PyTorch, a tensor can be modified with requires_grad_(), so that every calculation has the gradient calculated.

• In PyTorch, when the method ends with _ it means the function modifies the Tensor in place - it doesn't just return a new one.

• Can then tell PyTorch to calculate the gradients, using the backward() function.
  • Backward refers to "Back propagation" which means to take the gradient.

• We can view the gradients by checking the grad attribute of the tensor:

• We know that worked from the power rule: dx(x**2) = 2x (dx(3**3=2) = 6)

• Now we can repeat with a vector argument:

• Add sum() to function to take a vector (rank-1 tensor) and return a scalar (rank-0 tensor):

• We expect the gradients to be 6, 8 and 20 respectively. Let's see:

01:52:41 - Learning rates

• The gradient only tells us the slope of the function. Doesn't say how far to adjust the params.
  • If the slope is large, then there are more adjustments to do, but if small then we may be close.
• Can use a simple function that subtracts the gradients multiplied by some smaller number (usually between 0.001 and 0.1) called: learning rate.

w ~= gradient(w) * lr

• If the learning rate is too small, it will take a long time to "converge" (find the min)
• If too large, it may never find it.
• Picking a good learning rate is an essential part of gradient descent.

01:54:41 - Gradient Descent end-to-end example

• Imagine measuring speed of rollercoaster as it goes over and down a hump.

• A bit of random noise was added, which estimates at anytime what the speed of the roller coaster is. We can start by guessing it's a quadradtic function:

Step 1. Init to random values and tell PyTorch to check gradients with requires_grad_()

Step 2. Calculate the predictions:

• Create a function that shows how close predictions are to target. The predictions are shown in red.

• Obviously we arne't very close since we haven't done any training.

Step 3. Calculate the loss:

The goal is to improve it. To do it, we'll need to know the gradients.

Step 4. Calculate the gradients. An approximation of how params need to change.

Step 5. Update the params based on gradients ("step" the weights)

• We use .data attribute which ensures the gradient isn't calculated on that step.

Has loss improved?

Looks like it has! Quite a bit.

Let's do it a bunch more times.

• We can also visualise the loss getting better:

• Step 7 is to just stop. We usually stop when we stop seeing performance improvements on validation set.