fastai - Lesson 6 - Deep Learning for Coders (2020)

Had to install latest version to work around this issue: https://forums.fast.ai/t/poor-results-when-running-chapter-5-examples/87161

00:00:00 - Lesson 4 review

• Ensure you understand Cross Entropy loss before continuing on.

00:01:09 - Model interpretation

• A confusion matrix: displays the correct predictions in the diagonal.

• When you have a lot of classes, it can be better to use most_confused
  • Tells you combinations that it had most trouble with.

• Even experts don't agree about Birman and Ragdoll
  • When model makes sensible mistakes, it's a good sign.

00:03:09 - Improving learning rate to train a better model

• One way to make model better is to improve learning rate.
• Bigger learning rate = faster training.
  • However, too might and the model will overstep the minimum.

00:05:00 - Learning Rate Finder

• [[Learning Rate Finder]]
  • Idea by Leslie Smith:
    • Start with tiny learning rate.
    • Take a minibatch and step, then record the losses.
    • Increase learning rate by some %.
    • Step again.
    • Keep doing until loss gets worse.
    • Select learning rate a bit before you go to far.
    • Pick either:
      • One order of magnitude less (the min divided by 10).
      • Loss point where loss was decreasing.
• lr_find() in fastai computes that for you.

• You can then retrain using recommended epoch.

• Learning rate finder was only discovered in 2015.
  • Shows how many more discoveries in DL.
  • People in engineering tend to prefer this completely automated.

00:09:34 - Question: Is the learning rate plotter plotted against a single mini-batch?

• It's plotted while walking through the DataLoader, as you normally would when training.

00:10:13 - Question: Is network reset after each mini-batch, or does it accumulate?

• It's not reset. We want to see it improving.
• At end of training, we go back to previous rates.

00:10:53 - Question: why would a single learning rate found at the start of training, still be useful after many epochs?

• Next section.

00:11:17 - Question: for the learning rate finder, why use the steepest and not minimum?

• The minimum is the point where it's no longer learning.
• We want somewhere before:
  • Sylvan's favourite approach: minimum / 10.

00:12:09 - Transfer learning recap

• Firstly, neural network is a bunch of linear layers with activation functions between them.
• Each linear layers have parameters.
• After you train neural network on image net, have parameters that aren't random.
• Early layers learn about general ideas and later are more specfic.
• Transfer learning:
  • Take parameters that have been trained.
  • Get rid of very last layer and replace with random weights.
  • Freeze the other layers, and just train the last layer.

00:13:50 - Learning rate finder simplicity

• Seems surprising that the learner rate finder can be so useful even though so simple and approximate.
• Wasn't really know until fastai started talking about it.

00:15:37 - Transer learning continued

• fine_tune is the method fastai uses to do it.

• Freeze everything up to the last layer: self.freeze()
• Then just fit the randomly added weights.
• Unfreeze all the other layers.
• Fit for some number of epochs.

00:18:13 - Fine tune by "hand"

• We don't have to use fine_tune:

• cnn_learner already freezes the model for you.

• Then we can manually unfreeze the model:

00:18:45 - running lr_find after unfreezing

• You would assume it would be the wrong learning rate, so we can run lr_find again.

• The shape here is different because the model is trained.
  • We generally take the point where it starts to get worse. Like 10-4 and divide by 10 back from that.

00:19:32 - Discriminative learning rates

• Since we know later layers are more specific to the problem, and the earlier general.
  • Want bigger learning rate for the later layers and smaller for earlier.
  • Originally developed in ULMFiT
    • Based on paper by Jason Yosinski who showed that layers should train at different speeds (though he didn't actually run the experiment)
• Fastai lets you pass slice object
  • Slice is an object
  • Very first layer will be 1e-6 and the last will be 1e-4
    • Learning rates in between will be equally sliced.

00:22:06 - fit_one_cycle

• Starts at a low learning rate
• increases it gradually for first 1/3 until it gets to lr_max
• Then remaining 2/3, it gradually decreases.
• Developed by Leslie Smith (same person who did learning rate finder).
  • Ignored by academic community.
• Looking at results above, we don't necessarily just want to take model with lowest weights.
  • We'd like to train model again with number of epochs equal to the best error rate.
  • Change epochs 12 to 8 and train from scratch.

00:22:56 - Plot loss

• Often see validation loss get worse before error rate gets worse.

00:24:40 - Improving model with deeper architectures and fp16

• Could try creating a deeper architecture: more pairs of linear layers and activation functions.
  • For pretrained models, there's usually a small section of layer numbers available: resnet18, resnet34, resnet50 etc.
• Bigger models have more parameters and more gradients.
  • It's common to run out of GPU memory: Cuda runtime error: out of memory.
    • If it happens, restart notebook and try again with smaller learning rate.
• Can add fp_16() to use less precise numbers (half precision floating point, also called fp16).

• Trick is to use small models for as long as possible, to clean data and get pipeline ready etc.

00:28:36 - Question: how do you know or suspect that you can do better?

• Always assume you can do better, because you never know.
• Ask yourself: do you need to do better? Will model be useful enough now?
  • Do something practical with model as soon as possible.

00:29:41 - Questionaire and further research

• Read paper by Leslie Smith that introduced the learning rate finder.
• Try to improve accuracy in chapter.

00:30:48 - Multi-label classification

Continued in next notebook.

00:30:48 - Multi-label classification

• More than one or more labels per image.

00:32:01 - Question: is dropping floating point number precision have impact on final loss?

• Sometimes it makes it better.
  • Doing some random, that creates more uncertainty.
  • Usually doesn't make the difference either way.

00:33:53 - Exploring Pascal dataset.

• Pascal is a famous dataset with more than one kind of classified object per image.

• Comes with a CSV which includes the labels for each image.

• List of categories in each image, a space delimited string (eg horse person)

00:34:30 - Pandas

• The pd above stands for Pandas: important library for data processing.
• Main class is DataFrame, which is table of rows and columns.
• Can access rows and columns with iloc property.
• Getting column 0:

• Getting row 0:

• Can also get column by name:

• Can create a DataFrame from a dict:

• Then create a new column by adding other columns:

• Book recommendation: Python for Data Analysis by Wes Mckinney.

00:37:10 - Review Dataset and DataLoader

• We want to create DataBlock that we can train from.
• Reminder about Dataset and DataLoader
  • Dataset - anything you can index into:

• DataLoader - an interator that gives you a stream of mini batches with independent and dependent vars

• Can use * which inserts each element of list as arg in function.

00:40:04 - Datasets and DataLoaders

• Datasets is an object that has a training and validation dataset.

• Sometimes easiest to start by creating and testing Datasets:

• Can use functions to create independent and dependent variables:
  • (for example, open image and create lable from filename).

• Can then create a DataLoaders instance:

00:43:23 - DataBlock

• Instead of creating DataLoaders from scratch, can use a DataBlock:

• It's returning the same thing in for both variables, as that's the deault behaviour.
• You can pass get_x and get_y functions:

• To save (serialize) DataBlock, you can't use lambdas.
• Next, we can add the full path to the image, and split on the label name:

• Since we want the image opened as an image, blocks argument lets you say what sort of "block" you want for each item in tuple.
• In the example below, we use MultiCategoryBlock which expects a list of strings:

00:48:36 - One-Hot Encoding

• The labels there are [[One-Hot Encoded]]
  • Each position corresponds to a possible label.
  • If the label isn't in the image, it's 0. If it is 1.
• We can take a look at the first label, and see what it is:

00:50:44 - Question: shouldn't tensor be of integers?

• Conceptually it's a tensor of integers, but for the purpose of floating point calculations in cross entropy, we just make it a float.

00:51:30 - splitter function

• The DataBlock uses a random split by default.
• This dataset tells you what's in the validation set: usually a good idea to use the dataset's validation, so you can compare results.
• The splitter function can be used to return the idxes in the train and validation set:

00:52:39 - Question: why don't we use 8-bit precision?

• It's too flat and bumpy - hard to get good gradients.
  • Sometimes they do a thing called "Quantization" to allow a model to do inference much faster.

00:53:54 - Finish DataBlock

• Now we add item_tfms to allow collation into DataLoader:

• Then dataloaders to get a DataLoader back:

• And summary to ensure it all worked:

00:54:48 - Binary cross entropy

• To start', we'll grab a learner using the dls above:

• Get one batch of predictions:

• Examine one set of activations for an item in the batch:

00:56:10 - Question: Is DataBlock API compatible with libraries like Dask?

• DataBlock API allows you to pass anything that's indexable, including Dask.

00:57:20 - Question: what path should the images be in?

• Anywhere. You specify an absolute path in the get_x function.

00:58:49 - Binary cross entropy cont

• The activations above aren't between 0 and 1, but you can use the [[Sigmoid]] function for that:

• Not that log is applied here again so the training is "better behaved".
  • The function works on a single output or multiple.
• In PyTorch, the function is F.binary_cross_entropy, or the module version is nn.BCELoss
• The above loss functions don't use initial Sigmoid. Use F.binary_cross_entropy_with_logits or nn.BCEWithLogitsLoss if providing logits.
• For single-label datasets like MNIST or Pets, F.nll_loss (doesn't include Softmax) and F.cross_entropy (includes Softmax) are the equivalents.

01:03:29 - Multi-label metric

• Previously using error_rate (which is 1-accuracy), but that only works for single label dataset (argmax)
• In multi-label, can't use argmax because could have 0 or more labels.
  • Need to introduce a threshold where above considered 1 and below 0.
• In learner, you pass the metric as a function. To change threshold can use a function called partial:

• Note that we don't need to loss func, as fastai figures it out:

01:07:13 - Tuning threshold

• One way to find best threshold is to calculate accuracy at a range of thresholds and plot

• The fact it's a curve and not a bumpy function, is a good indication we're on right track.

01:09:09 - Regression (cont next notebook)

01:09:09 - Regression (cont next notebook)

• Will learn how to "invent" applications using DataBlock
• Image regression: for an image, return some kinda of continuous value (age prediction is one example)
• In this example, use Biwi Kinect Head Pose Dataset to find the centre of a person's face.

• 24 directories, each corresponding to person photographed.

• The label is a text file with the pose data. The filenames are the same as the first part of the image file name, ending in pose.txt

• Fastai way to create PIL image is PILImage.create

• Note that in computer vision the dimensions are columns by rows.
• This function gets the head centre point:

• That function can be passed to get_y which labels each item

• Note that the validation set is an entire person. To enure that we don't trick ourselves into thinking we're generalising better than we are.
• FuncSplitter allows you to split with a function.
• Normalize subtracts the mean and divides by std.

• Coordinates is a [1, 2] tensor (a single point).

01:16:47 - Image channels

• A colour picture is a rank-3 tensor. First axis is channels: red, green and blue.

• A mini-batch of images is a rank-4 tensor.

01:17:45 - Training image regression model

• When we create a learner, we pass y_range which forces a range of targets.

def sigmoid_range(x, lo, hi):
    return torch.sigmoid(x) * (hi-lo) + lo

01:19:14 - Question: can you have more than 3 channels as input?

• Yes. Though pretrained models will need some changes to support this (copying layers for example)

01:20:13 - Training cont

• No loss function specified. fastai provides MSE by default for regression

• Since it's squared error, we can get the square root to find average coord prediction error:

01:25:14 - Collaborative filtering and latent factors

• What is collab filtering?
  • On Netflix, you might be watching scifi movies made in 70s.
  • Could find other people that have watched the same movies you've watched, and can recommend other films they've watched that you haven't.
• Foundation of this idea is that there are some features in items (ie movies) called latent factors that can be "uncovered"

01:28:09 - MovieLens

• Contains tens of millions of movie rankings:

• Main table can be opened with pandas:
  • Note that there's no header, so we provide our own column names.

01:28:53 - Question: do you perform cross-validation in Deep Learning?

• Doing cross-validation or not is more a question about dataset size.
• Less common now due to bigger data sizes.
• Still used in Kaggle competitions where metric accuracy is quite important.

01:30:32 - Question: what other applications for collab filtering other than recommender system are there?

• Depends on how you define recommender system: diagnosis patients, figuring out what a person is likely to click on are sort of recommender systems.

01:31:19 - Question: how would you approach video classification?

• Usually approached as image classification per frame.
• Could also make the 4th axis time, but it would require a lot of memory.

01:33:20 - MovieLens cont.

• ML_100k is a small subset of the full dataset (there's a 25M dataset too).
• Can cross tab to get a better idea of what the dataset is:

01:35:32 - Latent factors

• Assume you have columns for each movie that described: how much action, how much drama and how much comedy on a scale of -1 to 1.

• We could then describe a user based on their preferences for those rating categories:

• We can then do element-wise multiplication to tell us how much they correlate:

• Multiplying 2 vectors and adding the results is called: [[Dot Product]]
• We could calculate the match between a less action-packed movie:

• We don't actually know what the latent factors are in our dataset: but we can learn them.

01:37:51 - Learning latent factors

• We can visualise it in Excel, starting by setting latent factors to random values

• A matrix multiple of a row by column is identical of dot product of 2 vectors
• To do it in PyTorch, we start by replacing movie id with title. Just to make it easier to look at:

• merge by default will merge on a shared column name.

• Can then build DataLoaders object from table.
  • Takes first column for user by default.
  • Use item_name to specify item column

• Can represent movie and user latent factors as simple matrices:

• randn returns normally distributed numbers.
• Look up in index is not an operation deep learning models can do
  • However, can represent look up in index as matrix product with a one-hot encoded vector

01:47:34 - Question: how different is collab filtering with sparse data compared to dense data?

• See fastai Linear Algebra course.

01:47:58 - Question: in practise, should we tune the number of latent factors?

• Yes. It's another hyperparameter that can be tuned.

01:48:16 - Embedding

• Doing matrix multiplication with a matrix of one-hot encoded vctors is an inefficient way to do the computation.
• Embedding:
  • A computational shortcut
  • Has the gradients of a one-hot matrix multiplication
  • Has the speed of an array lookup

01:50:38 - Object-oriented programming

• In PyTorch, a model archiecture is a class that inherits from nn.Module
• Class syntax:

class Dog:
  def __init__(self, name):
      self.name

  def speak(self):
       print(f'{self.name} says wolf')

• The __init__ method is the classes constructor and is referred to as a "dunder method" (double underscore).
• The syntax for inheritance is as follows:

class Hound(Dog):
   def speak(self):
      print(f'{self.name} the hound says wolf')

01:53:40 - Module in PyTorch

• Module is a PyTorch class that fastai has customised:

• Function forward gets the user id and movie id as 2 columns.
• We can use those to lookup the embeddings.

• We can now use a standard Learner to train the model

• To improve the results, we can make it better by forcing predictions between 0 and 5 using sigmoid_range

01:59:02 - Bias term

• When we multiple the latent factors, it doesn't take account of the fact that some users rate movies badly. Or a movie is universally loved.
  • We can add a single number for each movie and user, that it added to the result which takes that into account.

02:01:05 - Regularisation

• We can see that the model appears to be overfitting quickly.
• With images, we have data augmentation. What can we use in collab filtering to allow training for longer? Regularisation
• To look at next week.