Lesson 3

00:00:00 - Examples of students study materials

• AWS fastai GPU Image Setup.

• tmux on AWS.

• Case Study: A world class image classifier for dogs and cats (err.., anything)

  • Summary of last lessons.

• Estimating an Optimal Learning Rate For a Deep Neural Network

• A practitioner's guide to PyTorch

• Visualizing Learning rate vs Batch size

• Do smoother areas of the error surface lead to better generalization?

• Convolutional Neural Network in 5 minutes

• Decoding the ResNet architecture

• Fast.AI beginner forum.

00:06:30 - Recap of lessons so far

• So far looked at image classification and CNNs as a practitioner.
• Going to look at CNN theory today.
• Then, intro to CNN for structured data.
• Language RNN intro.
• Collaborative filtering intro.
• Then a deep dive into those topics, basically in reverse.

00:08:30 - Review of previous week's assignment

• Make sure you know how to download data (see Kaggle API).
  • Tip for just downloading the dataset you want: install CurlWget.
    • After starting a download, it stores all your headers and cookies needed to download something on the command-line.
  • Can put the data anywhere and symlink to it in your project directory: ln -s /some/path/to/data data/my-data.
  • Add -l flag to ls to show the symlink source.
• Introduces bn.freeze: use if your images are very similar to ImageNet images.
  • Causes the batch normalization moving averages to not be updated.

00:21:52 - Dogs and Cats in Keras/Tensorflow

• See notebook for walkthrough.
• Had to first install CUDA 9.0 (not 9.1)

sudo dpkg -i cuda-repo-ubuntu1604_9.0.176-1_amd64.deb
sudo apt-key adv --fetch-keys \
     http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
sudo apt-get update
sudo apt-get install cuda-9-0

• Keras ImageDataGenerator expects image in a similar way to the from_paths method, with a folder per class.
  • Can pass the types of data augmentation you want to do to the constructor.

• Keras tends to need more lines to achieve similar results to Fast.ai's library.

• Unfreeze earlier layers and continue training:

00:30:00 - Reasons to use Keras/TF over PyTorch

• Deploying to mobile: PyTorch on mobile is early.
• If you work for a company already using TF.
  • Harder to get same state of the art results in Keras/TF, but doable.
    • Have to reimplement SGDR.
    • Have to implement differential learning rates and batch norm freezing.

00:32:08 - Revisit Dog Breeds

00:32:45 - Submitting to Kaggle

• For each competition, there's an evaluation page that tells you what to submit.

• For dog breeds:

  " Submissions are evaluated on Multi Class Log Loss between the predicted probability and the observed target.

  Submission File For each image in the test set, you must predict a probability for each of the different breeds. The file should contain a header and have the following format:

  id,affenpinscher,afghan_hound,..,yorkshire_terrier
  

000621fb3cbb32d8935728e48679680e,0.0083,0.0,...,0.0083 etc.

"

* Pass in `is_test=True` to TTA to use test data (need to pass the test data into the data class):

log_preds, y = learn.TTA(is_test=True) probs = np.mean(np.exp(log_preds), axis=0)

* Can load it into a Pandas data frame as follows:

df = pd.DataFrame(probs) df.columns = data.classes df.insert(0, 'id', [o[5:-4] for o in data.test_ds.fnames])

* Can save DataFrame as CSV:

df.to_save('filename.gz', compression=gzip, index=False)

* Can use the `FileLink` function to download file from the notebook:

from IPython.display import FileLink FileLink('filename.gz')

• To make a prediction for a single file:

  trn_fms, val_tfms = tfms_from_model(arch, sz)
  
  ds = FilesIndexArrayDataset([fn], np.array([0]), val_tfms, PATH)
  dl = DataLoader(ds)
  preds = learn.predict_dl(dl)
  np.argmax(preds)
  

  • Shorter version:

    im = trn_tfms(Image.open(PATH + fn))
    preds = learn.predict_array(im[None])
    np.argmax(preds)
    

    • Note that the image needs to be transformed first. tfms_from_model returns train and val transform function.
    • Also, predict_array expects a mini-batch with an extra first axis, im[None] does that:

00:42:27 - CNN theory

• Convolution is a matrix, which is used to perform a dot product on every section of an image, to generate a smaller image with some feature highlighted.
  • Visual example from Image Kernels Explained Visually by Victor Powell.

00:43:19 - Visual CNN explanation by Otavio Good

• Otavio Good (creator of WordLens, now part of Google Translate) created a visualisation of CNNs.

1. Start by creating another matrix, which is the result of running a convolution over ever image block. The convolutions have been learned by the model, though in this example it's finding certain vertical edges or dark to light transitions.

2. Generate more matrices by doing the same thing against the source image using other filters.

3. Put the generated matrices through your non-linearity, which is mostly Relu, which throws away the negatives:

4. Next layer is a max pool. Which replaces every 2x2 grid with it's max pixel:

5. Run all the generated layers through another set of filter, activations and max pool.

6. For the layers returned, use the learned weights in the final layer to make final predictions:

00:49:50 - Spreadsheet CNN explanation

• Available as a Google Doc here.

• Newer CNN architectures generally don't contain fully-connected final layers.

  • VGG contains 2 fully-connected layers.

01:05:27 - Audience questions

• Question 1: What would a multi channel input look like?

• Answer 1: You'd have a filter for each channel, that would be concatenated.

  • Same principals apply to data with another dimension than channels.

• Kernel values start with random values and SGD is used to find the optimial kernels.

01:08:46 - Final Layers

• Final layers would require a set of weights for each possible class output.

  • You'd perform a dot product against the output layers and those weights.

  • You'd then have a set of numbers returned, one for each class:

    • cat = -1.83
    • dog = 2.85
    • plane = 3.86
    • fish = 4.08
    • building = 4.07

  • Since they would be log values, you'd take the exp to remove any negatives:

    • cat = math.exp(-1.83) = 0.16041356777517274
    • dog = math.exp(2.85) = 17.28778184056764
    • plane = math.exp(3.86) = 47.46535136885352
    • fish = math.exp(4.08) = 59.14546984988227
    • building = math.exp(4.07) = 58.556962591892386

  • Then run the softmax activation function to get their final values.

01:12:00 - What's an Activation Function?

• A function that's applied to activations.
  • Always takes in 1 number and returns another number: max(0, $some_activation)
• Why?
  • If you were to stack a bunch of linear functions together, you'd end up with another linear function.
  • By adding non-linearity, you can then approximate any type of function.
• In Deep Learning activation function = non-linearity.

01:14:08 - Softmax

• Softmax activation has 2 probability-like properties that make it attractive for the last layer:

  • All values sum to one.
  • All values are between 0 and 1.

• Softmax is calculated as follows: math.exp(output) / sum(math.exp(i) for i in all_outputs)

• Because you have calculated the exp value before the final calculation, it tends to emphasis larger values a lot.

  • Final results tend to point to a single thing - making it not suitable for multilabel classification.

01:15:59 - Most important math for Deep Learning

• Logarithms:
  • log(x * y) = log(x) + log(y)
  • log(x/y) = log(x) - log(y)
  • if log(x) == y then e**y == x
• Plotting logs:

• Plotting exp:

01:20:25 - Revisit Planets competition

• The planetary dataset has images with multilabel like haze, water etc.
  • Since we want to predict multithings, softmax isn't so good.
• Good practise to "anthropomorphise your activations functions":
  • Softmax - "I want to pick a single thing"
• Fast.AI knows what to do under the hood with multilabels.
  • It will detect multilabel data and change the underlying activation function.

01:25:25 - Data object attributes

• Data object (ImageClassifierData) has these attributes:
  • val_ds - Validation Dataset
    • Use to get a single image.
  • val_dl - Validation DataLoader
    • Use to get a transformed minibatch.
• PyTorch is more Pythonic than Tensorflow, which "reinvents the wheel" a lot.

01:41:17 - f2 metric

• f beta function: how much do you weight false negatives vs false positives?

• f2: f beta with beta = 2

  • Used because that's what Planets used for the competition.
  • Defined as follows:

    fbeta_score(targets, predictions, 2, average='samples')
    

• Fast.ai design philosophy: get everything you probably want by default, but allow you to change it.

01:44:24 - Sigmoid (multilabel activation function)

• Similar to softmax, but doesn't divide the values by the sum.
• It should return a number between 0 and 1.
• Allows multiple things to be high.
• On the last layer, it's calculated as math.exp(activiation) / (1 + math.exp(activation)).
• Sigmoid math notation: ${\displaystyle S(x)={\frac {1}{1+e^{-x}}}={\frac {e^{x}}{e^{x}+1}}.}$
  • Shape:

• used in Logistic Reg.

01:47:30 - Audience questions

• Question 1: why only learn the last layer on the Planet dataset, given how different it is to ImageNet?

• Answer 1: to ensure final layer weights aren't random when training earlier layers - they'll likely take longer to train otherwise.

• Question 2: what is trained when you unfreeze earlier layers - the kernels?

• Answer 2: Training in conv layers is training filters/kernels. Only FC layers have trainable weights.

• Question 3: In example of training satellite images, how do you get an that's 64x64?

• Answer 3: Depends on the transform. When using data augmentation, it takes a randomly choosen crop.

• Question 4: Is the choice of metric different to the loss function?

• Answer 4: Yes - loss function is covered next week.

• Question 5: Multipass modelling, does the training have to be multiclass?

• Answer 5: Might be possible.

• Question 6: Differential learning rates: are they spread evenly across the layers?

• Answer 6: Yes. It uses layer groups, which were hand picked by Jeremy.

  • Last one always refers to the last layer.
  • Can be overriden in the library.

• Question 7: How can you dump the layers of the model?

• Answer 7:

• Question 8: why was my plot empty when I ran lr_find?
• Answer 8: if you have a tiny dataset, you can't get enough minibatches for lr_find.

01:59:47 - Intro to structured data

• Academia mostly ignores structured data, because it's harder to share.
• Couple of new imports:

• If you're not familiar with Pandas, checkout book called Python for Data Analysis.

• Download Rossman dataset:

• Also download external datasets provided by Kaggle participants: