Lesson 6

00:00:15 - Blog posts

• Optimization for Deep Learning Highlights in 2017 by Sebastian Ruder
  • Covers learning rate tuning techniques.
  • Optimization stuff covered in Fast.ai

00:02:18 - Collaborative filter review

• Model recap: summed to user and movie embeddings, adde the relevant biases and pushed through a sigmoid to force it between 1 and 5.

00:03:29 - Interpretting model results

• Retrain model from last lesson:

• Can grab model attribute from learn as follows:

• Nice thing about PyTorch layers and models is they can be called as if they were a function.
  • To get a prediction from m you could call m(...).

• Note: new version of PyTorch has done away with Variables (V): everything is done with tensors.
• Fetching the top 3000 movie biases, returns you the following:

• Jeremy's approach to PyTorch vs Numpy: try to use Numpy for everything, because it tends to be easier to work with. Use PyTorch when you need the GPU.

00:13:11 - Audience questions

• Q1: If you wanted to build a model with the GPU, how would you bring it to production (if you wanted to do inference on a CPU)?
• A1: You can move a model onto the CPU with m.cpu():

• You can also move your variable to the cpu with the same method.

• Note: if you machine doesn't have a GPU, then the model will default to being on the CPU.

• Q2: If you train the model on the GPU, then save the embeddings and weights, do we have to do anything special to load to CPU?

• A2: The load_model function in Fast.ai will do that for you.

00:15:40 - Zip function in Python

• Iterates through a number of lists at the same time:

00:16:13 - Worst and best movies

• Take a look at the lowest rated movies:

• This is a much more authenticate way to find out the worst movies, because averages can be skewed to popular movies.
  • The model removes the fact that people have different way of rating movies and different people have different tastes.

• Look at the highest rated movies:

• Do the same thing for embeddings:

00:19:09 - Dimensionality reduction with PCA

• Can "compress" from high dimensional spaces into lower dim spaces using PCA.
  • PCA: principle components analysis.
  • Linear technique.
  • Almost identical to Singular-Value Decomposition.
  • Covered in Computational Linear Algebra

• PCA tends to reduce down to the the "main" axis within a set of latent factors.

00:21:12 - Component analysis

• Don't know ahead of time what PCA is: it's just the main axis in the space of latent factors.

• Can look at it to interpret.

• First component:

• Appears to be serious vs easy watching or class vs blockbustter.

• Maybe "crowd favorite" vs "polarising"?

• can draw a scatterplot of the first 2 components:

00:24:24- Evaluating embeddings of other models

• Q1: Can you interpet other types of embeddings (like the Rossman ones)?

• A1: Yes (starts next section).

• Entity Embeddings of Categorical Variables paper

• Embeddings are just simple matrix multiples with one-hot encoded vector:

  Example:

  • Cat 1: [1, 0, 0]
  • Cat 2: [0, 1, 0]
  • Cat 3: [0, 0, 1]

  Embedding matrix:

  [0.1 0.2 0.3] [0.4 0.5 0.6] [0.7 0.8 0.9]

  • Cat 1 emb (transposed): [0.1,0.2, 0.3]
  • Cat 2 emb (transposed): [0.4, 0.5, 0.6]
  • Cat 3 emb (transposed): [0.7, 0.8, 0.9]

• Paper uses neural net trained embeddings, replacing categorical variable with entity embeddings and feeds it into gradient boosted trees (GBM) and random forest and gets similar performance to neural network (not sure what problem).

  • Good way to give people who don't know neural networks, the power of NNs with conventional neural networks.
  • Also, random forests tend to learn a lot faster than neural networks.

• By plotting city embeddings, the states that are close to each other, were close in lower dimensional space:

• Plotting embeddings hasn't really been widely studied.

00:31:30 - Audience questions

• Q1: How does this approach for generating embeddings work compared to skip grams?

• A1: Skip grams are specific to NLP. They turn an unlabeled problem into labelled by replacing a word from a sentence for some set of sentences, then have the neural network try to figure out if the which ones are changed.

  • Comes from Word2Vec.
  • Jeremy thinks we should pretrain embeddings ourselves and move past Glove and Word2Vec.

• Q2: Is a dummy task required for training embeddings?

• A2: Not necessary, in the Rossman example we used a "real" task - predicting store sales.

• Q3: Does the task matter for generating the embeddings?

• A3: It hasn't been studied enough - lot's of possiblities for that.

• Q4: When training language models, is an embedding learned on one dataset, useful for antoher one?

• Q5: More research is required.

00:41:03 - More on Rossman

• Best not to come into dataset preparation with any assumptions: deleting stores with no sales may have preventing the model learning that days before tend to have more sales

00:43:12 - Audience questions

• Q1: Can you define shallow learning better?
• A1: Anything that doesn't have a hidden layer: just a dot product.

00:43:37 - Rossman cont.

• Public score would have put him at 338th place on the Public Leaderboard, yet 3rd on the Private.

• 3rd place winner deleted all closed days from the dataset: may have won if he had of seen this visualisation that shows the impact of keep closed stores in the dataset. Can see the anomaly with sales before and after closed days:

  • Important to avoid assumptions when preparing the data: let the results speak for themselves.

00:49:04 - Techniques for exploring the code base

• Can get the source code of a class of function using ??: ColumnarModelData??
• Most editors support ability to open stuff directly via SSH and to navigate through jumping from definition to definition.
• In Vim, can use * on some definition to find other uses of it in a file.
• ColumnarModelData.get_learner returns a StructuredLearner, which wraps a MixedInputModel.

00:51:08 - MixedInputModel

00:52:00 - Audience questions

• Q1: Does MixedInputModel always expect categorical embeddings and continuous data?

• A1: Yes, can past an empty lift for either and it will deal with them.

  • Fast.ai doesn't deal with a lot of edge cases because PyTorch has plans to do away with Variables in later versions, which should change a lot of the required hacks.

• Q2: How can we write something that is slightly different from what is in the library?

• A2: Covered next week but mostly in Part 2.

• emb_szs is a list of tuples which include the number of rows (how many unique classes there are) and embedding size (how many elements in the vector).

• __init__ (constructor) walkthrough:

  • The first line of the constructor initialises the embeddings:

    self.embs = nn.ModuleList([nn.Embedding(c, s) for c, s in emb_szs])
    

    • ModuleList is used to register layers inside a list.

  • Later, we create the linear layers, create one initial layer and any additional ones passed in:

    szs = [n_emb+n_cont] + szs
    self.lins = nn.ModuleList([
        nn.Linear(szs[i], szs[i+1]) for i in range(len(szs)-1)])
    

  • We create Dropout layers using a similar approach:

    self.emb_drop = nn.Dropout(emb_drop)
    self.drops = nn.ModuleList([nn.Dropout(drop) for drop in drops])
    

  • And batch norm:

    self.bns = nn.ModuleList([
       nn.BatchNorm1d(sz) for sz in szs[1:]])
    

  • Init layers with kaiming_normal:

    self.bns = nn.ModuleList([nn.BatchNorm1d(sz) for sz in szs[1:]])
    

• forward (forward prop) walkthrough:

  • Iterate through layers and generate predictions:

    for l,d,b in zip(self.lins, self.drops, self.bns):
        x = F.relu(l(x))
        if self.use_bn: x = b(x)
        x = d(x)
    

  • Push final output through signmoid (if an output range is requested):

    x = F.sigmoid(x)
    x = x*(self.y_range[1] - self.y_range[0])
    x = x+self.y_range[0]
    

01:00:00 - Dig into SGD

• Simple example of using SGD to learn y = a*x+b:

• Generate some random data:

• Plot random data:

• Want to recover 3 and 8 using SGD.
• Want to define a loss function.
• Output is continuous output, so problem requires regression. Common loss function is mean squared error.

• If, for example, we thought a=10 and b=5, then the MSE would be as follows:

• We have defined the model (linear regression) and the evaluation criteria (aka loss function), now we need to handle optimisation.

• Generate 1000 examples and turn them into variables:

• Create random weights, wrapped in Variable, settings the gradients to be calculated:

• Do gradient descent loop:

01:07:03 Gradient descent without PyTorch

• Can plot optimization using FuncAnimation:

01:09:15 - RNNs

• Download the collected works of Nietzsche:

• Going to build a character model which learns to write like Nietzsche.

01:10:24 - Why we need RNNs

1. Variable length sequence.

• Very hard with conv nets, very straight forward in RNNs.

2. Long-term dependency

• Want to keep track of some state over long-term dependancies.
  • Modelling code example: model needs to know if it's inside some sort of section, like a function or <div> block.

3. Stateful representation: understand context of what's come previously.
4. Memory

• Example: Swiftkey use RNN to predict next character you might type.

01:13:06 - Jeremy's neural net notation

• Example with simple neural network:

  • First arrow = matrix product followed by relu - generates a set of activations.
  • Second arrow = softmax to generate output activations.

• Notation:
  • Rectangle = input.
  • Circle = hidden layer.
  • Triangle = output layer.
  • Every shape = activations.
  • Arrow = layer operation.

• CNN example:

• Batch size removed from diagram.
• Activation type removed, because it's always quite similar.

1. Perform convolution with stride 2. Leaves you with activation of size num_filters, height / 2, width / 2.
2. Perform a flatten and matrix product, leaving you with number of activations.
3. Perform softmax to generate output.

• RNN example:

• Want to try to predict the 3rd char using an input of the previous 2 chars.

1. Input first character of each string and generate some activations with a matrix product.
2. Put those activations through another matrix product, this time pass in and add the character 2s input to generate the next set of activations.
3. Perform softmax to predict characters.

01:20:38 - Implement char model

• Grab a set of all chars you see, sorted:

• Want to have zero value or null in dataset (for padding)

• Want to map each character to a unique id (and back again):

• idx is the data used for model, which converts char to index

• Can confirm it's working by converting indexes back into characters:

• Create inputs to model (predict the 4th character given the previous 3):

• Create inputs:

• Create output:

• Need to decide how many activations and how many latent factors (aka how big is the embedding matrix):

• Create the 3-char model as a nn.Module.
  • Note: we are creating 3 matrixes. One for characters, one for activations between FC and one for softmax output. Each arrow colour represents the same matrix:

01:29:58 - Audience questions

• Q1: What are the dimensions for each layer?
• A1:
  • self.e = length 42
  • self.l_in will return something of size hidden
  • self.l_hidden is square, so will return size n_hidden too
  • self.l_out returns vocab_size

• Can replace the 4 h = lines in forward, with a for loop, which will makes it a recurrent neural net.

• Only using Fast.ai for the data loader:

• Can call the model as a function by converting tensors to mini-batch:

• Create an optimizer using standard PyTorch:

• m.parameters() returns a list of parameters to optimize.

• Since we don't have learning rate annealing, we can decreate the lr ourselves:

• Write a function to test the model:

• Note that character model currently is a standard FC model, only thing slightly different is including the state of h in the FC layers.

01:37:43 - Implement char model as RNN

• Want to simplify char model diagram using a loop:

• Could be simplified future, by not making first char a special case:

01:39:37 - Audience questions

• Q1: Why reuse the same weight matrices - is it similar to filters in conv nets?
• A1: Sort of. A convolution is a different operation but they do both share a single set of weights.

• This time we'll use 8 chars to predict the 9th char:

• For each 0 through 7, create a list of each 8th char with that starting point:

• Each column is a series of 8 chars from the text:

• Then y is the next char:

01:42:57 - Create and train model

• Why use a hyperbolic tanh (F.tanh)?
  • Very common to use in a state-to-state transition because it prevents values getting too high or too lower.
  • Similar to a sigmoid but appears to return values from -1 to 1.

• Might want to rethink adding the h to inp because you might loose some information in the process. Instead, consider concatting them:

• Can test model as above:

01:48:47 - Implement RNN with PyTorch

• PyTorch has tools to write the RNN loop and build the appropriate linear layers, using the nn.RNN class:

• One minor difference between out implementation and PyTorches: h is a list of activations from each layer. That's why we have to get the last one: outp[-1]

• Test model again:

• Useful to try to reimplement nn.RNN yourself and see if you get the same results out of PyTorch's implementation.

01:55:30 - Multi-output model

• One more change to model: returning an output after every iteration (note where the triangle is placed)

• Change to the for loop would be to accumulate the softmax outputs for each iteration:

  res = []
  for c in cs:
    # Use cat instead of `h+inp`
    inp = torch.cat((h, self.e(c)), 1)
    inp = F.relu(self.l_in(inp))
    h = F.tanh(self.l_hidden(inp))
    res.append(F.log_softmax(self.l_out(h))
  

• Then, instead of picking 8 chars and predicting the 9th, take 8 characters in a single x instance, and use the corresponding chars offset by one in y.
• First, take non-overlapping sets of chars:

• Allows for training model "more efficiently".

• (02:00:41) One complexity: using negative log likelihood loss function, expects to receive 2 rank 1 tensors (with mini-batch: rank 2).
  • In this output, we have 8 timesteps each with 84 probabilities (a rank 3 tensor).
  • Can fix it as follows:

• fit just implementes the training loop: should be all standard PyTorch stuff except the data.

02:06:05 - Audience questions

• Q1. Should the sequence length be much longer in the multi-output model, given the first input only has 0s to go off?
• A1. Ideally, you'd line up the model, so that each batch passes some state to the next batch.
  • Could move the zero initialisation for V(torch.zeros(1, bs, n_hidden)) into the constructor, then it's shared between forward prop calls.

02:08:20 - Using the identity matrix to initialise recurrent weight matrix

• If matrix multiple increases activations each time, it could cause exploding or vanishing gradients.
  • Want to ensure initial l_hidden should be of a size that doesn't cause activations to increase or decrease.
  • Geoffrey Hinton paper A Simple Way To Initialize Recurrent Networks of Rectified Linear Units proposes using identity matrix to initialize recurrent weight matrix.
• Performed in one line below:

• Clearly that helped a lot!