Lesson 4

In [1]:

# Put these at the top of every notebook to get automatic reloading and inline plotting
%reload_ext autoreload
%autoreload 2
%matplotlib inline

import math
import os
from glob import glob
import datetime

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from PIL import Image

from fastai import fastai
from fastai.fastai import transforms
from fastai.fastai import conv_learner
from fastai.fastai import model
from fastai.fastai import dataset
from fastai.fastai import sgdr
from fastai.fastai import plots as faplots
from fastai.fastai import metrics
from fastai.fastai import structured
from fastai.fastai import column_data
from torchvision.models import resnet18, resnet34
from fastai.fastai.torch_imports import resnext101_64
import torch
import dill

00:00:24 - Fast.AI student article recap

00:03:30 - Lessons recap and plan

Lesson 1, 2 and 3 = CNN image intro
Lesson 4 = Structured data, language RNN and collaborative filtering.
Lesson 5 = SGD and loss function + collaborative filtering in depth.

00:04:59 - Dropout

Activation is just a number, calculated using some parameters.
Firstly, examine the precomputed ConvLearner object below, which only includes the last, fully-connected, layers of the model:

In [8]:

PATH = 'data/dogbreed/'
arch = resnet34
sz = 224
bs = 64

In [12]:

tfms = transforms.tfms_from_model(arch, sz, aug_tfms=transforms.transforms_side_on, max_zoom=1.1)
data = dataset.ImageClassifierData.from_csv(
    PATH, 'train', f'{PATH}labels.csv',
    test_name='test', suffix='.jpg', tfms=tfms, bs=bs)
learn = conv_learner.ConvLearner.pretrained(arch, data, ps=0.5, precompute=True)

In [13]:

learn

Out[13]:

Sequential(
  (0): BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True)
  (1): Dropout(p=0.5)
  (2): Linear(in_features=1024, out_features=512, bias=True)
  (3): ReLU()
  (4): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True)
  (5): Dropout(p=0.5)
  (6): Linear(in_features=512, out_features=120, bias=True)
  (7): LogSoftmax()
)

Note the following layers:
- BatchNorm1d - covered in the last lesson.
- Linear - matrix multiplier.
  - A matrix which is size (1024, 512).
  - Takes in 1024 activations and returns 512 activations.
- ReLU - replaces negatives with 0.
- LogSoftmax - put the previous linear layers activations and return log probability.
Dropout with p=0.5, simply means: delete 50% of the activations at random:
Forces the model not to overfit:
- If one activation has learned about an "exact" dog breed, then when it's removed, it's going to hurt the loss.
- Model has to find a fit that works, even without certain activations.
Critical in making modern deep learning work.
- Solved the problem of generalisation.

00:13:24 - Audience questions

Question 1: Do you have to do anything to accomodate for the fact you're throwing away activations?
Answer 1: All handled behind the scenes by PyTorch. If you set p=0.5, PyTorch will double the number of activations to account for the deleted ones.

For minor precision reasons, we take the log of the softmax on the final layer, which means we need to convert it with exp.
You can pass in ps=0.5 to change the Dropout on the added layers, not on the pretrained layers.
Setting ps=0 will remove Dropout all together:

In [14]:

conv_learner.ConvLearner.pretrained(arch, data, ps=0, precompute=True)

Out[14]:

Sequential(
  (0): BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True)
  (1): Linear(in_features=1024, out_features=512, bias=True)
  (2): ReLU()
  (3): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True)
  (4): Linear(in_features=512, out_features=120, bias=True)
  (5): LogSoftmax()
)

Can also pass xtra_fc=[<list of extra FC layers>] as a list of additional fully-connected layers. Empty list would exclude all but the final layer:

In [15]:

conv_learner.ConvLearner.pretrained(arch, data, ps=0, precompute=True, xtra_fc=[])

Out[15]:

Sequential(
  (0): BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True)
  (1): Linear(in_features=1024, out_features=120, bias=True)
  (2): LogSoftmax()
)

00:17:54 - Audience questions

Question 1: What sort of p should we use by default?
Answer 1: Fast.ai defaults to .25 for 2nd layer and 0.5 for last layer - works well for most things.
- Bump it up if overfitting.
- Lower if underfitting.
- Bigger models will generally need more dropout.
Question 2: Setting p=0.5 is that 50%?
Answer 2: Yes.
Question 3: How can you determine if you're overfitting?
Answer 3: Yes, the training error will be lower than the validation.
- Overfitting isn't always a bad thing, as long as your validation loss is going down.
Question 4: Setting p is probabilty of deleting activations, not of keeping activations?
Answer 4: Yes
Question 5: Why does the average activation matter?
Answer 5: If you deleted activations, it will have a knock on effect to later calculations. Your goal is to delete activations, without changing the meaning of future activations.
Question 6: Why are you using a linear activation for an earlier activation?
Answer 6: It's the only choice you have based on what previous layers return.
Question 7: Can you have different dropout by layer?
Answer 7: Pass in array to ps argument:

In [18]:

conv_learner.ConvLearner.pretrained(arch, data, ps=[0.25, 0.125], precompute=True)

Out[18]:

Sequential(
  (0): BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True)
  (1): Dropout(p=0.25)
  (2): Linear(in_features=1024, out_features=512, bias=True)
  (3): ReLU()
  (4): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True)
  (5): Dropout(p=0.125)
  (6): Linear(in_features=512, out_features=120, bias=True)
  (7): LogSoftmax()
)

J doesn't have good intuition for when to use different values off Dropout on final layers.
Question 8: Why measure the loss, instead of accuracy going up?
Answer 8: Because loss is actually being used by SGD so it's good to see it.
Question 9: Does Dropout impact the learning rate?
Answer 9: In theory, but not much in practise.

00:25:04 - Structure data problem (Rossman Kaggle competition)

Main dataset provides the following info: at a particular store, how much did they sell?
See this notebook for data preprocessing.

00:26:58 - Preparing features

Winner from the competition came up with useful columns.
List of things we're going to treat as "categorial variables"
- Could treat year, month and day as continous, but categorical usually works better in practise.
If it's categorical in the data, you'll have to treat it the same in the model. If continuous, you get to pick.

In [28]:

PATH = 'data/rossman/'

In [29]:

cat_vars = ['Store', 'DayOfWeek', 'Year', 'Month', 'Day', 'StateHoliday', 'CompetitionMonthsOpen', 'Promo2Weeks', 'StoreType', 'Assortment', 'PromoInterval', 'CompetitionOpenSinceYear', 'Promo2SinceYear', 'State', 'Week', 'Events', 'Promo_fw', 'Promo_bw', 'StateHoliday_fw', 'StateHoliday_bw', 'SchoolHoliday_fw', 'SchoolHoliday_bw']

In [30]:

contin_vars = [
    'CompetitionDistance', 'Max_TemperatureC', 'Mean_TemperatureC', 'Min_TemperatureC',
    'Max_Humidity', 'Mean_Humidity', 'Min_Humidity', 'Max_Wind_SpeedKm_h',
    'Mean_Wind_SpeedKm_h', 'CloudCover', 'trend', 'trend_DE',
    'AfterStateHoliday', 'BeforeStateHoliday', 'Promo', 'SchoolHoliday'
]

In [31]:

joined = pd.read_feather(f'{PATH}joined')
joined_test = pd.read_feather(f'{PATH}joined_test')

Cardinality refers to number of items in a category: "cardinality of days of week is 7".

00:30:59 - Audience questions

Q1: Heuristic for when to bin continuous variables?
A1: Jeremy doesn't bin continuous variables (process of putting continuous data into categorical "bins"), though apparently it may be useful in certain cases.
Q2: If you're using year as a category, what happens if it's never seen that year?
A2: It'll be treated as an "unknown" year: unknown then just becomes another category.
Q3: If our training dataset doesn't have a category, and test has unknown, how does it work?
A3: It'll probably have learned a way to predict unknowns or will just have a random vector.

00:33:17 - Preparing features cont

Can see we have 844k rows:

In [32]:

n = len(joined); n

Out[32]:

Convert to category using Pandas:
- Note that PyTorch expects a float32, which is why the continuous variables are being converted.

In [33]:

for v in cat_vars: joined[v] = joined[v].astype('category').cat.as_ordered()
for v in contin_vars: joined[v] = joined[v].astype('float32')
    
dep = 'Sales'

joined = joined[cat_vars + contin_vars + [dep, 'Date']]

Run on a sample:

In [34]:

idxs = dataset.get_cv_idxs(n, val_pct=150000/n)
joined_samp = joined.iloc[idxs].set_index('Date')
samp_size = len(joined_samp); samp_size

Out[34]:

Date has some booleans, some integers and some letters that Pandas displays as strings:

In [35]:

joined_samp.head(2)

Out[35]:

	Store	DayOfWeek	Year	Month	Day	StateHoliday	CompetitionMonthsOpen	Promo2Weeks	StoreType	Assortment	...	Max_Wind_SpeedKm_h	Mean_Wind_SpeedKm_h	CloudCover	trend	trend_DE	AfterStateHoliday	BeforeStateHoliday	Promo	SchoolHoliday	Sales
Date
2015-06-16	257	2	2015	6	16	False	24	0	a	a	...	18.0	8.0	6.0	63.0	71.0	22.0	0.0	1.0	0.0	5934
2013-10-30	4	3	2013	10	30	False	24	0	c	c	...	29.0	16.0	2.0	74.0	66.0	27.0	-56.0	0.0	0.0	7800

2 rows × 39 columns

Fast.ai library includes a function called proc_df:
- pulls out dependent variables (in this example Sales).
- Scales dataset to between -1 and 1 by subtracting the mean and dividing by the standard deviation.

In [36]:

df, y, nas, mapper = structured.proc_df(joined_samp, 'Sales', do_scale=True)

Also returns a special object which includes the mean and std for subtracting and dividing the test set.

In [37]:

mapper

Out[37]:

DataFrameMapper(default=False, df_out=False,
        features=[(['CompetitionDistance'], StandardScaler(copy=True, with_mean=True, with_std=True)), (['Max_TemperatureC'], StandardScaler(copy=True, with_mean=True, with_std=True)), (['Mean_TemperatureC'], StandardScaler(copy=True, with_mean=True, with_std=True)), (['Min_TemperatureC'], StandardScaler(co...rue, with_std=True)), (['CloudCover_na'], StandardScaler(copy=True, with_mean=True, with_std=True))],
        input_df=False, sparse=False)

Also handles missing values by setting them to 0 for categorical, or replacing with median variable for continues.

In [38]:

nas

Out[38]:

{'CompetitionDistance': 2320.0, 'CloudCover': 6.0}

In [41]:

y_log = np.log(y)

In [42]:

df.head(2)

Out[42]:

	Store	DayOfWeek	Year	Month	Day	StateHoliday	CompetitionMonthsOpen	Promo2Weeks	StoreType	Assortment	...	Mean_Wind_SpeedKm_h	CloudCover	trend	trend_DE	AfterStateHoliday	BeforeStateHoliday	Promo	SchoolHoliday	CompetitionDistance_na	CloudCover_na
Date
2015-06-16	257	2	3	6	16	1	25	1	1	1	...	-0.651394	0.266629	-0.194358	0.487305	-0.525077	1.109608	1.115768	-0.489079	-0.050661	-0.294389
2013-10-30	4	3	1	10	30	1	25	1	3	3	...	0.700532	-2.197401	0.769911	-0.026011	-0.363976	-0.602128	-0.896244	-0.489079	-0.050661	-0.294389

2 rows × 40 columns

00:38:36 - Selecting validation sets with timeseries data

Cross-val should not be random with time series data.
Generally use the most recent data for holdout set, like real life.
- Could take the last 25% of rows, as follows:

In [43]:

train_ratio = 0.75
train_size = int(samp_size * train_ratio); train_size
val_idx = list(range(train_size, len(df)))

A better option, for this case, would be to get the same time period as the test set:

In [44]:

val_idx = np.flatnonzero(
    (df.index<=datetime.datetime(2014,9,17)) & (df.index>=datetime.datetime(2014, 8, 1)))

Rachel posted an article on creating validations set here.

00:39:35 - Understand metric for evaluating model

Kaggle use root mean squared percent error as the evaluation metric for Rossman competition: $\text{RMSPE }=\sqrt{\frac{1}{n} \sum\limits_{i=1}^{n}{(\frac{y_i-\hat{y}_i}{y_i})}^2}$
- Since we know that $log(\frac{a}{b})=log(a)-log(b)$:
```
>> math.log(10/5)
>> 0.6931471805599453

>> math.log(10) - math.log(5)
>> 0.6931471805599456
```
- If you take the log of the data, the root mean squared error will actually get you the root mean squared percent error. You can inverse the results with exp.

In [45]:

def inv_y(a): return np.exp(a)

def exp_rmspe(y_pred, targ):
    targ = inv_y(targ)
    pct_var = (targ - inv_y(y_pred)) / targ
    return math.sqrt((pct_var ** 2).mean())

max_log_y = np.max(y1)
y_range = (0, max_log_y * 1.2)

Create a ModelData object using the data frame with ColumnarModelData.from_data_frame

In [46]:

md = column_data.ColumnarModelData.from_data_frame(PATH, val_idx, df, y_log, cat_flds=cat_vars, bs=128)

PATH - path where you want to store any temp files or models etc.
val_idx - list of rows to put in validation set.
df - DataFrame.
yl - dependent variables.
cat_flds - which fields should be considered categorical?
bs - batch size.

Get number of levels of each categorical var:

In [23]:

cat_sz = [
    (c, len(joined_samp[c].cat.categories) + 1) for c in cat_vars]

In [24]:

cat_sz

Out[24]:

[('Store', 1116),
 ('DayOfWeek', 8),
 ('Year', 4),
 ('Month', 13),
 ('Day', 32),
 ('StateHoliday', 3),
 ('CompetitionMonthsOpen', 26),
 ('Promo2Weeks', 27),
 ('StoreType', 5),
 ('Assortment', 4),
 ('PromoInterval', 4),
 ('CompetitionOpenSinceYear', 24),
 ('Promo2SinceYear', 9),
 ('State', 13),
 ('Week', 53),
 ('Events', 22),
 ('Promo_fw', 7),
 ('Promo_bw', 7),
 ('StateHoliday_fw', 4),
 ('StateHoliday_bw', 4),
 ('SchoolHoliday_fw', 9),
 ('SchoolHoliday_bw', 9)]

00:45:39 - Embeddings

Firstly continuous variables:
- Bunch of floating point numbers in vector, for example one with 20 columns. (1, 20)
- Then put through a matrix product with anther matrix with a matching number of rows to the vectors columns: (20, 100)
- That returns a vector with 100 columns: (1, 100), which are then put through a relu activation.
- Then put through a relu activations.
- Lasly, put through another matrix multiplication, of (100, 1) which returns a single value: (1, 1).
Aside from adding dropout and stuff, there aren't many interesting architectural choices when using fully-connected layers.

Categorical:
- For each set of categorical vars (eg day of week): Create another matrix with n columns and a row for each category.
  - Called an "embedding matrix".
- Then, at training time, pull out the appropriate row and concat to continuous variables.
- Basically, it's a way to convert categorical variables into continuous numbers.
- Side effect of the embedding is that they're sometimes human interpretable and can be quite interesting.

00:55:56 - Audience questions

Q1: Is there a heuristic for selecting the size of embeddings?
A1: Half the cardinality of category (eg 8//2=4) fields for day of week), as long as it's no greater than 50.

In [ ]:

emb_szs = [(c, min(50, (c+1)//2)) for _, c in cat_sz]

In [49]:

emb_szs

Out[49]:

[(1116, 50),
 (8, 4),
 (4, 2),
 (13, 7),
 (32, 16),
 (3, 2),
 (26, 13),
 (27, 14),
 (5, 3),
 (4, 2),
 (4, 2),
 (24, 12),
 (9, 5),
 (13, 7),
 (53, 27),
 (22, 11),
 (7, 4),
 (7, 4),
 (4, 2),
 (4, 2),
 (9, 5),
 (9, 5)]

Q2: Would you always build embedding matrix for each categorical feature?
A2: Yes.
Q3: Besides random initialisation, is there another way to init embedding?
A3: You could either use random, or find another pretrained embedding matrix. Perhaps one has been trained to predict similar products.
Q4: Why not just use a one-hot encoded values instead of embedding?
A4: You could just pass in 7 vars, 6 of them 0 and 1 of them 1. Problem: concept of Sunday, could only be associated with a single floating point number. With embeddings, it becomes a concept in 4 dimensional space, which can give them "rich semantic concepts".
- Examples:
  - weekends may have a different behaviour, causing some number to be higher.
  - one column may represent certain products selling better on certain days.

01:07:14 - Dealing with dates

Fast.ai function called add_datepart that takes a dataframe and a datefield column name.
- It then removes the date and replaces it with categorical variables about the date: day of week, is it end of quarter, start of month etc.

In [52]:

date_df = pd.DataFrame({'Date': [datetime.datetime(2017, 3, 1), datetime.datetime(2017, 3, 2)]})

In [53]:

date_df.head()

Out[53]:

	Date
0	2017-03-01
1	2017-03-02

In [54]:

structured.add_datepart(date_df, 'Date', drop=False)

In [55]:

date_df.head()

Out[55]:

	Date	Year	Month	Week	Day	Dayofweek	Dayofyear	Is_month_end	Is_month_start	Is_quarter_end	Is_quarter_start	Is_year_end	Is_year_start	Elapsed
0	2017-03-01	2017	3	9	1	2	60	False	True	False	False	False	False	1488326400
1	2017-03-02	2017	3	9	2	3	61	False	False	False	False	False	False	1488412800

01:10:05 - Training model

In [64]:

m = md.get_learner(
    emb_szs=emb_szs,
    n_cont=len(df.columns)-len(cat_vars),
    emb_drop=0.04,
    out_sz=1,
    szs=[1000, 500],
    drops=[0.001, 0.01],
    y_range=y_range
)

lr = 1e-3

emb_szs - embedding matrix sizes.
n_cont - number of continous variables.
emb_drop - how much dropout to use.
out_sz - output size.
szs - how many activations to use.
drops - how much dropout to use in the later layers.

In [58]:

emb_szs

Out[58]:

[(1116, 50),
 (8, 4),
 (4, 2),
 (13, 7),
 (32, 16),
 (3, 2),
 (26, 13),
 (27, 14),
 (5, 3),
 (4, 2),
 (4, 2),
 (24, 12),
 (9, 5),
 (13, 7),
 (53, 27),
 (22, 11),
 (7, 4),
 (7, 4),
 (4, 2),
 (4, 2),
 (9, 5),
 (9, 5)]

Can now do the standard stuff, like getting the learning rate:

In [59]:

m.lr_find()

HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))

 66%|██████▌   | 735/1120 [00:50<00:26, 14.70it/s, loss=0.329]

In [60]:

m.sched.plot(100)

In [ ]:

lr = 1e-3

Note that we're passing in metrics to be printed at the end of each epoch.

In [65]:

m.fit(lr, 3, metrics=[exp_rmspe])

HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))

epoch      trn_loss   val_loss   exp_rmspe                      
    0      0.026518   0.022992   0.159085  
    1      0.018788   0.029464   0.152526                       
    2      0.017182   0.021571   0.13363

Out[65]:

[0.021571023, 0.13362987312569344]

In [66]:

m.fit(lr, 3, metrics=[exp_rmspe], cycle_len=1)

HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))

epoch      trn_loss   val_loss   exp_rmspe                      
    0      0.012154   0.01711    0.121186  
    1      0.011085   0.018633   0.12522                        
    2      0.010783   0.01749    0.121782

Out[66]:

[0.017490305, 0.1217821107997797]

The winners of the Kaggle competition did a lot more feature engineering. Generally deep learning approaches require somewhat less feature engineering.

01:15:02 - Audience questions

Q1: Are we using time series in any of the fits?
A1: Yes, by converting the date data into categorical variables.
Q2: In the earlier CNNs we didn't pass data during the fit. Is that the case here?
A2: We aren't passing in anything at fit time. The model that we build depends on the data, hence the learner being returned from the data: m.get_learner.
Q3: Do I just need to figure out what to put into categorical variables / embeddings on my own dataset?
A3: Steps for your own dataset:
1. Put your data in a Pandas dataframe and list the categorical variables and continuous variables.
2. Figure out which rows go in the validation dataset.
3. Call ColumnarModelData.from_data_frame(PATH, val_idx, df, yl, cat_flds=cat_vars, bs=128, passing in your data.
4. Create your list of how big your embedding matrices should be.
5. Call get_learner to get your learner.
6. Call fit to fit the data.
Q4: How can data augmentation be used here?
A4: Data augmentation doesn't really exist with structured data yet.

01:23:34 - Natural Language Processing

NLP is the most up-and-coming area of Deep Learning, about 2-3 years behind image classification.
Language modelling:
- build a model where given a few words of a sentence, can you predict the next word?
- Swiftkey used DL for it.
Language models can be used in classification tasks (like sentiment analysis).

01:31:35 - Audience questions

Q1: Why not just learn the classification model without a language model?
A1: We already know that transfer learning is useful. Also, the IMDB reviews are sometimes quite large and it's too much to try to learn English and how to predict sentiment.
Q2: Is this similar to the char-rnn?
A2: Yes, except language models usually work at a word level, rather than a character level: they tend to be more powerful.
Q3: To what extent are these generated words and sentences seen before or randomly created?
A3: All the words must have been seen before. The sentences generated would be similar but not exact to what's generated.

01:36:37 - New imports

Torchtext - Pytorch's NLP library.

In [2]:

import torchtext
from torchtext import vocab, data
from torchtext.datasets import language_modeling

Some new text-specific fast.ai stuff:

In [3]:

from fastai.fastai.rnn_reg import *
from fastai.fastai.rnn_train import *
from fastai.fastai.nlp import *
from fastai.fastai.lm_rnn import *

import dill

Going to be working with IMDB large movie review dataset.
- Very well studied in academia.
- 50k reviews.
Going to start by creating a language model first.

In [4]:

PATH = 'data/aclImdb/'

TRN_PATH = 'train/all/'
VAL_PATH = 'test/all/'

TRN = f'{PATH}{TRN_PATH}'
VAL = f'{PATH}{VAL_PATH}'

In [6]:

!wget http://files.fast.ai/data/aclImdb.tgz --directory-prefix=data

--2018-04-20 04:56:49--  http://files.fast.ai/data/aclImdb.tgz
Resolving files.fast.ai (files.fast.ai)... 67.205.15.147
Connecting to files.fast.ai (files.fast.ai)|67.205.15.147|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 145982645 (139M) [text/plain]
Saving to: ‘data/aclImdb.tgz’

aclImdb.tgz         100%[===================>] 139.22M  11.9MB/s    in 21s     

2018-04-20 04:57:11 (6.61 MB/s) - ‘data/aclImdb.tgz’ saved [145982645/145982645]

In [7]:

!tar -xzf data/aclImdb.tgz -C data

01:37:54 - Analysing IMDB dataset

In [8]:

!ls {PATH}

imdbEr.txt  imdb.vocab	README	test  train

Look inside training folder:

In [9]:

trn_files = !ls {TRN}
trn_files[:10]

Out[9]:

['0_0.txt',
 '0_3.txt',
 '0_9.txt',
 '10000_0.txt',
 '10000_4.txt',
 '10000_8.txt',
 '1000_0.txt',
 '10001_0.txt',
 '10001_10.txt',
 '10001_4.txt']

Example review:

In [10]:

review = !cat {TRN}{trn_files[7]}
review[0]

Out[10]:

"Everybody has seen 'Back To The Future,' right? Whether you LIKE that movie or not, you've seen an example of how to make a time-travel movie work. A torn-up poster for 'Back To The Future' shows up in this movie, representing, perhaps unintentionally, what the makers of 'Tangents' (aka 'Time Chasers') did to the time-travel formula. Then again, the movie claims to have been made in 1994, but it looks -- and sounds -- like it was produced at least ten years earlier, so maybe they achieved time-travel after all.<br /><br />Start with an intensely unappealing leading man. I mean, what woman doesn't love gangly, whiny, lantern-jawed, butt-chinned, mullet-men with Coke-bottle glasses? Oh, none of you? Prepare to tough it out, ladies, cuz that's what this movie gives you.<br /><br />Second, add a leading lady who -- while not entirely unattractive -- represents many '80s clichés: big hair, too much makeup, two different plaids, shoulder pads, acid-washed mom-jeans, etc.<br /><br />Throw in a Michael Medved look-alike who wears pink blazers with white pants, a stunningly transparent villain who talks like Mortimer Snerd and has an office that looks like a circus-themed library, and evil henchmen who have nothing better to do than direct air traffic. That's our cast, folks. Enjoy!<br /><br />I could try to explain the plot, but it will take a lot less time for you to just track down a copy of this movie and watch it yourself. If YOU figure out the plot, please don't hesitate to share it with me.<br /><br />I would strongly advise watching this movie with the help of the folks at Mystery Science Theater 3000. I don't think it could stand on its own.<br /><br />The film, 'Tangents': 3 stars -- at least they tried.<br /><br />MST3K's 'Time Chasers' episode: 8 stars -- they actually succeeded."

Check how many words in the dataset:

In [11]:

!find {TRN} -name '*.txt' | xargs cat | wc -w

17486581

In [12]:

!find {VAL} -name '*.txt' | xargs cat | wc -w

Need to convert text into a list of numbers, first step is to convert into a list of words:

In [13]:

!python -m spacy download en

Collecting https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz
  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz (37.4MB)
    100% |████████████████████████████████| 37.4MB 81.8MB/s ta 0:00:0110% |▏                               | 174kB 438kB/s eta 0:01:25    58% |██████████████████▊             | 21.9MB 3.1MB/s eta 0:00:06
Installing collected packages: en-core-web-sm
  Running setup.py install for en-core-web-sm ... done
Successfully installed en-core-web-sm-2.0.0
You are using pip version 9.0.3, however version 10.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

    Linking successful
    /home/ubuntu/src/anaconda3/envs/fastai/lib/python3.6/site-packages/en_core_web_sm
    -->
    /home/ubuntu/src/anaconda3/envs/fastai/lib/python3.6/site-packages/spacy/data/en

    You can now load the model via spacy.load('en')

In [5]:

spacy_tok = spacy.load('en')

In [15]:

[t for t in spacy_tok(review[0])][:10]

Out[15]:

[Everybody, has, seen, ', Back, To, The, Future, ,, ']

With Torchtext, need to create a "field".
- Field is a definition of how to process text.
- In this example, we're telling it to tokenise everything and convert to lower case:

In [14]:

TEXT = data.Field(lower=True, tokenize='spacy')

Need to use dill to dump the TEXT object, since it doesn't work with Pickle.

In [29]:

!mkdir {PATH}models

mkdir: cannot create directory ‘data/aclImdb/models’: File exists

In [17]:

dill.dump(TEXT, open(f'{PATH}models/TEXT.pkl', 'wb'))

In [97]:

TEXT = dill.load(open(f'{PATH}models/TEXT.pkl', 'rb'))

Can now create a ModelData object:

In [16]:

bs=64; bptt=70

In [17]:

FILES = dict(train=TRN_PATH, validation=VAL_PATH, test=VAL_PATH)
md = LanguageModelData.from_text_files(PATH, TEXT, **FILES, bs=bs, bptt=bptt, min_freq=10)

PATH - standard path.
TEXT - torchtext field definition.
**FILES - location of training and validation files.
min_freq=10 - any word that occur less than 10 times, ignore.
bptt - stands for "back prop through time": how long are the sentences we'll put on the GPU at a time.
Model words as follows:
- Fills the TEXT object with a vocab attribute.
  - Vocab refers to unique words found in text.

Print # batches; # unique tokens; # tokens in the training set; # sentences

In [24]:

len(md.trn_dl), md.nt, len(md.trn_ds), len(md.trn_ds[0].text)

Out[24]:

(4583, 37392, 1, 20540756)

Map integer IDs to unique tokens:

In [25]:

TEXT.vocab.itos[:12]

Out[25]:

['<unk>', '<pad>', 'the', ',', '.', 'and', 'a', 'of', 'to', 'is', 'in', 'it']

Convert words to ids:

In [26]:

TEXT.vocab.stoi['the']

Out[26]:

In [27]:

TEXT.vocab.itos[2]

Out[27]:

'the'

In LanguageModelData there is only one item for each dataset, which is all words of text joined together:

In [28]:

md.trn_ds[0].text[:12]

Out[28]:

['picking',
 'up',
 'the',
 'jacket',
 'of',
 'this',
 'dvd',
 'in',
 'the',
 'video',
 'store',
 'i']

torchtext handles turning words into integer automatically:

In [29]:

TEXT.numericalize([md.trn_ds[0].text[:12]], device=-1)

Out[29]:

Variable containing:
 3701
   68
    2
 5641
    7
   13
  300
   10
    2
  392
 1089
   12
[torch.LongTensor of size 12x1]

In [30]:

TEXT.vocab.itos[1003]

Out[30]:

'admit'

01:45:47 - Audience questions

Q1: Common to do stemming or lemmatising?
A1: Not really - usually best to leave it alone as much as possible.
Q2: Is context important?
A2: Context is very important. Hence why the words are ordered: this isn't bag-of-words.

01:47:52 - Batching

Start with the full corpus of reviews concatenated.
Break the words into n sections.
Create matrix, where each column is one of these sections.
Row each batch, take something like 70 rows, so you have a matrix that's (70, 64). Which would give you 64 different document fragments.

LanguageModelData object create batches with 64 columns (batch size) and varying sequence lengths.

In [31]:

next(iter(md.trn_dl))

Out[31]:

(Variable containing:
   3701    297      9  ...    1667     12     14
     68  13351    359  ...      43    173     33
      2     51     24  ...     472      6    324
         ...            ⋱           ...         
      7     91    157  ...       3      6     83
     13   1001     56  ...     500   1943    867
     25     52      7  ...      13      7     54
 [torch.cuda.LongTensor of size 69x64 (GPU 0)], Variable containing:
     68
  13351
    359
   ⋮   
   1659
   1310
     13
 [torch.cuda.LongTensor of size 4416 (GPU 0)])

Note that the size is not exactly 70 row (in that example it's 69 rows). Torchtext randomly changes the row size when doing bptt.
Couple of examples of the data in the columns:

In [32]:

batch = next(iter(md.trn_dl))

In [33]:

print(' '.join(TEXT.vocab.itos[int(w)] for w in batch[0][:,0]))

picking up the jacket of this dvd in the video store i was intrigued . having watched this sorry excuse for a western i think director dwight should give a medal for the guy who designed the jacket and has lured to their doom other unsuspecting viewers . the script of the film is potentially interesting ... and i bet that 's what the impressive members of this film 's

In [34]:

print(' '.join(TEXT.vocab.itos[int(w)] for w in batch[0][:,1]))

plays psychos or really bad people . it 's funny that this would later lead him play a true loon like in " 12 monkeys " and that he would be on the other end of the spectrum in david fincher 's " seven " . <eos> when elvis filmed ' speedway ' in june of 1967 it was a time of change . he had been married just over

01:55:50 - Training model

Want to create an embedding matrix with 200 cols with a row for each token.

In [11]:

em_sz = 200 # size of each embedding vector
nh = 500    # number of hidden activations per layer
nl = 3      # number of layers
bs = 64

Large amounts of Momentum don't work that well, so we create a version of Adam will less momentum (default is 0.9)

In [14]:

opt_fn = partial(optim.Adam, betas=(0.7, 0.99))

Create the learner:
- Fast.ai uses state of the art AWD LSTM Language Model by Stephen Merity, which uses lots of Dropout. No known ways to find best dropout - need to experiment.

In [18]:

learner = md.get_model(
    opt_fn, em_sz, nh, nl, dropouti=0.05, dropout=0.05,
    wdrop=0.1, dropoute=0.02, dropouth=0.05)
learner.reg_fn = partial(seq2seq_reg, alpha=2, beta=1)

Note the multiple dropout arguments (details available in the last lecture).
- If you are overfitting, increase all dropout params, decrease if underfitting.

Clip the weights to ensure weights don't explode:

In [19]:

learner.clip = 0.3

Train the model:

In [ ]:

learner.fit(3e-3, 4, wds=1e-6, cycle_len=1, cycle_mult=2)

HBox(children=(IntProgress(value=0, description='Epoch', max=15), HTML(value='')))

epoch      trn_loss   val_loss                                 
    0      4.825006   4.709303  
    1      4.649139   4.522534                                
    2      4.53842    4.440321                                
    3      4.592644   4.470359                                
    4      4.513704   4.394659                                
 77%|███████▋  | 3551/4583 [16:28<04:47,  3.59it/s, loss=4.45]

02:02:28 - Audience questions

Q1: How does this compare to using Word2Vec or Glove?
A1: This technique appears to be more powerful.
Q2: What is the model architecture used?
A2: A RNN using an LSTM (long short term memory).

In [ ]:

learner.save_encoder('adam1_enc')

In [39]:

learner.load_encoder('adam1_enc')

In [40]:

learner.fit(3e-3, 4, wds=1e-6, cycle_len=10, cycle_save_name='adam3_10')

HBox(children=(IntProgress(value=0, description='Epoch', max=40), HTML(value='')))

epoch      trn_loss   val_loss                                
    0      4.470424   4.371257  
    1      4.467573   4.365174                                
    2      4.448483   4.341675                                
    3      4.424094   4.320922                                
 49%|████▉     | 2251/4583 [10:21<10:44,  3.62it/s, loss=4.41]

IOPub message rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_msg_rate_limit`.

Current values:
NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)
NotebookApp.rate_limit_window=3.0 (secs)

    5      4.375303   4.275402                                
    6      4.351441   4.255511                                
    7      4.304695   4.238498                                
    8      4.278119   4.230899                                
    9      4.290093   4.229005                                
 92%|█████████▏| 4214/4583 [19:30<01:42,  3.60it/s, loss=4.46]

IOPub message rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_msg_rate_limit`.

Current values:
NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)
NotebookApp.rate_limit_window=3.0 (secs)

 78%|███████▊  | 3566/4583 [16:30<04:42,  3.60it/s, loss=4.45]

IOPub message rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_msg_rate_limit`.

Current values:
NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)
NotebookApp.rate_limit_window=3.0 (secs)

 66%|██████▌   | 3012/4583 [13:56<07:16,  3.60it/s, loss=4.44]

IOPub message rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_msg_rate_limit`.

Current values:
NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)
NotebookApp.rate_limit_window=3.0 (secs)

 53%|█████▎    | 2440/4583 [11:18<09:56,  3.59it/s, loss=4.42]

IOPub message rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_msg_rate_limit`.

Current values:
NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)
NotebookApp.rate_limit_window=3.0 (secs)

 41%|████      | 1878/4583 [08:41<12:31,  3.60it/s, loss=4.39]

IOPub message rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_msg_rate_limit`.

Current values:
NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)
NotebookApp.rate_limit_window=3.0 (secs)

    14     4.364251   4.277149                                
 28%|██▊       | 1292/4583 [05:58<15:12,  3.61it/s, loss=4.36]

IOPub message rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_msg_rate_limit`.

Current values:
NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)
NotebookApp.rate_limit_window=3.0 (secs)

    15     4.341483   4.257161                                
 16%|█▌        | 714/4583 [03:18<17:57,  3.59it/s, loss=4.34]

IOPub message rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_msg_rate_limit`.

Current values:
NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)
NotebookApp.rate_limit_window=3.0 (secs)

    16     4.314206   4.237568                                
  3%|▎         | 135/4583 [00:37<20:43,  3.58it/s, loss=4.3]

IOPub message rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_msg_rate_limit`.

Current values:
NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)
NotebookApp.rate_limit_window=3.0 (secs)

 90%|████████▉ | 4121/4583 [19:04<02:08,  3.60it/s, loss=4.27]

IOPub message rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_msg_rate_limit`.

Current values:
NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)
NotebookApp.rate_limit_window=3.0 (secs)

 76%|███████▋  | 3502/4583 [16:12<05:00,  3.60it/s, loss=4.25]

IOPub message rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_msg_rate_limit`.

Current values:
NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)
NotebookApp.rate_limit_window=3.0 (secs)

 63%|██████▎   | 2883/4583 [13:23<07:53,  3.59it/s, loss=4.25]

IOPub message rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_msg_rate_limit`.

Current values:
NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)
NotebookApp.rate_limit_window=3.0 (secs)

 42%|████▏     | 1929/4583 [08:57<12:19,  3.59it/s, loss=4.45]

IOPub message rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_msg_rate_limit`.

Current values:
NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)
NotebookApp.rate_limit_window=3.0 (secs)

    20     4.45407    4.330963                                
    21     4.431024   4.325056                                
    22     4.41102    4.308909                                
    23     4.394272   4.28938                                 
    24     4.360949   4.267609                                
    25     4.332253   4.245154                                
    26     4.302126   4.226168                                
    27     4.273302   4.212529                                
    28     4.249753   4.205445                                
    29     4.238341   4.203876                                
    30     4.435757   4.320856                                
    31     4.427365   4.311517                                
    32     4.399558   4.299383                                
    33     4.376254   4.277548                                
    34     4.349744   4.259276                                
    35     4.314749   4.236638                                
    36     4.283758   4.218967                                
    37     4.27303    4.206128                                
    38     4.242893   4.198182                                
    39     4.247306   4.196779

Out[40]:

[array([4.19678])]

In [41]:

learner.save_encoder('adam1_10_enc')

In [42]:

learner.fit(3e-3, 1, wds=1e-6, cycle_len=20, cycle_save_name='adam3_20')

HBox(children=(IntProgress(value=0, description='Epoch', max=20), HTML(value='')))

epoch      trn_loss   val_loss                                
    0      4.244789   4.198284  
    1      4.427986   4.316352                                
    2      4.411712   4.310209                                
    3      4.406754   4.305369                                
    4      4.404938   4.297316                                
    5      4.386481   4.287112                                
    6      4.386738   4.280646                                
    7      4.374347   4.271933                                
    8      4.352447   4.256832                                
    9      4.344912   4.248394                                
    10     4.321443   4.236432                                
    11     4.312613   4.225651                                
    12     4.280505   4.215626                                
    13     4.282414   4.206636                                
    14     4.25259    4.198179                                
    15     4.235448   4.192406                                
    16     4.239449   4.188233                                
    17     4.216629   4.184596                                
    18     4.213963   4.183657                                
    19     4.206912   4.183356

Out[42]:

[array([4.18336])]

In [43]:

learner.save_encoder('adam1_20_enc')

In [73]:

learner.load_encoder('adam1_20_enc')

In [122]:

learner.fit(3e-3, 1, wds=1e-6, cycle_len=10)

HBox(children=(IntProgress(value=0, description='Epoch', max=10), HTML(value='')))

epoch      trn_loss   val_loss                                
    0      4.402206   4.306556  
    1      4.399321   4.296581                                
    2      4.396214   4.284479                                
    3      4.35387    4.26331                                 
    4      4.328579   4.244382                                
    5      4.30673    4.223781                                
    6      4.279538   4.206563                                
    7      4.263763   4.193859                                
    8      4.242565   4.18583                                 
    9      4.21544    4.184016

Out[122]:

[array([4.18402])]

In [123]:

learner.save_encoder('adam3_10_enc')

In [20]:

learner.load_encoder('adam3_10_enc')

02:04:53 - Evaluate model

In [124]:

m=learner.model
ss=""". So, it wasn't quite was I was expecting, but I really liked it anyway! The best"""
s = [[i.text for i in spacy_tok(ss)]]
t=TEXT.numericalize(s)
' '.join(s[0])

Out[124]:

". So , it was n't quite was I was expecting , but I really liked it anyway ! The best"

Manual steps to test language model:

In [125]:

# Set batch size o 1
m[0].bs = 1

# Turn off Dropout
m.eval()

# Reset hidden state
m.reset()

# Get predictions from model
res, *_ = m(t)

# Put batch size batch to what it was
m[0].bs=bs

See top 10 predictions for next word:

In [126]:

nexts = torch.topk(res[-1], 10)[1]
[TEXT.vocab.itos[o] for o in to_np(nexts)]

Out[126]:

['thing',
 'was',
 'movie',
 'of',
 ',',
 'film',
 'is',
 'performance',
 'scene',
 '<unk>']

In [127]:

print(ss,"\n")
for i in range(50):
    n = res[-1].topk(2)[1]
    n = n[1] if n.data[0] == 0 else n[0]
    print(TEXT.vocab.itos[n.data[0]], end=' ')
    res,*_ = m(n[0].unsqueeze(0))
print('...')

. So, it wasn't quite was I was expecting, but I really liked it anyway! The best 

thing , and the film is a bit of a disappointment . the film is a bit of a disappointment , but it 's a good film . <eos> i have to say that i am a huge fan of the original series . i have seen the original , ...

02:05:10 - Use pretrained language model for classification

Sentiment model must use the same vocab as language model.

In [6]:

TEXT = pickle.load(open(f'{PATH}models/TEXT.pkl','rb'))

In [7]:

IMDB_LABEL = data.Field(sequential=False)
splits = torchtext.datasets.IMDB.splits(TEXT, IMDB_LABEL, 'data/')

In [8]:

t = splits[0].examples[0]

In [9]:

t.label, ' '.join(t.text[:16])

Out[9]:

('pos',
 'this modern film noir with its off beat humour and dizzying succession of plot twists delivers')

Fast.ai can create a ModelData object straight from torchtext splits:

In [21]:

md2 = TextData.from_splits(PATH, splits, bs)

In [22]:

m3 = md2.get_model(opt_fn, 1500, bptt, emb_sz=em_sz, n_hid=nh, n_layers=nl, 
           dropout=0.1, dropouti=0.4, wdrop=0.5, dropoute=0.05, dropouth=0.3)
m3.reg_fn = partial(seq2seq_reg, alpha=2, beta=1)
m3.load_encoder(f'adam3_10_enc')

Increase the max gradient clipping to get SGDR to perform better:

In [ ]:

m3.clip = 25.

Use differential learning rates:

In [137]:

lrs=np.array([1e-4,1e-4,1e-4,1e-3,1e-2])

Want to freeze all but the last layers:

In [138]:

m3.freeze_to(-1)
m3.fit(lrs/2, 1, metrics=[accuracy])

HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))

epoch      trn_loss   val_loss   accuracy                    
    0      0.671642   0.397493   0.833486

Out[138]:

[array([0.39749]), 0.8334862434097735]

Can then unfreeze layers:

In [139]:

m3.unfreeze()
m3.fit(lrs, 1, metrics=[accuracy], cycle_len=1)

HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))

epoch      trn_loss   val_loss   accuracy                    
    0      0.497285   0.291096   0.882659

Out[139]:

[array([0.2911]), 0.8826590023739705]

In [140]:

m3.fit(lrs, 7, metrics=[accuracy], cycle_len=2, cycle_save_name='imdb2')

HBox(children=(IntProgress(value=0, description='Epoch', max=14), HTML(value='')))

epoch      trn_loss   val_loss   accuracy                    
    0      0.408477   0.269311   0.902395  
    1      0.392739   0.248186   0.905367                    
    2      0.365437   0.258713   0.906903                    
    3      0.354326   0.25865    0.905984                    
    4      0.342144   0.271602   0.910488                    
    5      0.336043   0.252554   0.911087                    
    6      0.317494   0.248025   0.916649                    
    7      0.318077   0.246904   0.917465                    
    8      0.312234   0.225775   0.920106                    
    9      0.290931   0.243646   0.917602                    
    10     0.277402   0.283826   0.917648                    
    11     0.272603   0.248851   0.920308                    
    12     0.29203    0.260669   0.919272                    
    13     0.272543   0.266593   0.919176

Out[140]:

[array([0.26659]), 0.9191755028207785]

In [ ]:

m3.fit(lrs, 1, metrics=[accuracy], cycle_len=1)

HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))

In [ ]:

m3.save('split_model_v1')

In [25]:

m3.load('split_model_v1')

02:08:00 - Evaluate performance

In [26]:

accuracy_np(*m3.predict_with_targs())

Out[26]:

0.92404

Jeremy manages to receive 94.5%, which is better than the state of the art according the Bradbury et al paper, Learned in translation: contextualized word vectors.

02:11:39 - Intro to collaborative filtering

Very little new to learn: most things have already been covered in earlier parts of the course.
See Lesson 5 notebook for walkthrough.

In [ ]:

Fast.ai: Deep Learning 2018 - Lesson 4

Lesson 4

00:00:24 - Fast.AI student article recap

00:03:30 - Lessons recap and plan

00:04:59 - Dropout

00:13:24 - Audience questions

00:17:54 - Audience questions

00:25:04 - Structure data problem (Rossman Kaggle competition)

00:26:58 - Preparing features

00:30:59 - Audience questions

00:33:17 - Preparing features cont

00:38:36 - Selecting validation sets with timeseries data

00:39:35 - Understand metric for evaluating model

00:45:39 - Embeddings

00:55:56 - Audience questions

01:07:14 - Dealing with dates

01:10:05 - Training model

01:15:02 - Audience questions

01:23:34 - Natural Language Processing

01:31:35 - Audience questions

01:36:37 - New imports

01:37:54 - Analysing IMDB dataset

01:45:47 - Audience questions

01:47:52 - Batching

01:55:50 - Training model

02:02:28 - Audience questions

02:04:53 - Evaluate model

02:05:10 - Use pretrained language model for classification

02:08:00 - Evaluate performance

02:11:39 - Intro to collaborative filtering