fastai - Lesson 7 - Deep Learning for Coders (2020)

00:00:00 - Weight decay

• Issue from last lesson: loss started to go down then up again.
  • We're overfitting.
• Note that we don't need a metric because mean-squared error is close enough to what we care about.
• We reduce overfitting by reducing the capacity of the model
  • Ensure it doesn't memorise the dataset.
  • Reducing params does bias model towards simple shapes.
• Instead, force params to be smaller, unless they need to be big.
• Weight decay, also known as L2 regularisation, adds the sum of all params squared to loss.
  • Tries to encourage weights to be as small as possible: loss function is incentivised to keep weights small.
• Can simply add the weights to the loss function:

loss_with_wd = loss + wd * (parameters ** 2).sum()

• Since the loss function is used to take the gradient, and we know the derivative of p**2 with respect to p is 2*p, we can just update the gradients:

weight.grad += wd * 2 * weight

• In fastai, you can just pass wd amount to the fit call:

• Modern deep learning we prefer weight decay vs simplier models.

00:07:10 - Writing our own embedddings layer

• To create our own layer, it needs to inherit from nn.Module
• We can get a list of parameters using the parameters() method.
  • To ensure something in a module needs to be learned, you wrap in nn.Parameter

• Create a randomly initialised tensor as param:

00:12:31 - Interpreting embeddings and biases

• You can call learn.model to get model in learner.
• We can interpret bias initially.
• Start by printing movie with smallest bias.

• These are the movies that people liked the least. Even people that like this genre didn't like the movie.
• Can do the opposite:

• Even people who don't like the genre still liked the movie, generally.

• Can also use PCA analysis to reduce factors down to 2. Which lets us plot them:

• Can see that there's different clusters of movies.

00:16:20 - Using collab_learner

• collab_learner wraps up all the functionality above in a single function.

• Can see the names of layers by printing model:

00:17:37 - Embedding distance

• Can calculate the distance between 2 coordinates using cosign similarity.
• Then, find the movies that are most similar to another movie:

00:19:53 - Collab filtering with NN

• Could put item and movie factors into a tensor, then feed into neural network.
• fastai has a function get_emb_sz that gives you recommended sizes for your data:

• Though model isn't as good, it does allow us flexibility to have different embedding sizes for movies or user.
• Same can be achieved with collab_learner function using use_nn=True argument:

• The EmbeddingNN model used in collab_learner is only 1 line of code, as it inherits from TabularModel

00:24:05 - Embeddings for any categorical variable

• Embeddings can handle any type of discrete categorical data: sex, postcode
  • Number of "discrete levels" is called cardinality.

00:25:01 - Rossman sales competition

• Kaggle competition to predict how many things on each day would be sold.
• One of gold medalisted used Deep Learning: one of the first examples of it in the wild.
  • Wrote a paper: Entity Embeddings of Categorical Variables
• In the paper, they explored the embedding matrix for regions in Germany and found that mapping the embeddings to 2d space, turn out to resemble a map of Germany
• They also compared the distance between stores and the embedding distance between stores, and found a strong correlation.
• Similar for days of week and months of year.

00:28:55 - Embeddings at Google Play

• Google wrote a paper called Wide & Deep Learning for Recommender Systems which utilised embeddings, as well as continuous features like Age, App installs, engagement sessions.
  • They also performed cross product transformation on the user installed apps and app the user has has impressions for.

00:30:06 - Other approaches to tabular data

• Deep Learning isn't always the best approach for tabular data.
• An alternative is ensembles of decisions trees (Random Forest and Gradient Boosting Machines)
  • Usually you'll want to "try both"

00:31:29 - Question: why does y_range go from 0 to 5.5 (instead of 5)?

• Because Sigmoid function will never reach exactly 5 as it asymptotes. To predict 5, you'll need a number bigger than it.

00:32:06 - Question: are embeddings used only for high cardinal variables?

• They can be used for any cardinal variable with 2 or more levels.

00:33:03 - Decision Tree ensembles

• The technology has some useful tools for interpreting data and tend to require less hyperparam tuning.
• Jeremy's first approach is to start with a random forest, unless it's high cardinality data.

00:34:50 - Scikit-learn

• Popular library for creating non-deep learning ML models.
  • Refered to as sklearn
• Wes McKinney's book has more about Scikit Learn.

00:35:30 - Blue Book for Bulldozer's Kaggle competition

• Goal to predict the sale price of heavy equipment.
• Kaggle competitions provide interesting datasets, good feedback, leaderbords and blog posts that share teaching.

00:37:02 - Exploring the dataset

• Can download Kaggle data using the API, however, in this example I have it attached to the kernel.
• The main table is train.csv
• Contains:
  • SalesID - unique id of sale
  • MachineID - unique id of machine
  • saleprice - what machine sold at auction
  • saledate - date of the sale.

• First thing to look at is ordinal columns. Things you know are discrete values that have an order.
• Can tell Pandas about suitable ordering with categorical values with ordered=True

• Need to set dependant variable, which is the log of the sale price.

00:40:14 - Understanding Decision Trees

• Decision Trees asks a series of binary (yes/no) questions about data.
• Since we don't know what binary questions to ask, we need to find a way to automate that.
• Simple procedure for doing it:
  1. Loop through each column of dataset
  2. For each column, loop through each possible level (ie item in category)
  3. Split data into 2 groups, based on if less/greater than value or equal or not equal if categorical.
  4. Find the average sale price for each group and see how close to actual label.
  5. After looping through all columns and levels, find split point that gives best predictions.
  6. For each dataset in the split point, perform steps 1-5.
  7. Continue until you reach a stopping criteria - like when a group only has 20 items in it.

00:44:57 - Handling dates

• To take advantage of dates, fastai's add_datepart function expands date columns into more columns.
  • An example of [[Feature Engineering]]

00:45:58 - Cleaning strings and missing data

• Two TabularProcs are used: Categorify and FillMissing
  • Like a Transform but with 2 differences:
    • Modifies object in-place to save memory
    • Runs transform once when data is first passed in
• Categorify replaces column with numerical categorical column.
• FillMissing - add is_missing=True as separate column and settings missing values to the mean.

00:47:31 - Validation set for time series

• Since it's a timeseries, a random validation set isn't going to be approriate.
  • You'll want to use the model in the future: not a date in the past.
• Test set is 2 weeks of data after the data in training set.
  • We should therefore do the same for valid (make last couple of weeks of data).

00:48:46 - Data processing continued

• Separate continuous and categorical variables: cont_cat_split does this for you

• TabularPandas behaves like fastai Datasets object:

• .show shows you the text representation. Under the hood, the data is represented by integers:

• Can save using the save_pickle function:

• And load with load_pickle

00:50:55 - Create a decision tree

• Use a DecisionTreeRegressor when the depedant variable is continuous:

• We can see what is learned using draw_tree function

• To train it further, you can use a higher number of leaf nodes.

00:56:32 - dtreeviz

• A library by Terrance Parr for visualising Decision Trees.
• Looking at the YearMade value, we can see that there are a number values for YearMade as 1000.

00:57:57 - Building a bigger decision tree

• We can remove max_leaf_nodes and train a bigger model:

• Now to evaluate with mse.

• Model appears to be perfect. However, we should check the val set to see if we're overfitting:

• The reason is because we have a number of lead nodes close to the training set.

• We can add a new stop criteria to stop splitting once leaf node has 25 samples

• How can we get the best of booth worlds? Build really deep trees that don't overfit? Bagging...

01:00:42 - Categorical variables revisited

• You usually don't need to one-hot encoded categorical variables with decision trees normally, as it has capability to split on a single "level".

01:30:39 - Bagging

• To improve Decision Tree, you'll use a small technical report called Bagging Predictors by Leo Breiman.
• He described "Bootstrap replicates", which is a subset of the training dataset.
• The steps are as follows:
  • Randomly choose a subset of the rows of your data ("bootstrap replicates of your learning set")
  • Train model on subset
  • Save model and return to step 1 a few times.
  • Predict using all models, then take the average of each models predictions.

01:06:22 - Random Forest

• Breiman developed a variation of this approach, where he would also select a subset of columns randomly when making binary splits. Called [[Random Forest]]

• n_jobs=-1 - use all CPU cores you have.
• n_estimators - how many trees / models to train
• max_samples - how many rows to use for each tree.
• max_features - how many randomly chosen columns to use for each split point.
• min_samples_leaf - stopping criteria

• It's often quite hard to improve on this model - with Random Forests, the defaults turns out to be quite good.

01:08:58 - Analysing Random Forests

• Docs have lots of information about how n_estimators improves error rates.
  • Generally more trees = more accurate models.
• Can access underlying Decision Trees via estimators_

• Can visualise how much adding more models to the ensemble helps with our rmse:

01:11:20 - Out-of-bag error

• Is the validation set worse than the training set because we're over-fitting?
  • Can out out-of-bag error to find out.
• How it works:
  • Each tree is trained on a subset of the data. For each tree, calculate validation predictions on the rows that weren't included.
  • Like validation with a validation set

01:14:09 - Model interpretation

• 5 things we usually want to know about a model:
  • How confident are we with a particular row of data?
  • For predicting a particular item, which factors were most important?
  • Which columns are most important? Which can we ignore?
  • Which columns are effectively redundant with each other?
  • How do predictions vary as we vary these columns?

01:15:14 - Tree variance for prediction confidence

• We've already seen that we can take the mean of all predictions for all trees.
• We can also take the std. This will give us an indication of which rows some of the trees didn't agree on.

• We may decide that predictions that have wild disagreement amongst the model we should be more cautious of.

01:16:47 - Feature importance

• The features that are most strongly driving the independant variable.

• We can also plot them:

• How it works:
  • Loop through each tree and explore each branch.
  • What feature / column was used to split? How much better after split?
  • Add up the amount of improvement for each column at each split and normalise so all add to one.
  • That gives you the feature importance weight.

01:18:53 - Remove low-importance features

• We may decide to remove the less important features. The ones less than 0.005 importance.

• Can then retrain model using just this subset of columns

• Our accuracy is pretty close.
• With only 21 columns, it's a lot easier for us to understand what the model is doing and to "fit it in our heads"

• We can see from this plot that there are some features that have similar importance.

01:20:42 - Remove redundant features

• Can use fastai's cluster_columns

• Will find pairs of columns that are similar.
• Noting that some columns are way out to the right, like saleYear and saleElapsed
  • When one is low, the other tends to be.

• We can write a function that trains on a subset of rows, for performance reasons.

• Now remove each variable one at a time:

• There appears to be 5 variables that we don't need.

• It's pretty similar to model with all the fields.

01:24:13 - Partial dependence

• The next thing to learn about, for the columns that are most important, what is relationship between that column and

• Can do the same thing for YearMade but have to make a histogram:

• We can look at a partial dependance plot.
  • As YearMade increases, the predicted sale price also increases:

• Can see there's a clear relationship between YearMade and sale price.
• Want to answer the question: all other things being equal, how does YearMade impact results?
• How it works:
  • In our dataset, replace all values with 1950 and take the average.
  • Then do that for all YearMade values.
  • Then plot the results.
• For the productsize example, the lowest costs are apparently the ones with no size label.
  • We'd want to learn why that is.

01:29:29 - Tree interpreter

• Last question to answer: for predictions from a single row of data, what are the most important factors that influence the decision?
  • Can use treeinterpreter

• Works similar to computing feature importance:
  • Put row of data through first decision tree and see what split is used at each point.
  • For each split, can see what increase or decrease in prediction.
  • Do for every tree and add total chainge in importance.

• prediction is the prediction the RF makes.
• bias is the prediction based on taking the mean of dependant variable.
• contributions tells total change in prediction due to indepedant variables.
  • sum of contributions + bias should be equal to prediction.

• Can then plot it like this:

• Start at 0, which is the average log sale price of whole dataset (they call the bias).
  • YearMade makes the prediction lower.
  • ProductSize has positive influence.
  • Couple_System has positive influence.
  • ModelID has positive.
  • fiModelDesc positive.
  • etc
• Useful in production, as it can help you understand why predictions are being made.

01:34:05 - Extrapolation

• We only have one dependant variable, PyTroch expects a matrix. So we can use unsqueeze to add a dimension. Change from [40] to [40, 1]

• Can also achieve this with the special value None:

• Can create a Random Forest for the first 30 data points:

• test the model on the entire dataset:

• The Random Forest is unable to extrapolate outside of bounds of data from training set.
• Therefore, we need to ensure that our validation set doesn't contain [[Out-of-Domain]] data.

01:37:41 - Finding out of domain data

• Technique known as [[Adversarial Validation]].
• Can use a random forest to prediction whether data is in validation or test set.

• We can use feature importance to see what's driving the difference:

• saleElapsed is clear, as that's how we split the train and valid set.
• SalesID and MachineID are likely not random and increase over time.
• What happens if we remove those columns from the model?

• Looks like we can remove SalesID and MachineID without losing accuracy.

• What's more, the model will be more resiliant over time.
• We can also see that the sales for the model are mostly in the last year:

• Results of training on a subset:

• The issue of extrapolation doesn't happen with a neural network, as it's using linear layers.

01:42:26 - Question: how does feature importance relate to correlation?

• It doesn't particularly relate.
• Correlation is used more with linear models.

01:43:29 - Using an NN

• Can leverage the work to trim unwanted columsn in Random Forest by using same set of columns for neural network:

• Categorical columns are handled differently in neural networks.
• To create embeddings, need to know which columns should be categorical.
• We'll say that if a column has < 9k variables, we'll say it's categorical.

• One variable that we don't want to treat as categorical is saleElapsed, since we're planning to extrapolate outside of bounds of dataset.

• Can see that ModelID and ModelDesc seem to have similar cardinality. See we can likely remove one.

• With a neural network, we're required to normalize the data before inputting it. This isn't required with a random forest.

• Good idea to set y_range for regression models, find the min and max of dependant variables

• Can create a Learner for a tabular model.
• We are increasing default layer size to 500 from 250.

• Slightly better than our Random Forest.

01:49:19 - Ensembling

• The Random Forest and neural network both are good and bad at some things.
• Ensemble gives us "best of both worlds".
• Easiest is to just take the average:

• We do even better:

01:51:26 - Boosting

• How boosting works:
  • Train small model that under fits dataset.
  • Calculate predictions using small model.
  • Subtract predictions from targets: called "residuals".
    • Represents error for each point in training set.
  • Go to step one, instead of ordinal targets use residuals as target for training
  • Continue doing this until you reach a stopping criterion, like max trees or validation set gets worse.
  • Then, add results of model together (not average).
• Lot of varients of this: Gradient Boost machines (GBMs) and gradient boosted decision trees (GBDTs)
• In this approach, each new tree attempting to fit the error of all previous trees combined.

01:53:23 - Question: is dropping features better than regularisation for overfitting?

• Removing columns isn't to avoid overfitting - it's to simplify the model.
  • It will mean less trees, but it doesn't mean regularisation.

01:54:14 - Question: is there a heuristic for picking number of layers in linear model?

• Not really. 2 is a good start.

01:55:00 - Boosting continued

• There's nothing to stop the model from overfitting, unlike a Random Forest where more trees means better performance.
  • Use regularisation.
• More often win Kaggle competitions than Random Forests do.

01:56:02 - Combining embeddings with other methods

• From Entity Embeddings paper.
• They build NN and got entity embeddings.
• Used a random forest using embeddings as predictors.

01:57:44 - Conclusion