fastai - Lesson 8 - Deep Learning for Coders (2020)

Notes taken while watching the Deep Learning for Coders (2020) - Lesson 8 video

Lex Toumbourou · Aug 28, 2021

Notes taken while watching the Lesson 8 - Deep Learning for Coders (2020) video.

In [1]:

!pip install git+https://github.com/fastai/fastai.git &> /dev/null

00:00:00 - Intro and NLP Review

In Lesson 1, we create a sentiment classifier, but haven't looked at what's going on behind the scene.

00:01:31 - Language models in NLP

The pretrained model we used for the IMDB sentiment analysis, was a pretrained language model.
One example of a language model: trying to predict next word of text.
- To be able to this, a language needs to have a good understanding of language.
You can download pretrained language models through various Model Zoos.

00:04:36 - Review of lesson 1.

We downloaded a pretrained model and fine tuned it:

In [2]:

from fastai.text.all import *

dls = TextDataLoaders.from_folder(untar_data(URLs.IMDB), valid='test')
learn = text_classifier_learner(dls, AWD_LSTM, drop_mult=0.5, metrics=accuracy)
learn.fine_tune(4, 1e-2)

100.00% [144441344/144440600 00:03<00:00]

100.00% [105070592/105067061 00:02<00:00]

epoch	train_loss	valid_loss	accuracy	time
0	0.449429	0.379644	0.831000	02:17

epoch	train_loss	valid_loss	accuracy	time
0	0.297342	0.359466	0.860680	04:14
1	0.251144	0.222691	0.910480	04:16
2	0.198196	0.195068	0.922920	04:14
3	0.146078	0.191531	0.928360	04:15

00:05:08 - Improving results by creating a domain-specific language model

One trick to use is start with a Wikitext language model, then improve it by creating a domain-specific language model. In this example, an IMDb language model.
- Will learn IMDb specific words.

00:05:58 - Building a language model from scratch

Sentences can be different lengths. Documents can be very long.
We already looked at using categorical variables as independant variables:
- Make list of all possible levels of the categorical variable (vocab)
- Replace each level with index in vocab
- Create embedding matrix containing a row for each level
- Use embedding matric as first layer of nn
We can almost do the same thing with text.
- Create a list of all words to generate vocab
- Only question is how to use a sequence?
Independant variable is sequence of words from first to 2nd last.
Dependant variable is 2nd word down to last.
When we create vocab, a lot of words will be already in embedding matrix
- May be new ones: informal slang, words not in Wikipedia etc.
For words in Wikipedia, we will use the pretrained embedding vector.
For new words, we will create a new random row.
List of steps:
- Tokenisation - convert into a list of words (characters or substrings)
- Numericalisation - make a list of unique words in the vocab. Convert each word into a number by looking up the index in vocab
- Language model data loader - fastai has LMDataLoader which handles creating dependent varible offset from independent by one token.
- Language model - need a special model that can handle sequences of arbitrary length.

00:10:27 - Tokenisation

Converting a text into a list of words.
Questions to ask: What to do with punctuation? How to deal with words like "don't"? Long medical or chemical words? etc
3 common approaches:
- Word-based approached: used by default in English. Split on spaces and applies some language specific rules to separate parts of meaning. Like turning "don't" into "do n't". Punctuation marks also split into separate tokens.
- Subword based: split words into smaller parts, based on common occurring substrings. "occasion" might be tokenised as "o c ca sion"
- Character-based: split sentence into individual characters.

00:12:19 - Word tokenisation

fastai doesn't invent its own tokenisers, but provide consistent interface to a range of tokenisers.
Start by getting all text files from IMDB dataset:

In [3]:

from fastai.text.all import *
path = untar_data(URLs.IMDB)

In [4]:

files = get_text_files(path, folders=['train', 'test', 'unsup'])

Can get and read first one:

In [5]:

txt = files[0].open().read(); txt[:75]

Out[5]:

'Jiang Xian uses the complex backstory of Ling Ling and Mao Daobing to study'

Default English word tokeniser called spaCy.

In [6]:

spacy = WordTokenizer()
toks = first(spacy([txt]))
print(coll_repr(toks, 30))

(#143) ['Jiang','Xian','uses','the','complex','backstory','of','Ling','Ling','and','Mao','Daobing','to','study','Mao',"'s",'"','cultural','revolution','"','(','1966','-','1976',')','at','the','village','level','.'...]

Another example:

In [7]:

first(spacy(['The U.S. dollar $1 is $1.00']))

Out[7]:

(#8) ['The','U.S.','dollar','$','1','is','$','1.00']

Fastai adds some additional functionality. It adds some "special tokens" that start with character xx
- xxbos indicates start of new text or "beginning of stream".
- xxmaj - next word beings with capital.
- xxunk indicates next word is unknown

In [8]:

tkn = Tokenizer(spacy)

In [9]:

coll_repr(tkn('&copy;    Fast.ai www.fast.ai/INDEX'), 31)

Out[9]:

"(#11) ['xxbos','©','xxmaj','fast.ai','xxrep','3','w','.fast.ai','/','xxup','index']"

00:17:38 - Subword tokeniser

In Chinese, they don't use spaces. Instead, they use subword tokenisation.
- Look at corpus of documents and find most commonly occurring groups of letters: they become vocab.

In [10]:

txts = L(o.open().read() for o in files[:2000])

This function setup is used to prepare or train some transformations. In this example, it trains on a subset of the corpus

In [11]:

def subword(sz):
    sp = SubwordTokenizer(vocab_sz=sz)
    sp.setup(txts)
    return ' '.join(first(sp([txt]))[:40])

In [12]:

subword(1000)

Out[12]:

'▁J i ang ▁ X ian ▁us es ▁the ▁comp le x ▁back st or y ▁of ▁L ing ▁L ing ▁and ▁Ma o ▁Da o b ing ▁to ▁st u d y ▁Ma o \' s ▁" c ul'

With a larger vocab, most common English words end up in vocab itself

In [13]:

subword(10000)

Out[13]:

'▁J ian g ▁X ian ▁uses ▁the ▁complex ▁back story ▁of ▁L ing ▁L ing ▁and ▁Ma o ▁Da ob ing ▁to ▁study ▁Ma o \' s ▁" cul t ural ▁revolution " ▁( 1966 - 1 9 7 6'

Jeremy predicts that subword will be the most popular form of tokenisation.

00:21:21 - Question: how can we determine if pretrained model is suitable for downstream task? If there's limited vocab overlap, should we create a language model from scratch

In same language, Wikitext usually works well.
If you were using Genomic sequences, or Greek then you would likely need a domain-specific language model.

00:23:25 - Numericalization

After splitting into tokens, the next step is numericalization.

In [14]:

toks200 = txts[:200].map(tkn)
toks200[0]

Out[14]:

(#158) ['xxbos','xxmaj','jiang','xxmaj','xian','uses','the','complex','backstory','of'...]

Can then pass into setup to create vocab

In [15]:

num = Numericalize()
num.setup(toks200)
coll_repr(num.vocab, 20)

Out[15]:

"(#2152) ['xxunk','xxpad','xxbos','xxeos','xxfld','xxrep','xxwrep','xxup','xxmaj','the',',','.','and','a','of','to','is','in','it','i'...]"

The tokens are in order of frequency.
- Defaults:
  - min_freq=3 - minimum frequency required.
  - max_vocab=60000 - limit the size of the embedding matrix

Can now call num as if it was a function:

In [16]:

nums = num(toks)[:20]; nums

Out[16]:

TensorText([   0,    0, 1268,    9, 1269,    0,   14,    0,    0,   12,    0,    0,
          15, 1270,    0,   22,   24,    0,  795,   24])

Can convert back by indexing into vocab:

In [17]:

' '.join(num.vocab[o] for o in nums)

Out[17]:

'xxunk xxunk uses the complex xxunk of xxunk xxunk and xxunk xxunk to study xxunk \'s " xxunk revolution "'

00:25:43 - Putting texts into batches for language model

Items in a mini-batch need to be the same size to be stacked in Tensor.
- With images, we resize them all to a consistent size.
How do we do it with text?
We could break into contiguous blocks that match our batch size. Here's an example of a batch size of 6:

In [18]:

stream = "In this chapter, we will go back over the example of classifying movie reviews we studied in chapter 1 and dig deeper under the surface. First we will look at the processing steps necessary to convert text into numbers and how to customize it. By doing this, we'll have another example of the PreProcessor used in the data block API.\nThen we will study how we build a language model and train it for a while."
tokens = tkn(stream)
bs,seq_len = 6,15
d_tokens = np.array([tokens[i*seq_len:(i+1)*seq_len] for i in range(bs)])
df = pd.DataFrame(d_tokens)
df

Out[18]:

	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14
0	xxbos	xxmaj	in	this	chapter	,	we	will	go	back	over	the	example	of	classifying
1	movie	reviews	we	studied	in	chapter	1	and	dig	deeper	under	the	surface	.	xxmaj
2	first	we	will	look	at	the	processing	steps	necessary	to	convert	text	into	numbers	and
3	how	to	customize	it	.	xxmaj	by	doing	this	,	we	'll	have	another	example
4	of	the	preprocessor	used	in	the	data	block	xxup	api	.	\n	xxmaj	then	we
5	will	study	how	we	build	a	language	model	and	train	it	for	a	while	.

That won't scale to a giant corpus, so we have to divide into subarrays with a fixed sequence length.
Let's try a sequence length of 5:

In [19]:

bs,seq_len = 6,5
d_tokens = np.array([tokens[i*15:i*15+seq_len] for i in range(bs)])
df = pd.DataFrame(d_tokens)
df

Out[19]:

	0	1	2	3	4
0	xxbos	xxmaj	in	this	chapter
1	movie	reviews	we	studied	in
2	first	we	will	look	at
3	how	to	customize	it	.
4	of	the	preprocessor	used	in
5	will	study	how	we	build

Then the next batch:

In [20]:

#hide_input
bs,seq_len = 6,5
d_tokens = np.array([tokens[i*15+seq_len:i*15+2*seq_len] for i in range(bs)])
df = pd.DataFrame(d_tokens)
df

Out[20]:

	0	1	2	3	4
0	,	we	will	go	back
1	chapter	1	and	dig	deeper
2	the	processing	steps	necessary	to
3	xxmaj	by	doing	this	,
4	the	data	block	xxup	api
5	a	language	model	and	train

And so on.

00:29:24 - LMDataLoader

Don't have to do these steps yourself, LMDataLoader in fastai handles it for you.

In [21]:

nums200 = toks200.map(num)

In [22]:

dl = LMDataLoader(nums200)

In [23]:

x, y = first(dl)
x.shape, y.shape

Out[23]:

(torch.Size([64, 72]), torch.Size([64, 72]))

64 is batch size and 72 is sequence size.
Can examine the first 20 tokens of independent variable:

In [24]:

' '.join(num.vocab[o] for o in x[0][:20])

Out[24]:

'xxbos xxmaj xxunk xxmaj xxunk uses the complex xxunk of xxmaj xxunk xxmaj xxunk and xxmaj xxunk xxmaj xxunk to'

then dependent variable, which is the same thing offset by one token:

In [25]:

' '.join(num.vocab[o] for o in y[0][:20])

Out[25]:

'xxmaj xxunk xxmaj xxunk uses the complex xxunk of xxmaj xxunk xxmaj xxunk and xxmaj xxunk xxmaj xxunk to study'

00:31:07 - Creating language model with DataBlock

In the DataBlock instance, we're using a TextBlock from the block argument.
- Passing in a class method, allows the tokenisation to be cached.

In [26]:

get_imdb = partial(get_text_files, folders=['train', 'test', 'unsup'])

dls_lm = DataBlock(
    blocks=TextBlock.from_folder(path, is_lm=True),
    get_items=get_imdb, splitter=RandomSplitter(0.1)
).dataloaders(path, path=path, bs=128, seq_len=80)

Note the items are text items from folders.
We can then call show_batch:

In [27]:

dls_lm.show_batch(max_n=2)

	text	text_
0	xxbos xxmaj this movie is so xxup stupid ! i mean .. why does xxmaj liam xxmaj neeson take this woman so seriously , having only heard her say 3 xxup words ! xxmaj when they found her in the woods , they should xxunk have committed her to psychiatric hospital to try and make her a real human being . xxmaj just to see xxmaj jodie " hey - see how stupid i look " xxmaj foster dance around	xxmaj this movie is so xxup stupid ! i mean .. why does xxmaj liam xxmaj neeson take this woman so seriously , having only heard her say 3 xxup words ! xxmaj when they found her in the woods , they should xxunk have committed her to psychiatric hospital to try and make her a real human being . xxmaj just to see xxmaj jodie " hey - see how stupid i look " xxmaj foster dance around chanting
1	tracking down vehicles that may have entered the crime scene from camera - visible locations adjacent to the crime scene as part of developing clues . \n\n * xxmaj in xxmaj england , driving is on the left . xxmaj the director goes out of his way to have the car at the crime scene park on the right , several meters away from the flower kiosk , when it could have easily parked immediately behind , or even on	down vehicles that may have entered the crime scene from camera - visible locations adjacent to the crime scene as part of developing clues . \n\n * xxmaj in xxmaj england , driving is on the left . xxmaj the director goes out of his way to have the car at the crime scene park on the right , several meters away from the flower kiosk , when it could have easily parked immediately behind , or even on the

00:33:23 - Fine tuning a language model

Fine tuning language model creates a learning which learns to predict next word of music review.
AWD_LSTM is the architecture.
drop_mult is amount of Dropout (covered later in the course).

In [28]:

learn = language_model_learner(
    dls_lm, AWD_LSTM, drop_mult=0.3,
    metrics=[accuracy, Perplexity()],
    path='/kaggle/working'
).to_fp16()

In [29]:

learn.fit_one_cycle(1, 2e-2)

epoch	train_loss	valid_loss	accuracy	perplexity	time
0	4.006212	3.899826	0.300805	49.393833	27:45

00:35:07 - Saving and loading models

Can use learn.save to save intermediary results:

In [30]:

learn.save('1epoch')

Out[30]:

Path('/kaggle/working/models/1epoch.pth')

And load with learn.load:

In [31]:

learn.load('1epoch')

Out[31]:

<fastai.text.learner.LMLearner at 0x7fa49e0a2e10>

Can then finetune after unfreezing:

In [32]:

learn.unfreeze()
learn.fit_one_cycle(10, 2e-3)

epoch	train_loss	valid_loss	accuracy	perplexity	time
0	3.763982	3.759459	0.316605	42.925179	30:04
1	3.684646	3.699707	0.323066	40.435436	30:15
2	3.623890	3.654301	0.328286	38.640499	30:13
3	3.564344	3.620885	0.332433	37.370613	30:22
4	3.501577	3.598828	0.335188	36.555359	30:30
5	3.427867	3.579867	0.337651	35.868767	30:05
6	3.362838	3.575086	0.339170	35.697678	30:37
7	3.302284	3.569533	0.340268	35.499996	30:19
8	3.239792	3.574208	0.340490	35.666363	30:12
9	3.215606	3.578157	0.340285	35.807476	30:14

We can then save just the encoder.
Encoder is all of the model that isn't the final layer.

In [33]:

learn.save_encoder('finetuned')

00:36:44 - Question: Do language models attempt to provide meaning?

Language model do tend to get good at understanding nuances of languages.

00:37:56 - Text generation (next notebook)

00:37:56 - Text generation

Firstly, to load the model from the previous notebook:

In [1]:

from fastai.text.all import *

In [2]:

path = untar_data(URLs.IMDB)

get_imdb = partial(get_text_files, folders=['train', 'test', 'unsup'])

dls_lm = DataBlock(
    blocks=TextBlock.from_folder(path, is_lm=True),
    get_items=get_imdb, splitter=RandomSplitter(0.1)
).dataloaders(path, path=path, bs=128, seq_len=80)

learn = language_model_learner(
    dls_lm, AWD_LSTM, drop_mult=0.3,
    metrics=[accuracy, Perplexity()],
    path='/kaggle/working'
).to_fp16()

In [3]:

learn.load_encoder('/kaggle/input/lesson-8/models/finetuned')

Out[3]:

<fastai.text.learner.LMLearner at 0x7f374aed6bd0>

Then make some predictions using a short sentence as a prompt:

In [4]:

TEXT = "I liked this movie because"
N_WORDS = 40
N_SENTENCES = 2
preds = [learn.predict(TEXT, N_WORDS, temperature=0.75) 
         for _ in range(N_SENTENCES)]
print("\n".join(preds))

i liked this movie because there were n't any twists or turns and was never dull . But the acting was pretty good considering everything else . The characters were believable and believable . Tom Sizemore and Emily Dafoe
i liked this movie because of its relatively low budget . The special effects are n't bad either -- except for an exaggerated facial expression . 

 But now for the sudden appearance of these aliens . In fact , these aliens

In [5]:

TEXT = "I thought this movie would be more like Jaws but it was "
N_WORDS = 40
N_SENTENCES = 2

preds = [
    learn.predict(TEXT, N_WORDS, temperature=0.75)
    for _ in range(N_SENTENCES)
]

print("\n".join(preds))

i thought this movie would be more like Jaws but it was n't a rip off . It was mostly about middle spectacular men searching for boat and finding each other with their own basic twists and turns . This was written and directed by Al Pacino who
i thought this movie would be more like Jaws but it was more like a remake of Jaws than a remake or update . 

 Although they did n't compare this to Jaws 3 because it still has n't changed the storyline since it was made .

There are better ways to do language generation, but this tells us that the model has learned something.

00:39:51 - Creating classification model

Create another DataBlock. This time, we provide a vocab from the language by passing in a vocab (dls_lm.vocab) to TextBlock.from_folder
Also aren't passing is_lm=True instead passing CategoryBlock as the dataset has a sentiment label.

In [6]:

dls_clas = DataBlock(
    blocks=(TextBlock.from_folder(path, vocab=dls_lm.vocab), CategoryBlock),
    get_y=parent_label,
    get_items=partial(get_text_files, folders=['train', 'test']),
    splitter=GrandparentSplitter(valid_name='test')
).dataloaders(path, path=path, bs=128, seq_len=72)

In [7]:

dls_clas.show_batch(max_n=3)

	text	category
0	xxbos xxmaj match 1 : xxmaj tag xxmaj team xxmaj table xxmaj match xxmaj bubba xxmaj ray and xxmaj spike xxmaj dudley vs xxmaj eddie xxmaj guerrero and xxmaj chris xxmaj benoit xxmaj bubba xxmaj ray and xxmaj spike xxmaj dudley started things off with a xxmaj tag xxmaj team xxmaj table xxmaj match against xxmaj eddie xxmaj guerrero and xxmaj chris xxmaj benoit . xxmaj according to the rules of the match , both opponents have to go through tables in order to get the win . xxmaj benoit and xxmaj guerrero heated up early on by taking turns hammering first xxmaj spike and then xxmaj bubba xxmaj ray . a xxmaj german xxunk by xxmaj benoit to xxmaj bubba took the wind out of the xxmaj dudley brother . xxmaj spike tried to help his brother , but the referee restrained him while xxmaj benoit and xxmaj guerrero	pos
1	xxbos xxmaj titanic directed by xxmaj james xxmaj cameron presents a fictional love story on the historical setting of the xxmaj titanic . xxmaj the plot is simple , xxunk , or not for those who love plots that twist and turn and keep you in suspense . xxmaj the end of the movie can be figured out within minutes of the start of the film , but the love story is an interesting one , however . xxmaj kate xxmaj winslett is wonderful as xxmaj rose , an aristocratic young lady betrothed by xxmaj cal ( billy xxmaj zane ) . xxmaj early on the voyage xxmaj rose meets xxmaj jack ( leonardo dicaprio ) , a lower class artist on his way to xxmaj america after winning his ticket aboard xxmaj titanic in a poker game . xxmaj if he wants something , he goes and gets it	pos
2	xxbos xxmaj warning : xxmaj does contain spoilers . \n\n xxmaj open xxmaj your xxmaj eyes \n\n xxmaj if you have not seen this film and plan on doing so , just stop reading here and take my word for it . xxmaj you have to see this film . i have seen it four times so far and i still have n't made up my mind as to what exactly happened in the film . xxmaj that is all i am going to say because if you have not seen this film , then stop reading right now . \n\n xxmaj if you are still reading then i am going to pose some questions to you and maybe if anyone has any answers you can email me and let me know what you think . \n\n i remember my xxmaj grade 11 xxmaj english teacher quite well . xxmaj	pos

00:41:04 - Question: Do tokeniser do stemming or lemitisation, or is that outdated?

Stemming and lemitisation is not part of tokenisation.
We have stems for a reason, so we don't remove them.

00:42:21 - Handling different sequence lengths

With the language model, we can concat all documents together and split into substrings based on batch size. This ensures each mini-batch is the same size (batch_size x sequence length).
- We can't do that with shorter movie reviews. Each movie needs to be associated with dependant variable.

In [8]:

spacy = WordTokenizer()
tkn = Tokenizer(spacy)
files = get_text_files(path, folders=['train', 'test', 'unsup'])
txts = L(o.open().read() for o in files[:2000])

toks200 = txts[:200].map(tkn)
toks200[0]

num = Numericalize()
num.setup(toks200)

In [9]:

nums_samp = toks200[:10].map(num)

Can look at the lengths, and notice that they're all quite different:

In [10]:

nums_samp.map(len)

Out[10]:

(#10) [158,319,181,193,114,145,260,146,252,295]

What we need to do is add padding.
- Add special xxpad token to each sequence to make them all the size of the largest sequence.
fastai also tries to get similarly sized sentences together to try to minimise padding.
All of this happens when you call TextBlock.from_folder

00:45:30 - Create and fine tune classifier

In [11]:

learn = text_classifier_learner(
    dls_clas, AWD_LSTM, drop_mult=0.5, metrics=accuracy).to_fp16()

Now we can load the encoder:

In [12]:

learn = learn.load_encoder('/kaggle/input/lesson-8/models/finetuned')

In [13]:

learn.fit_one_cycle(1, 2e-2)

epoch	train_loss	valid_loss	accuracy	time
0	0.269207	0.195298	0.924400	01:46

The results are similar to the first classifier, but took under 2 minutes to train.
In NLP, it's better to unfreeze a layer at time:

In [14]:

learn.freeze_to(-2)
learn.fit_one_cycle(1, slice(1e-2/(2.6**4), 1e-2))

epoch	train_loss	valid_loss	accuracy	time
0	0.253995	0.181843	0.931440	01:58

Can unfreeze a few more layers:

In [15]:

learn.freeze_to(-3)
learn.fit_one_cycle(1, slice(5e-3/(2.6**4), 5e-3))

epoch	train_loss	valid_loss	accuracy	time
0	0.211313	0.166533	0.936720	02:42

Then the whole model:

In [16]:

learn.unfreeze()
learn.fit_one_cycle(2, slice(1e-3/(2.6**4), 1e-3))

epoch	train_loss	valid_loss	accuracy	time
0	0.173864	0.164418	0.938880	03:18
1	0.170001	0.163286	0.939640	03:17

Can get to 95.1% accuracy by training on all texts backwards.

00:48:54 - Question: how can a model trained to predict next word work on a different domain like sentiment?

To be able to predict next word of sentence, you have to know a lot about language and the world. That knowledge transfers to other domains.

00:51:00 - Question: how do you do data augmentation on text?

One approach is to pass text through a translator and back again.
Some goods ideas in this paper: Unsupervised Data Augmentation for Consistency Training.

00:51:52 - Ethics and risks of text generation

FCC asked for comments about a proposal to repeal Net Neutrality. Turned out less than 800k of 22M comments where unique.
What would happen if someone created a million Twitter bots so that 99% of the content were fake deep learning bots?

00:56:22 - Language model from scratch

Going to learn how a recurrent neural network works.

00:56:52 - Question: are there model interpretabilty tools for language models?

There are some, but won't be covered in this part of the course.

00:58:11 - Preparing the dataset: tokenisation and numericalisation

Jeremy created a dataset called human numbers that contains first 10k numbers written out in English.
- Seems very few people create datasets, even though it's not particularly hard.

In [2]:

from fastai.text.all import *
from fastcore.foundation import L

path = untar_data(URLs.HUMAN_NUMBERS)

In [3]:

Path.BASE_PATH = path

In [4]:

path.ls()

Out[4]:

(#2) [Path('valid.txt'),Path('train.txt')]

In [5]:

lines = L()
with open(path/'train.txt') as f:
    lines += L(*f.readlines())
with open(path/'valid.txt') as f:
    lines += L(*f.readlines())
    
lines

Out[5]:

(#9998) ['one \n','two \n','three \n','four \n','five \n','six \n','seven \n','eight \n','nine \n','ten \n'...]

Concat them together and put a dot between them for tokenising.

In [6]:

text = ' . '.join([l.strip() for l in lines])
text[:100]

Out[6]:

'one . two . three . four . five . six . seven . eight . nine . ten . eleven . twelve . thirteen . fo'

Tokenise by splitting on spaces:

In [7]:

tokens = L(text.split(' '))
tokens[100:110]

Out[7]:

(#10) ['.','forty','two','.','forty','three','.','forty','four','.']

Create a vocab by getting unique tokens:

In [8]:

vocab = L(tokens).unique()
vocab, len(vocab)

Out[8]:

((#30) ['one','.','two','three','four','five','six','seven','eight','nine'...],
 30)

Convert tokens into numbers by looking up index of each word:

In [9]:

word2idx = {w: i for i,w in enumerate(vocab)}
nums = L(word2idx[i] for i in tokens)
tokens, nums

Out[9]:

((#63095) ['one','.','two','.','three','.','four','.','five','.'...],
 (#63095) [0,1,2,1,3,1,4,1,5,1...])

That gives us a small easy dataset for building a language model.

01:01:31 - Language model from scratch

Create a independent and dependent pair: first 3 words are input, next is dependent.

In [10]:

L((tokens[i:i+3], tokens[i+3]) for i in range(0, len(tokens)-4, 3))

Out[10]:

(#21031) [(['one', '.', 'two'], '.'),(['.', 'three', '.'], 'four'),(['four', '.', 'five'], '.'),(['.', 'six', '.'], 'seven'),(['seven', '.', 'eight'], '.'),(['.', 'nine', '.'], 'ten'),(['ten', '.', 'eleven'], '.'),(['.', 'twelve', '.'], 'thirteen'),(['thirteen', '.', 'fourteen'], '.'),(['.', 'fifteen', '.'], 'sixteen')...]

Same thing numericalised:

In [11]:

seqs = L((tensor(nums[i:i+3]), nums[i+3]) for i in range(0, len(nums)-4, 3))
seqs

Out[11]:

(#21031) [(tensor([0, 1, 2]), 1),(tensor([1, 3, 1]), 4),(tensor([4, 1, 5]), 1),(tensor([1, 6, 1]), 7),(tensor([7, 1, 8]), 1),(tensor([1, 9, 1]), 10),(tensor([10,  1, 11]), 1),(tensor([ 1, 12,  1]), 13),(tensor([13,  1, 14]), 1),(tensor([ 1, 15,  1]), 16)...]

Can batch those with a DataLoader object
- Take first 80% as training, last 20% as validation.

In [12]:

bs = 64
cut = int(len(seqs) * 0.8)
dls = DataLoaders.from_dsets(seqs[:cut], seqs[:cut], bs=64, shuffle=False)

01:03:35 - Simple Language model

3 layer neural network.
- One linear layer that is reused 3 times.
- Each time the result is added to the embedding.

In [13]:

class LMModel(Module):
    def __init__(self, vocab_sz, n_hidden):
        self.input2hidden = nn.Embedding(vocab_sz, n_hidden)
        self.hidden2hidden = nn.Linear(n_hidden, n_hidden)
        self.hidden2output = nn.Linear(n_hidden, vocab_sz)
    
    def forward(self, x):
        hidden = self.input2hidden(x[:,0])
        hidden = F.relu(self.hidden2hidden(hidden))
    
        hidden = hidden + self.input2hidden(x[:,1])
        hidden = F.relu(self.hidden2hidden(hidden))
    
        hidden = hidden + self.input2hidden(x[:,2])
        hidden = F.relu(self.hidden2hidden(hidden))

        return self.hidden2output(hidden)

01:04:49 - Question: can you speed up fine-tuning the NLP model?

Do something else while you wait or leave overnight.

01:05:44 - Simple Language model cont.

2 interesting happening:
- Some of the inputs are being fed into later layers, instead of just the first.
- The model is reusing hidden state throughout layers.

In [14]:

learn = Learner(dls, LMModel(len(vocab), 64), loss_func=F.cross_entropy, metrics=accuracy)
learn.fit_one_cycle(4, 1e-3)

epoch	train_loss	valid_loss	accuracy	time
0	1.800355	1.929378	0.462197	00:03
1	1.364388	1.685311	0.466120	00:02
2	1.399541	1.464157	0.490787	00:03
3	1.362921	1.412893	0.495007	00:03

We can find out if that accuracy is good by making a simple model that predicts most common token:

In [15]:

c = Counter(tokens[cut:])
mc = c.most_common(5)
mc

Out[15]:

[('thousand', 7104),
 ('.', 7103),
 ('hundred', 6405),
 ('nine', 2440),
 ('eight', 2344)]

In [16]:

mc[0][1]/len(tokens[cut:])

Out[16]:

0.15353028894988222

01:14:41 - Recurrent neural network

We can refactor in Python using a for loop.
- Note that hidden = 0 is being broadcast into the hidden state.

In [17]:

class LMModel2(Module):
    def __init__(self, vocab_sz, n_hidden):
        self.input2hidden = nn.Embedding(vocab_sz, n_hidden)
        self.hidden2hidden = nn.Linear(n_hidden, n_hidden)
        self.hidden2output = nn.Linear(n_hidden, vocab_sz)
    
    def forward(self, x):
        hidden = 0.
        
        for i in range(3):
            hidden = hidden + self.input2hidden(x[:,i])
            hidden = F.relu(self.hidden2hidden(hidden))

        return self.hidden2output(hidden)

In [18]:

learn = Learner(dls, LMModel2(len(vocab), 64), loss_func=F.cross_entropy, metrics=accuracy)
learn.fit_one_cycle(4, 1e-3)

epoch	train_loss	valid_loss	accuracy	time
0	1.827151	1.976822	0.457739	00:02
1	1.383131	1.667376	0.468141	00:03
2	1.398363	1.497497	0.490252	00:03
3	1.382042	1.432224	0.490965	00:03

That's what a Recurrent Neural Network is!

01:18:39 - Improving the RNN

Note that we're setting the previous state to 0.
- However, the hidden state from sequence to sequence contains useful information.
- We can rewrite to maintain state of RNN.

In [19]:

class LMModel3(Module):
    def __init__(self, vocab_sz, n_hidden):
        self.input2hidden = nn.Embedding(vocab_sz, n_hidden)
        self.hidden2hidden = nn.Linear(n_hidden, n_hidden)
        self.hidden2output = nn.Linear(n_hidden, vocab_sz)
        self.hidden = 0.
        
    
    def forward(self, x):
        for i in range(3):
            self.hidden = self.hidden + self.input2hidden(x[:,i])
            self.hidden = F.relu(self.hidden2hidden(self.hidden))
    
        output = self.hidden2output(self.hidden)
        self.hidden = self.hidden.detach()

        return output
    
    def reset(self):
        self.h = 0.

01:19:41 - Back propagation through time

Note that we called self.hidden.detach() each forward pass to ensure we're not back propogating through all the previous forward passes.
- Known as Back propagation through time (BPTT)

01:22:19 - Ordered sequences and callbacks

Samples must be seen in correct order - each batch needs to connect to previous batch.
At the start of each epoch, we need to call reset.

In [20]:

def group_chunks(ds, bs):
    m = len(ds) // bs
    new_ds = L()
    for i in range(m):
        new_ds += L(ds[i + m*j] for j in range(bs))
    return new_ds

cut = int(len(seqs) * 0.8)
dls = DataLoaders.from_dsets(
    group_chunks(seqs[:cut], bs),
    group_chunks(seqs[cut:], bs),
    bs=bs, drop_last=True, shuffle=False
)

At start of epoch, we call reset
- Last thing to add is little tweak of training model via aCallback called ModelReseter

In [21]:

from fastai.callback.rnn import ModelResetter

In [22]:

learn = Learner(dls, LMModel3(len(vocab), 64), loss_func=F.cross_entropy, metrics=accuracy, cbs=ModelResetter)
learn.fit_one_cycle(10, 3e-3)

epoch	train_loss	valid_loss	accuracy	time
0	1.708671	1.819227	0.483173	00:02
1	1.231601	1.618543	0.513221	00:02
2	1.094240	1.709943	0.462019	00:02
3	1.020911	1.706639	0.529808	00:02
4	0.977250	1.659540	0.542788	00:02
5	0.909005	1.681440	0.568750	00:02
6	0.873126	1.604766	0.578125	00:02
7	0.811297	1.745289	0.600000	00:02
8	0.774836	1.684849	0.606250	00:02
9	0.762098	1.629690	0.618510	00:02

Training is doing a lot better now.

01:25:00 - Creating more signal

Instead of putting output stage outside the loop, can put it in the loop.
- After every hidden state, we get a prediction.
Can we change the data so dependant variable has each of the three words after each three input words.

In [23]:

sl = 16
seqs = L((tensor(nums[i:i+sl]), tensor(nums[i+1:i+sl+1]))
        for i in range(0, len(nums) - sl - 1, sl))
cut = int(len(seqs) * 0.8)

dls = DataLoaders.from_dsets(group_chunks(seqs[:cut], bs),
                             group_chunks(seqs[cut:], bs),
                             bs=bs, drop_last=True, shuffle=False)

In [24]:

[L(vocab[o] for o in s) for s in seqs[0]]

Out[24]:

[(#16) ['one','.','two','.','three','.','four','.','five','.'...],
 (#16) ['.','two','.','three','.','four','.','five','.','six'...]]

We can modify the model to return a list of outputs.

In [25]:

class LMModel4(Module):
    def __init__(self, vocab_sz, n_hidden):
        self.input2hidden = nn.Embedding(vocab_sz, n_hidden)
        self.hidden2hidden = nn.Linear(n_hidden, n_hidden)
        self.hidden2output = nn.Linear(n_hidden, vocab_sz)
        self.hidden = 0.

    def forward(self, x):
        outputs = []

        for i in range(sl):
            self.hidden = self.hidden + self.input2hidden(x[:,i])
            self.hidden = F.relu(self.hidden2hidden(self.hidden))
            outputs.append(self.hidden2output(self.hidden))

        self.hidden = self.hidden.detach()

        return torch.stack(outputs, dim=1)
    
    def reset(self):
        self.h = 0.

We have to write a custom loss function to flatten the outputs:

In [26]:

def loss_func(inp, targ):
    return F.cross_entropy(inp.view(-1, len(vocab)), targ.view(-1))

In [27]:

learn = Learner(dls, LMModel4(len(vocab), 64), loss_func=loss_func, metrics=accuracy, cbs=ModelResetter)
learn.fit_one_cycle(15, 3e-3)

epoch	train_loss	valid_loss	accuracy	time
0	3.248546	3.115103	0.164551	00:01
1	2.386219	1.933674	0.466797	00:01
2	1.760738	1.806952	0.472331	00:01
3	1.467624	1.882069	0.503092	00:01
4	1.259531	1.883375	0.535889	00:01
5	1.079385	2.010900	0.555339	00:01
6	0.939819	1.925642	0.572917	00:01
7	0.831508	1.877170	0.591390	00:01
8	0.748246	2.093921	0.574544	00:01
9	0.673372	2.032348	0.602865	00:01
10	0.628112	2.123352	0.600667	00:01
11	0.594529	2.048167	0.618001	00:01
12	0.556409	2.081254	0.619466	00:01
13	0.534996	2.098750	0.619059	00:01
14	0.524235	2.071640	0.617269	00:01

01:28:29 - Multilayer RNN

Even though the RNN seemed to have a lot of layers, each layer is sharing the same weight matrix.
- Not that much better than a simple linear model.
A multilayer RNN can stake multiple linear layers within the for loop.
PyTorch provides the nn.RNN class for creating multilayers RNNs.

In [28]:

class LMModel5(Module):
    def __init__(self, vocab_sz, n_hidden, n_layers):
        self.input2hidden = nn.Embedding(vocab_sz, n_hidden)
        self.rnn = nn.RNN(n_hidden, n_hidden, n_layers, batch_first=True)
        self.hidden2output = nn.Linear(n_hidden, vocab_sz)
        self.hidden = torch.zeros(n_layers, bs, n_hidden)

    def forward(self, x):
        res, h = self.rnn(self.input2hidden(x), self.hidden)
        self.hidden = h.detach()
        return self.hidden2output(res)
    
    def reset(self):
        self.hidden.zero_()

In [29]:

learn = Learner(dls, LMModel5(len(vocab), 64, 2), loss_func=loss_func, metrics=accuracy, cbs=ModelResetter)
learn.fit_one_cycle(15, 3e-3)

epoch	train_loss	valid_loss	accuracy	time
0	3.041824	2.657038	0.445638	00:01
1	2.138109	1.801325	0.470947	00:01
2	1.706817	1.811761	0.436361	00:01
3	1.530675	1.894353	0.373617	00:01
4	1.375850	1.788797	0.450928	00:01
5	1.222879	1.682562	0.478353	00:01
6	1.038785	1.689814	0.497152	00:01
7	0.894064	1.736756	0.507975	00:01
8	0.791512	1.795761	0.511068	00:01
9	0.720978	1.885496	0.516113	00:01
10	0.670370	1.939440	0.519043	00:01
11	0.633727	1.986372	0.519613	00:01
12	0.609256	2.031443	0.520345	00:01
13	0.594163	2.023642	0.519775	00:01
14	0.585938	2.040995	0.518392	00:01

The model seems to be doing worse. Validation loss appears to be really bad now.

01:32:39 - Exploding and vanishing gradients

Very deep models can be hard to train due to exploding or vanishing gradients.
Doing a lot of matrix multiplications across layers can give you very big or very small results.
- This can also cause gradients to grow.
This is because numbers in a computer aren't stored precisely: they're stored as floating point numbers.
- Really big or small numbers become very close together and differences become practically 0.
Lots of ways to deal with this:
- Batch norm.
- Smart initialisation.
One simple technique is to use an RNN architecture called LSTM.

01:36:29 - LSTM

Designed such that there's mini neural networks that decide how much previous state should be kept or discarded.
Main detail: can replace matrix multiplication with LSTMCell sequence below:

In [32]:

class LSTMCell(nn.Module):
    def __init__(self, num_inputs, num_hidden):
        self.forget_gate = nn.Linear(num_inputs + num_hidden, num_hidden)
        self.input_gate = nn.Linear(num_inputs + num_hidden, num_hidden)
        self.cell_gate = nn.Linear(num_inputs + num_hidden, num_hidden)
        self.output_gate = nn.Linear(num_inputs + num_hidden, num_hidden)
        
    def forward(self, input, state):
        h, c = state
        h = torch.stack([h, input], dim=1)
        forget = torch.sigmoid(self.forget_gate(h))
        c = c * forget
        inp = torch.sigmoid(self.input_gate(h))
        cell = torch.tanh(self.cell_gate(h))
        c = c + inp * cell
        outgate = torch.sigmoid(self.output_gate(h))
        h = outgate * torch.tanh(c)
        return h, (h, c)

RNN that uses LSTMCell is called LSTM

In [35]:

class LMModel6(Module):
    def __init__(self, vocab_sz, n_hidden, n_layers):
        self.input2hidden = nn.Embedding(vocab_sz, n_hidden)
        self.rnn = nn.LSTM(n_hidden, n_hidden, n_layers, batch_first=True)
        self.hidden2output = nn.Linear(n_hidden, vocab_sz)
        self.hidden = [torch.zeros(n_layers, bs, n_hidden) for _ in range(2)]

    def forward(self, x):
        res, h = self.rnn(self.input2hidden(x), self.hidden)
        self.hidden = [h_.detach() for h_ in h]
        return self.hidden2output(res)
    
    def reset(self):
        for h in self.hidden:
            h.zero_()

In [36]:

learn = Learner(dls, LMModel6(len(vocab), 64, 2), loss_func=loss_func, metrics=accuracy, cbs=ModelResetter)
learn.fit_one_cycle(15, 1e-2)

epoch	train_loss	valid_loss	accuracy	time
0	3.012093	2.717043	0.217204	00:01
1	2.218240	2.029626	0.351644	00:01
2	1.658893	1.942375	0.456868	00:01
3	1.369292	2.049241	0.499268	00:02
4	1.129897	2.095356	0.525228	00:01
5	0.875359	1.999668	0.559977	00:01
6	0.627453	1.819402	0.649333	00:01
7	0.430494	1.841276	0.679118	00:01
8	0.288909	1.796739	0.671224	00:01
9	0.193200	1.857251	0.688558	00:01
10	0.134485	1.752179	0.709554	00:01
11	0.096178	1.834427	0.721191	00:01
12	0.072396	1.826156	0.721680	00:01
13	0.059678	1.836468	0.726237	00:01
14	0.053572	1.843715	0.725993	00:01

01:40:00 - Questions

Can we use regularisation to make RNN params close to identity matrix?
- Will look at regularisation approaches.
Can you check if activations are exploding or vanishing?
- Yes. You can output activations of each layer with print statements.

01:42:23 - Regularisation using Dropout

Dropout is basically deleting activations at random.
- By removing activations at random, no single activations can become too "overspecialised"
Dropout implementation:

In [37]:

class Dropout(nn.Module):
    def __init__(self, p):
        self.p = p
        
    def forward(self, x):
        if not self.training:
            return x
        
        mask = x.new(*x.shape).bernoulli_(1-p)
        return x * mask.div_(1-p)

A bermoulli random variable is a bunch of 1 and 0s with 1-p probability of getting a 1.
- By multipying that by our input, we end up removing some layers.

01:47:16 - AR and TAR regularisation

Jeremy has only seen in RNNs.
AR (for activation regularisation)
- Similar to Weight Decay.
- Rather than adding a multiplier * sum of squares * weights.
- We add multiplier * sum of squares * activations.
TAR (for temporal activation regularisation).
- TAR is used to calculate the difference of activations between each layer.

01:49:09 - Weight tying

Since predicting the next word is about converting activations to English words.
An embedding is about converting words to activations.
Hypothesis: since they're roughly the same idea, can't they use the same weight matrix?
- Yes! This appears to work in practice.

In [44]:

class LMModel7(Module):
    def __init__(self, vocab_sz, n_hidden, n_layers, p):
        self.i_h = nn.Embedding(vocab_sz, n_hidden)
        self.rnn = nn.LSTM(n_hidden, n_hidden, n_layers, batch_first=True)
        self.drop = nn.Dropout(p)
        self.h_o = nn.Linear(n_hidden, vocab_sz)
        
        # This new line of code ensures that the weights of the embedding will always be the same as linear weights.
        self.h_o.weight = self.i_h.weight
        self.h = [torch.zeros(n_layers, bs, n_hidden) for _ in range(2)]
        
    def forward(self, x):
        raw, h = self.rnn(self.i_h(x), self.h)
        out = self.drop(raw)
        self.h = [h_.detach() for h_ in h]
        return self.h_o(out), raw, out
    
    def reset(self):
        for h in self.h:
            h.zero_()

01:51:00 - TextLearner

We pass in RNNRegularizer callback:

In [45]:

learn = Learner(
    dls,
    LMModel7(len(vocab), 64, 2, 0.5),
    loss_func=CrossEntropyLossFlat(),
    metrics=accuracy, cbs=[ModelResetter, RNNRegularizer(alpha=2, beta=1)])

Or use the TextLearner which passes it for us:

In [48]:

learn = TextLearner(
    dls,
    LMModel7(len(vocab), 64, 2, 0.5),
    loss_func=CrossEntropyLossFlat(),
    metrics=accuracy
)

In [49]:

learn.fit_one_cycle(15, 1e-2, wd=0.1)

epoch	train_loss	valid_loss	accuracy	time
0	2.583476	2.096658	0.452637	00:01
1	1.619627	1.360577	0.628581	00:01
2	0.874051	0.910055	0.776449	00:01
3	0.441865	0.803067	0.819824	00:02
4	0.226623	0.656291	0.852620	00:01
5	0.127504	0.559974	0.862549	00:01
6	0.077614	0.450280	0.887614	00:01
7	0.056029	0.559095	0.874430	00:02
8	0.038954	0.491778	0.877441	00:02
9	0.030191	0.524026	0.879964	00:01
10	0.024196	0.501803	0.877767	00:01
11	0.020231	0.504942	0.877686	00:01
12	0.016885	0.424204	0.889811	00:01
13	0.014706	0.461084	0.884603	00:01
14	0.013446	0.466097	0.884603	00:01

We've now reproduced everything in AWD LSTM, which was state of the art a few years ago.

01:52:48 - Conclusions

Go idea to connect with other people in your community or forum who are along the learning journey.