Lesson 6 - Global Warning Sentiment Classifier

This notebook walks through building an RNN classification model using the Sentiment of Climate Change by Crowdflower.

In [1]:

import numpy as np
from functools import partial
import pandas as pd

import torchtext
import torch
from torch import nn
from torch.nn import functional as F
import spacy
import fastai.nlp
from fastai.learner import fit
from fastai.core import V
from fastai.metrics import accuracy
from fastai.lm_rnn import seq2seq_reg, repackage_var
from fastai.nlp import RNN_Learner
from IPython.display import display

from pathlib import Path

/Users/lex/anaconda3/envs/fastai-cpu/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/Users/lex/anaconda3/envs/fastai-cpu/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192, got 176
  return f(*args, **kwds)
/Users/lex/anaconda3/envs/fastai-cpu/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/Users/lex/anaconda3/envs/fastai-cpu/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192, got 176
  return f(*args, **kwds)
/Users/lex/anaconda3/envs/fastai-cpu/lib/python3.6/site-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.
  from numpy.core.umath_tests import inner1d

In [5]:

df = pd.read_csv('./data/global_warming_tweets.csv', encoding='latin1')

In [3]:

df.head()

Out[3]:

	tweet	existence	existence.confidence
0	Global warming report urges governments to act...	Yes	1.0000
1	Fighting poverty and global warming in Africa ...	Yes	1.0000
2	Carbon offsets: How a Vatican forest failed to...	Yes	0.8786
3	Carbon offsets: How a Vatican forest failed to...	Yes	1.0000
4	URUGUAY: Tools Needed for Those Most Vulnerabl...	Yes	0.8087

Drop all Tweets that aren't about global warming:

In [7]:

df.existence.value_counts(dropna=False)

Out[7]:

Y      2554
NaN    1865
N      1053
Yes     557
No       61
Name: existence, dtype: int64

In [8]:

df = df.dropna()

In [9]:

df.existence = df.existence.replace({'Yes': 'Y', 'No': 'N'})

In [10]:

df.existence.value_counts(dropna=False)

Out[10]:

Y    3111
N    1114
Name: existence, dtype: int64

In [11]:

df = df.sample(frac=1, random_state=42)

In [12]:

train_df = df[:-1000]
val_df = df[-1000:]

In [13]:

train_df.shape

Out[13]:

(3225, 3)

In [14]:

val_df.shape

Out[14]:

(1000, 3)

In [15]:

train_df.existence.value_counts()

Out[15]:

Y    2384
N     841
Name: existence, dtype: int64

In [16]:

val_df.existence.value_counts()

Out[16]:

Y    727
N    273
Name: existence, dtype: int64

Torchtext basics

The data.Field Torchtext classes describe how to process a set of text. In this example, we intend to lowercase each string:

HELLO -> hello

and tokenise it with Spacy.

In [17]:

nlp = spacy.load('en')

/Users/lex/anaconda3/envs/fastai-cpu/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/Users/lex/anaconda3/envs/fastai-cpu/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192, got 176
  return f(*args, **kwds)

In [19]:

train_df.iloc[0].tweet

Out[19]:

"Putting a Price on Climate Change: The People's World Conference on Climate Change in Cochabamba, Bolivia kicks... http://bit.ly/8XnI5N #p2"

In [20]:

tokens = nlp(train_df.iloc[0].tweet)

In [21]:

[t for t in tokens]

Out[21]:

[Putting,
 a,
 Price,
 on,
 Climate,
 Change,
 :,
 The,
 People,
 's,
 World,
 Conference,
 on,
 Climate,
 Change,
 in,
 Cochabamba,
 ,,
 Bolivia,
 kicks,
 ...,
 http://bit.ly/8XnI5N,
 #,
 p2]

In [35]:

TEXT_FIELD = torchtext.data.Field(lower=True, tokenize="spacy")
# Tokenisation is not applied when `sequential=False`.
LABEL_FIELD = torchtext.data.Field(sequential=False)

We'll also need a Dataset. Which in Torch text is an object that stores a list of examples which includes the text, it's label and the Field used to process it.

In [57]:

class DataFrameDataset(torchtext.data.Dataset):

    def __init__(self, df, text_field, label_field, is_test=False, **kwargs):
        fields = [('text', text_field), ('label', label_field)]
        examples = []
        for i, row in df.iterrows():
            label = row.existence if not is_test else None
            text = row.tweet
            examples.append(torchtext.data.Example.fromlist([text, label], fields))

        super().__init__(examples, fields, **kwargs)

    @staticmethod
    def sort_key(ex):
        return len(ex.text)

    @classmethod
    def splits(cls, text_field, label_field, train_df, val_df=None, test_df=None, **kwargs):
        train_data, val_data, test_data = (None, None, None)

        if train_df is not None:
            train_data = cls(train_df.copy(), text_field, label_field, **kwargs)
        if val_df is not None:
            val_data = cls(val_df.copy(), text_field, label_field, **kwargs)
        if test_df is not None:
            test_data = cls(test_df.copy(), text_field, label_field, True, **kwargs)

        return tuple(d for d in (train_data, val_data, test_data) if d is not None)

splits refers to the train, val and test split and is a big part of the Torchtext lingo. Here's how you would construct split:

In [58]:

train_ds, val_ds, test_ds = DataFrameDataset.splits(
  text_field=TEXT_FIELD, label_field=LABEL_FIELD, train_df=train_df, val_df=val_df, test_df=val_df)

We can now prepare our TEXT_FIELD by building the vocab on it, which simply maps tokens to ints:

In [59]:

TEXT_FIELD.vocab

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-59-c4c60c8136fc> in <module>()
----> 1 TEXT_FIELD.vocab

AttributeError: 'Field' object has no attribute 'vocab'

In [61]:

TEXT_FIELD.build_vocab(train_ds)

In [62]:

TEXT_FIELD.vocab

Out[62]:

<torchtext.vocab.Vocab at 0x1c3db4d940>

In [ ]:

In [63]:

TEXT_FIELD.vocab.stoi['global']

Out[63]:

In [64]:

TEXT_FIELD.vocab.itos[3]

Out[64]:

'global'

In [65]:

LABEL_FIELD.build_vocab(train_ds)

In [66]:

LABEL_FIELD.vocab.stoi

Out[66]:

defaultdict(<function torchtext.vocab._default_unk_index()>,
            {'<unk>': 0, 'Y': 1, 'N': 2})

In PyTorch, a dataset is very much like a list: it can be iterated through and we can also fetch individual items by id:

In [67]:

train_ds[0].text

Out[67]:

['putting',
 'a',
 'price',
 'on',
 'climate',
 'change',
 ':',
 'the',
 'people',
 "'s",
 'world',
 'conference',
 'on',
 'climate',
 'change',
 'in',
 'cochabamba',
 ',',
 'bolivia',
 'kicks',
 '...',
 'http://bit.ly/8xni5n',
 '#',
 'p2']

In [68]:

next(iter(train_ds)).text

Out[68]:

['putting',
 'a',
 'price',
 'on',
 'climate',
 'change',
 ':',
 'the',
 'people',
 "'s",
 'world',
 'conference',
 'on',
 'climate',
 'change',
 'in',
 'cochabamba',
 ',',
 'bolivia',
 'kicks',
 '...',
 'http://bit.ly/8xni5n',
 '#',
 'p2']

What we want is a tool to batch our examples up for us. Since an RNN is a sequence model, which can support variable length sequences (lists of data), what we want is a batching strategy that will let us train on multiple sentences at a time but still let us use a for loop over them. Torchtext solves the problem like this:

First element   |  Second element   |
['rt']          |  ['the']
['global']      |  ['warming']
['Hello']       |  ['world']

So the first element is 64x1, which is the first word in 64 different sequences and so on.

Torchtext provides a "bucket iterator" which handles batching. In Torchtext, they call these iterators what Torchvision called data loaders (no idea why that is).

Let's create an iterator for each split:

In [69]:

train_iter, val_iter, test_iter = torchtext.data.BucketIterator.splits(
    (train_ds, val_ds, test_ds), batch_sizes=(16, 256, 256), sort_key=lambda x: len(x.text))

In [70]:

train_batch = next(iter(train_iter))

In [71]:

train_batch.text.shape

Out[71]:

torch.Size([22, 16])

What we have is a sentence length of 34, with a batch size of 16.

In [72]:

train_batch = next(iter(train_iter))

In [73]:

train_batch.text.shape

Out[73]:

torch.Size([31, 16])

Notice how the sequence length changes each time? An RNN can handle that just fine.

Fast.AI provides a convenience wrapper around all this called fastai.nlp.TextData. Let's use that and make sure it appears to work the same:

In [74]:

PATH = Path('./data/global-sentiment')
PATH.mkdir(exist_ok=True)

In [75]:

bs = 16

In [76]:

text_data = fastai.nlp.TextData.from_splits(PATH, (train_ds, val_ds, test_ds), bs=bs, text_name='text', label_name='label')

In [77]:

train_batch, y = next(iter(text_data.trn_dl))

In [79]:

text_data.trn_dl.src

Out[79]:

<torchtext.data.iterator.BucketIterator at 0x1c3db4d048>

In [80]:

train_batch.shape

Out[80]:

torch.Size([22, 16])

In [81]:

train_batch, y = next(iter(text_data.trn_dl))

In [82]:

train_batch.shape

Out[82]:

torch.Size([31, 16])

Simple RNN classifier

It's the same. Okay, so now we understand TextData, let's build a simple RNN model. I will use a for loop initially, instead of using the nn.RNN class.

In [83]:

class RNNClassifier(torch.nn.Module):
    def __init__(self, embedding_dim, hidden_size, vocab_size):
        super().__init__()
        
        self.hidden_size = hidden_size
        
        self.embedding = torch.nn.Embedding(vocab_size, embedding_dim)
        self.input_layer = torch.nn.Linear(embedding_dim + hidden_size, hidden_size)
        self.hidden_layer = torch.nn.Linear(hidden_size, hidden_size)
        self.output_layer = torch.nn.Linear(hidden_size, 3)
        
    def forward(self, words):
        batch_size = words[0].size(0)
        
        hidden = V(torch.zeros(batch_size, self.hidden_size))
        
        for word in words:
            e = self.embedding(word)
            concat = torch.cat([hidden, e], dim=1)
            inp = self.input_layer(concat)
            inp = torch.nn.functional.relu(inp)

            hidden = torch.nn.functional.tanh(self.hidden_layer(inp))
            
        return torch.nn.functional.log_softmax(self.output_layer(hidden), dim=-1)

In [84]:

emb_size = 200
hidden_size = 256
vocab_size = len(TEXT_FIELD.vocab.stoi)

model = RNNClassifier(emb_size, hidden_size, vocab_size)
opt = torch.optim.Adam(model.parameters(), 1e-3)

In [85]:

learner = RNN_Learner.from_model_data(model, text_data)

In [86]:

learner.lr_find()

HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))

 77%|███████▋  | 155/202 [00:15<00:04,  9.92it/s, loss=2.27]

In [55]:

learner.sched.plot()

In [56]:

learner.fit(0.05, 1, metrics=[accuracy])

HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))

epoch      trn_loss   val_loss   accuracy                    
    0      0.578731   0.637823   0.688458

Out[56]:

[array([0.63782]), 0.688458434221146]

In [58]:

learner.fit(0.05, 3, metrics=[accuracy], cycle_mult=2)

HBox(children=(IntProgress(value=0, description='Epoch', max=7), HTML(value='')))

epoch      trn_loss   val_loss   accuracy                    
    0      0.560382   0.727254   0.533091  
    1      0.581105   0.616332   0.720541                    
    2      0.578248   0.671503   0.613751                    
    3      0.561999   0.624166   0.620006                    
    4      0.571391   0.749199   0.481487                    
    5      0.582996   0.760548   0.549687                    
    6      0.561095   0.701809   0.635997

Out[58]:

[array([0.70181]), 0.6359967715899919]

Results are not amazing. We know that loosing the hidden state in between forward passes is not ideal. Can we try the stateful RNN model.

In [76]:

class StatefulRNNClassifier(nn.Module):
    def __init__(self, vocab_size, n_fac, n_hidden, batch_size):
        self.vocab_size = vocab_size
        self.n_hidden = n_hidden

        super().__init__()
    
        self.e = nn.Embedding(vocab_size, n_fac)
        self.rnn = nn.RNN(n_fac, n_hidden)
        self.l_out = nn.Linear(n_hidden, 3)
        self.init_hidden(batch_size)
        
    def forward(self, cs):
        batch_size = cs[0].size(0)

        if self.h.size(1) != batch_size:
            self.init_hidden(batch_size)

        outp, h = self.rnn(self.e(cs), self.h)

        self.h = repackage_var(h)

        return F.log_softmax(
            self.l_out(outp[-1]), dim=-1).view(-1, 3)
    
    def init_hidden(self, bs):
        self.h = V(torch.zeros(1, bs, self.n_hidden))

In [87]:

rank_3_tensor = V(torch.zeros(1, 16, 256))

In [88]:

rank_3_tensor.shape

Out[88]:

torch.Size([1, 16, 256])

In [89]:

rank_3_tensor.view(-1, 256).shape

Out[89]:

torch.Size([16, 256])

In [80]:

emb_size = 200
hidden_size = 256
vocab_size = len(TEXT_FIELD.vocab.stoi)

model = StatefulRNNClassifier(vocab_size, emb_size, hidden_size, 16)
opt = torch.optim.Adam(model.parameters(), 1e-3)

In [81]:

learner = RNN_Learner.from_model_data(model, text_data)

In [82]:

learner.model.reg_fn = partial(seq2seq_reg, alpha=2, beta=1)
learner.model.clip = 25.

In [83]:

learner.lr_find()

HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))

In [84]:

learner.sched.plot()

In [85]:

learner.fit(0.002, 1, metrics=[accuracy])

HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))

epoch      trn_loss   val_loss   accuracy                    
    0      0.603616   0.800989   0.643109

Out[85]:

[array([0.80099]), 0.6431093623890234]

In [86]:

learner.fit(0.002, 3, metrics=[accuracy], cycle_mult=3)

HBox(children=(IntProgress(value=0, description='Epoch', max=13), HTML(value='')))

epoch      trn_loss   val_loss   accuracy                    
    0      0.581494   0.728677   0.680942  
    1      0.583486   0.70855    0.67161                     
    2      0.564412   0.678601   0.677058                    
    3      0.574034   0.671648   0.662732                    
    4      0.557836   0.658013   0.670652                    
    5      0.553677   0.667217   0.641394                    
    6      0.567426   0.659511   0.640486                    
    7      0.556039   0.655617   0.647902                    
    8      0.542657   0.638358   0.668937                    
    9      0.539025   0.60367    0.705408                    
    10     0.532796   0.652138   0.675646                    
    11     0.515752   0.631832   0.689064                    
    12     0.508733   0.629989   0.687248

Out[86]:

[array([0.62999]), 0.6872477804681194]

Swap out the RNN for an LSTM

An LSTM does a better job at keeping track at longer term dependancies. For example:

"The dog which already walked ... was tired"

"The dogs which already walked ... were tired"

It does it by introducing some new parameters: an input, output and forget gate, which are used to figure out which data needs to be remembered throughout the sequence and which can be forgotten. We can easily replace our nn.RNN object with nn.LSTM as follows:

In [88]:

class StatefulLSTMClassifier(nn.Module):
    def __init__(self, vocab_size, n_fac, n_hidden, batch_size, nl):
        self.nl = nl
        self.vocab_size = vocab_size
        self.n_hidden = n_hidden
        super().__init__()
        self.e = nn.Embedding(vocab_size, n_fac)
        self.rnn = nn.LSTM(n_fac, n_hidden, self.nl)
        self.l_out = nn.Linear(n_hidden, 3, 2)
        self.init_hidden(batch_size)
        
    def forward(self, cs):
        batch_size = cs[0].size(0)
        if self.h[0].size(1) != batch_size:
            self.init_hidden(batch_size)

        outp, h = self.rnn(self.e(cs), self.h)
        self.h = repackage_var(h)
        return F.log_softmax(self.l_out(outp[-1]), dim=-1).view(-1, 3)
    
    def init_hidden(self, bs):
        self.h = (V(torch.zeros(self.nl, bs, self.n_hidden)),
                  V(torch.zeros(self.nl, bs, self.n_hidden)))

In [89]:

emb_size = 200
hidden_size = 256
vocab_size = len(TEXT_FIELD.vocab.stoi)

model = StatefulLSTMClassifier(vocab_size, emb_size, hidden_size, 16, 2)
opt = torch.optim.Adam(model.parameters(), 1e-3)

In [90]:

learner = RNN_Learner.from_model_data(model, text_data)

In [92]:

learner.lr_find()

HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))

 98%|█████████▊| 197/202 [00:17<00:00, 11.19it/s, loss=2.25]

In [93]:

learner.sched.plot()

In [94]:

learner.fit(0.1, 3, metrics=[accuracy], cycle_mult=3)

HBox(children=(IntProgress(value=0, description='Epoch', max=13), HTML(value='')))

epoch      trn_loss   val_loss   accuracy                    
    0      0.5871     0.579715   0.698295  
    1      0.536572   0.570973   0.720188                    
    2      0.494631   0.5172     0.73073                     
    3      0.435166   0.528466   0.738852                    
    4      0.387567   0.5914     0.753077                    
    5      0.301375   0.59567    0.762359                    
    6      0.269587   0.750503   0.749243                    
    7      0.247465   0.697706   0.769724                    
    8      0.200259   0.783226   0.770682                    
    9      0.193476   0.782393   0.770077                    
    10     0.130344   1.004984   0.78067                     
    11     0.120232   0.835018   0.780872                    
    12     0.109632   0.982337   0.772801

Out[94]:

[array([0.98234]), 0.7728006456820016]

In [95]:

learner.fit(0.1, 3, metrics=[accuracy], cycle_mult=3)

HBox(children=(IntProgress(value=0, description='Epoch', max=13), HTML(value='')))

epoch      trn_loss   val_loss   accuracy                     
    0      0.088587   0.921525   0.744804  
    1      0.078537   1.08459    0.776987                     
    2      0.092496   0.824993   0.771943                     
    3      0.083295   0.967652   0.775575                     
    4      0.074912   0.947035   0.781326                     
    5      0.069188   0.84633    0.768412                     
    6      0.057192   1.007749   0.779005                     
    7      0.057949   1.02991    0.768311                     
    8      0.085552   1.149548   0.780115                     
    9      0.078086   0.957208   0.764679                     
    10     0.08235    1.033652   0.786622                     
    11     0.085178   0.885615   0.76816                      
    12     0.061863   0.909589   0.773002

Out[95]:

[array([0.90959]), 0.7730024213075061]

Use the Fast.ai RNN classifier

The TextData has it's own model, let's try that.

In [143]:

bs = 64
bptt = 70
em_sz = 200
nh = 500
nl = 3

In [144]:

opt_fn = partial(torch.optim.Adam, betas=(0.7, 0.99))
learner = text_data.get_model(
    opt_fn, 1500, bptt, emb_sz=em_sz, n_hid=nh, n_layers=nl, 
    dropout=0.1, dropouti=0.4, wdrop=0.5, dropoute=0.05, dropouth=0.3)

In [145]:

learner

Out[145]:

SequentialRNN(
  (0): MultiBatchRNN(
    (encoder): Embedding(9065, 200, padding_idx=1)
    (encoder_with_dropout): EmbeddingDropout(
      (embed): Embedding(9065, 200, padding_idx=1)
    )
    (rnns): ModuleList(
      (0): WeightDrop(
        (module): LSTM(200, 500)
      )
      (1): WeightDrop(
        (module): LSTM(500, 500)
      )
      (2): WeightDrop(
        (module): LSTM(500, 200)
      )
    )
    (dropouti): LockedDropout(
    )
    (dropouths): ModuleList(
      (0): LockedDropout(
      )
      (1): LockedDropout(
      )
      (2): LockedDropout(
      )
    )
  )
  (1): PoolingLinearClassifier(
    (layers): ModuleList(
      (0): LinearBlock(
        (lin): Linear(in_features=600, out_features=3, bias=True)
        (drop): Dropout(p=0.1)
        (bn): BatchNorm1d(600, eps=1e-05, momentum=0.1, affine=True)
      )
    )
  )
)

In [146]:

learner.reg_fn = partial(seq2seq_reg, alpha=2, beta=1)
learner.clip = 0.3

In [147]:

learner.fit(3e-3, 4, wds=1e-6, cycle_len=1, cycle_mult=2, metrics=[accuracy])

HBox(children=(IntProgress(value=0, description='Epoch', max=15), HTML(value='')))

epoch      trn_loss   val_loss   accuracy                    
    0      0.473344   0.461434   0.776937  
    1      0.393316   0.511505   0.776079                    
    2      0.222065   0.460314   0.804782                    
    3      0.301563   0.464333   0.792474                    
    4      0.222063   0.600617   0.807254                    
    5      0.110187   0.698418   0.80231                     
    6      0.079147   0.717064   0.798073                     
    7      0.218971   0.778986   0.793886                    
    8      0.163321   0.707321   0.801907                    
    9      0.144864   0.77834    0.794794                    
    10     0.102619   0.725458   0.801755                     
    11     0.075765   0.777664   0.795046                     
    12     0.046675   0.905923   0.7841                       
    13     0.041044   0.885935   0.802159                     
    14     0.028916   0.875052   0.800192

Out[147]:

[array([0.87505]), 0.8001916868442293]

In [148]:

learner.save_encoder('enc1')

In [ ]:

learner.save('model_adam1')

Viewing val predictions

In [200]:

pd.set_option('display.max_colwidth', -1)

In [192]:

probs = learner.predict()
preds = np.argmax(probs, axis=1)
pred_labels = [LABEL_FIELD.vocab.itos[p] for p in preds]

val_results = val_df.copy()
val_results['pred'] = pred_labels
val_results['prob'] = np.exp(np.max(probs, axis=1))

View correct results

In [201]:

val_results[val_results['pred'] == val_results['existence']][['tweet', 'existence', 'pred']].head()

Out[201]:

	tweet	existence	pred
4080	@conoverbeth I think the report is valuable in terms of visible, observable climate change impacts in the US right now (but no invlvmnt)	Y	Y
5494	Is this the end of migration? - Climate Change, Environment - The Independent http://retwt.me/1MShX (via @terriferic)	Y	Y
1952	Only a Third of Americans Believe Global Warming is Human-Caused: By Bob Ellis on April 26th, 2010 Good news! What... http://bit.ly/9XGqPB	N	N
1096	Climate change & sustainability will be a key driver of future economic development. Listen at http://bit.ly/9Uytgl	Y	Y
3150	Snowstorm: E Coast Blizzard Tied to Climate Change - @TIME http://bit.ly/9TRnGU	Y	Y

Incorrect results

In [202]:

val_results[val_results['pred'] != val_results['existence']][['tweet', 'existence', 'pred']].head()

Out[202]:

	tweet	existence	pred
5548	Houston Chronicle: 'The heat goes on: After a blitz by climate change skep.. http://retwt.me/1MQpP (via @climateprogress)	Y	N
1978	Brochure on Global Warming 1992: Brochure on Global Warming 1992 added by: neham 0 comments http://bit.ly/bbxgVF	Y	N
4189	Global Warming In One State: Like clockwork, whenever there is any kind of change in the natural landscape here in... http://bit.ly/bk1xyA	Y	N
403	SEC warns publically held companies to advise investors of risks of #climate change. [link]	Y	N
64	@SooperMexican Global warming. Clearly.	Y	N

Most confident wrong predictions

In [198]:

val_results[val_results['pred'] != val_results['existence']].sort_values('prob', ascending=False)[['tweet', 'existence', 'pred']].head()

Out[198]:

	tweet	existence	pred
3071	EU Referendum: British Council spends _£3.5 million on climate change propaganda http://retwt.me/1LSjS (via @afashionfan)	N	Y
3863	RT @BillWilsonALG: Lindsay Graham getting public love from Obama. Someone needs to tell Graham that climate change is a proven lie. http://bit.ly/9w8RHw #tcot	N	Y
3710	The Great Global Warming Collapse http://bit.ly/9VeLfD	N	Y
3368	RT @owillis: RT @mmfa Brain Freeze: Conservative media still using winter weather 2 attack global warming http://bit.ly/9nKEcc #p2	N	Y
5955	Global warming? Well, not exactly. As if we could tell anyway: OCRegister (blog) What a hoot when global warming a... http://bit.ly/9PKvc3	N	Y

Most confident right predictions

In [199]:

val_results[val_results['pred'] == val_results['existence']].sort_values('prob', ascending=False)[['tweet', 'existence', 'pred']].head()

Out[199]:

	tweet	existence	pred
2098	RT @charles_baker 3,000 Businesses Create New Ad for Climate Change Action: http://bit.ly/cADHuR (via @FaisalSethi + @DoGoodHQ)	Y	Y
5703	Gregory Unruh: Oven Dial or Light Switch: Carbon Lock-in and Abrupt Climate Change: Well over a decade ago, I bec... http://huff.to/auGUFY	Y	Y
1237	RT @hereisawayjose Bolivia President on Global Warming http://su.pr/2YY1Lc "Main cause of Earth destruction is capitalism" #Green #EarthDay	Y	Y
4042	. http://bit.ly/aqwt24 global warming facts World leaders back embattled Pachauri	Y	Y
5753	Global warming? Great!: "The threat from climate change is serious, it is urgent, and it is growing," President Ob... http://bit.ly/a3FWD0	Y	Y

Making test predictions

In [151]:

learner.data.test_dl.src.sort = False
learner.data.test_dl.src.sort_within_batch = False
learner.data.test_dl.src.shuffle = False

In [153]:

probs = learner.predict(is_test=True)
preds = np.argmax(probs, axis=1)

In [205]:

pd.DataFrame({
    'id': val_df.index,
    'sentiment': [LABEL_FIELD.vocab.itos[p] for p in preds]}).head(10)

Out[205]:

	id	sentiment
0	4080	Y
1	5494	Y
2	1952	N
3	5548	N
4	1096	Y
5	3150	Y
6	1978	N
7	4189	N
8	1544	Y
9	403	N

In [ ]:

Fast.ai: Deep Learning 2018 - Lesson 6 (Global Warming Sentiment Classifier)

Lesson 6 - Global Warning Sentiment Classifier

Torchtext basics

Simple RNN classifier

Swap out the RNN for an LSTM

Use the Fast.ai RNN classifier

Viewing val predictions

View correct results

Incorrect results

Most confident wrong predictions

Most confident right predictions

Making test predictions