Notes by Lex Toumbourou

When the data your model sees has diverged significantly from the training dataset, it's called a Domain Shift.

Howard et al. (2020) (pg. 104)

In the 2019 Data Science Bowl, PBS Kids ran a Kaggle competition to determine how well a player would do on a challenge based on their behavior so far. Because the training data was particular to the current version of the game, including very level-specific data like the coordinates of mouse clicks, the models would be sensitive to tiny changes made to the game. Changing the order of levels or moving a sprite could trigger a significant domain shift, making the current production data effectively Out-of-Domain.

Cover New Vans Vs. Old Vans by Danny Lopez on Flickr.