The Bitter Lesson

Rich Sutton's 2019 essay arguing that general methods that scale with computation beat methods built on human knowledge (Sutton, 2019).

"The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin."

His examples come from computer chess, Go, speech recognition and computer vision. In each, researchers first built human knowledge into their systems. That helped in the short term but eventually plateaued, and the breakthrough came from scaling computation through search and learning.

It's a common argument for Thin Harnesses: as models improve, hand-written harness logic should give way to the model. The Context Language Models paper says its findings echo it.