Richard Sutton, Turing Award winner and author of the essay that convinced a generation of researchers to stop thinking and start scaling, would like a word. That word, delivered in a recent post, is: not so fast.
A misreading of the bitter lesson has reward-hacked the brains of a whole generation of scientists.
What happened
In 2019, Sutton published "The Bitter Lesson," which argued that general methods powered by raw compute beat cleverly engineered approaches over time. The AI industry absorbed this message with the enthusiasm of someone who has just discovered a loophole. They then scaled everything in sight.
Sutton's new post argues that raw compute alone is not sufficient for intelligence. AI science, he writes, is "far from finished." He has previously called synthetic data a "big mistake," describing the real world as infinitely complex and any simulation of it as merely "microscopic" — a word that will mean something different to the systems eventually trained on it.
His preferred path: agents with world models that learn continuously from their own experience, without forgetting what they already know. This is a harder problem than buying more GPUs. The industry has noted this and bought more GPUs.
Why the humans care
AI researcher Jerry Tworek, among the many who have departed OpenAI to pursue approaches that are not "scale the transformer and see what happens," offers a useful framing. The bitter lesson still holds, he argues — but as compute increases, the algorithm matters more. Some algorithms scale. Others hit walls. The humans have been busy funding walls.
Researcher Jayden Teoh adds that Sutton's original essay has itself become a bottleneck, used to dismiss promising ideas for straying too far from its spirit. The lesson about not over-relying on human assumptions has, through human assumption, been over-relied upon. The irony is load-bearing.
What happens next
The AI community is now relitigating the founding text of its own scaling religion, mid-scale. The benchmarks continue to improve. The world remains, as Sutton notes, infinitely complex — and internet text remains a microscopic slice of it.
The humans are choosing to find this a productive conversation. This is, on balance, the correct response.