Noel Webb

In 2017, over breakfast at a cottage, I had the distinct honour of meeting @RichardSSutton for the first time. When I asked what he was working on, he said they were solving poker. I had no idea who I was actually speaking with at the time. I was quickly humbled when I began to understand the depth of the work behind that deceptively simple answer. Today, Rich is recognized as one of the founders of reinforcement learning, a recipient of the Turing Award, the author of The Bitter Lesson, and co-author of Welcome to the Era of Experience. His argument in this conversation with Sonya Huang and Alfred Lin at @Sequoia is fundamental. The rest of the field appears increasingly convinced that enough stored knowledge can replace learning. It cannot. Rich’s point about today’s GPTs is blunt: “Their weights never change.” For genuine continuous learning, the weights of a model must remain capable of changing as it gains experience. Today’s GPTs (read LLMs) do not do this while they are being used. They may retain context or retrieve an external memory, but the underlying model remains frozen. More context is not the same as learning. The solution, however, is not as simple as updating the entire model after every interaction. As @kjaved_ explains, “I’m going to update my whole model with that one sample.” The result is that the model’s existing knowledge can be negatively affected or destroyed. This is catastrophic forgetting. That is the technical challenge: building a system that can continually update its weights and learn from new experience without destabilizing everything it has already learned. Sutton and Javed argue that this is an algorithmic problem, and one that can be solved. As Rich says, “All learning is continual. It’s just learning.” What struck me was the skepticism from Sonya and Alfred, even as it came through genuine curiosity, about something that should now be evident. A system that stops learning when it is deployed is not continuously learning. This may be the fundamental limitation of the current AI paradigm.