I've been asked what's the biggest thing in 2024 other than LLMs. It's Robotics. Period. We are ~3 years away from the ChatGPT moment for physical AI agents. We've been cursed by the Moravec's paradox for too long, which is the counter-intuitive phenomenon that "tasks that humans find easy are extremely hard for AI, and vice versa". 2024 will be remembered as the first year that the AI community fights back big time against the curse. We will not win immediately, but we will be on the path of winning. In 2023, we've caught a glimpse of the future foundation models and platforms for robots: - Multimodal LLMs with robot arms as a physical I/O device: VIMA, PerAct, RvT (NVIDIA), RT-1, RT-2, PaLM-E (Google), RoboCat (DeepMind), Octo (Berkeley, Stanford, CMU), etc. - Algorithms that bridge the gap between System 1 high-level reasoning (LLMs) and System 2 low-level control: Eureka (NVIDIA), Code as Policies (Google), etc. - Insane amounts of progress on robust hardware: Tesla Optimus @elonmusk, Figure @adcock_brett, 1X @ericjang11, Apptronik, Sanctuary, Agility+Amazon, Unitree, etc. - Data has always been the Achilles' heel of robotics. The research community is coming together to curate the next ImageNet, such as the Open X-Embodiment (RT-X) dataset. It's still not diverse enough, but a baby step is a major step. - Simulation and synthetic data will play a critical role in solving robot dexterity and even computer vision in general. (1) NVIDIA Isaac can simulate reality at 1000x faster than real-time. The incoming data stream scales as compute scales. (2) Photorealism can be enabled by hardware-accelerated raytracing. The realistic renderings also come with groundtruth annotations for free, such as segmentation, depth, 3D pose, etc. (3) Simulators can even multiply real-world data to create much larger datasets, greatly reducing the expensive human demonstration efforts. MimicGen (NVIDIA) is a representative example. I'm all in, personally. The best is yet to come.

Before we have a million robots in the physical world, we will first see a billion embodied agents in virtual worlds. Gaming is the second major area I'm dedicated to in 2024. AI and Gaming are born for each other, and their happy marriage is just getting started. On one hand, open-ended games provide a "primordial soup" for generalist AI to emerge. An agent's capabilities are upper-bounded by the complexity of the world it lives in. Minecraft is a prime example. In 2023, we saw an explosion of new algorithms enabled by Minecraft. To name a few: - MineCLIP (from our team's open-source framework MineDojo): learn reward functions by watching 100,000s of YouTube gameplay videos. - VPT (OpenAI): imitate behavior by pseudo-labeling actions from YouTube. - Voyager: in-context lifelong learning with an explicit Skill Library of code. - STEVE-1: guide actor to follow commands in MineCLIP's latent space, inspired by DALLE-2 - And many more: DEPS, Jarvis-1, DreamerV3 ... Besides Minecraft, there are a lot more games that require extremely advanced perception, agility, exploration, reasoning, and planning. We are just starting to scratch the surface. I believe games (and simulation in general) will provide the next trillion high-quality tokens to train our foundation models. What's cool is that these tokens are actively selected by the agent itself through exploration. It can choose to experiment with things that maximally reduce its internal uncertainties - kind of like how human curiosity works. On the other hand, AI will lead to a paradigm shift in the Gaming industry. This year, we see a surge of community interest in Stanford Smallville, where 25 AI agents inhabit a digital town. They go to work, gossip, organize socials, make new friends, and even fall in love. It's an exciting experiment, but we still have not felt any impact on the real games out there. This is because our LLMs are too boring and too expensive. If you take a look at Smallville's chat log, you will find that the conversations are not fun at all. No parents talk to their kids in such polite manner. The unit economy of deploying so many agents also does not make sense at scale. That being said, I believe 2024 is an inflection point. The Digital Westworld is coming, and will transform the industry once and for all. Games will feel truly alive. The characters will interact with humans and each other, form relationships, take consistent actions over their lifetime, and react in human-like ways. Each game will have infinite replay value, and each player will have unique and tailored experience.