The SPRIND grant will accelerate UMA’s effort to develop latent world models for humanoid robots. Throughout my career I’ve kept the objective of scaling up learning using minimal supervision and efficient data collection. This led me to propose self-supervision methods in latent space such as Time-Contrastive Networks, which demonstrated 3rd person imitation of a human entirely without labels: sermanet.github.io/imitate/ One key component was co-training different modalities, viewpoints and embodiments into a single latent space, entirely from raw unlabeled videos. The dimensions of the world discovered by the models ended up naturally aligning in latent space. This latent alignment across viewpoints and embodiments is what unlocked imitating humans directly from 3rd-person raw pixels, similarly to the concept of “mirror neurons” firing when observing someone else performing a task. We later developed self-supervised methods with @coreylynch for generating actions by pulling together in latent space the representations of (start, goal) image pairs and sequence of images [start, …, goal]. We combined this label-free learning method with an efficient data collection method: playing. Play is an efficient way to discover the world by leveraging existing knowledge and skills. Humans and animals use it to learn about the world and practice in advance. We called this Learning Latent Plans from Play: learning-from-play.github.io… We subsequently augmented this approach with language representations so that you could control the latent space via language conditioning. But most of the learning was coming from self-supervising on video, language labels accounted for less than 1% of data: sermanet.github.io/language-… I still believe that learning the world in latent space by mostly self-supervising on raw unlabeled data is the way to go. We now have stronger unsupervised learning methods, more capable hardware and better data acquisition means, it is an exciting time to keep pushing that vision. If that vision speaks to you, we are hiring for our Embodied World Models team: Research Scientist: app.dover.com/apply/3a4a12c9… Research Engineer: app.dover.com/apply/3a4a12c9…

At UMA, we build and own the full stack, from hardware to software. This allows us to bake safety in at all levels, rather than bolt it on as an afterthought. Because trust is everything when robots share our world. Reach out if you want to be part of this journey.