CS PhD Student @UW | Previously @UCSanDiego | Intern @ToyotaResearch @allen_ai
Seattle, WA
Joined July 2019
- Tweets163
- Following716
- Followers915
- Likes1.8K
Ge Yan retweeted
Context isn’t just what you put in a prompt. It’s also what stands between a robot and its goal. Robot policies in everyday environments must generalize not only across objects, but also across the spatial contexts in which they appear.
Excited to share COWBOY 🤠 (CoRL 2026) — our sim-to-real recipe for contextual whole-body manipulation. One generalist grasping policy coordinates base, arm, dexterous hand, and active vision end-to-end, adapting its motion and grasps to object and scene geometry. Trained in simulation, deployed zero-shot in the wild.
w/ @JasonJZLiu @LiuMuxinl
🧵(1/N)
Ge Yan retweeted
I’m super excited to share that I’ve joined @physical_int as a Research Scientist!
Ge Yan retweeted
Every robot already carries contact sensors: its own joints. When a robot touches something, its joints drift slightly off their commands, and the encoders pick it up. Legged robots and humanoids already leverage this signal to walk blind - but is it enough for blind precise assembly?
We find that it is! Our policies can insert pegs, mesh gears, and thread nuts using only the robot's joint positions - no camera, no tactile, and no object poses. And the peg policy works zero-shot in the real world! 🧵
w/ my amazing co-first author @younghyo_park, and advised by @kushalk_ and @pulkitology.
Ge Yan retweeted
Today, Dream Labs is out of stealth.
In February 2026, while we were at NVIDIA, @scott_e_reed @youliangtan @GeYunhao and I started calling a new kind of robot model a World Action Model, one that learns how the world moves from video, then uses that to decide how to act.
Today, we’re releasing Dream-1 through a public API that anyone in the world can use.
When you rent a car you’ve never driven, you don’t learn to drive all over again. Within a few minutes, it feels like your own. Robots should work the same way: a good model should be able to pick up a new robot and get to work, without being retrained from scratch.
That’s what we’ve spent the last 6 months on, and it’s what Dream-1 does. It controls bimanual robots with parallel grippers it has never seen before, zero-shot, on unseen tasks in unseen environments.
Generalization has been my North Star in robotics: first by distilling video world models into VLAs (DreamGen), then by learning to act from those world models directly (DreamZero), and now by transferring across robots (Dream-1). I’ve come to believe it’s the only direction that lasts. Pushing a narrow model from 90% to 99.9% on a single task is real work, but much of it gets washed away once a model that generalizes arrives.
Since the start of the company we’ve stayed single-minded about generalization.That focus is what got Dream-1 to control robots it was never trained on, on tasks it was never shown. And it’s what makes a public API possible at all: a model that only does one task on one robot can’t serve millions of users.
Dream-1 is still a research beta, and it will make mistakes. But it already drives cars it’s never seen.
Generalization wins in the end, and the end is closer than it looks.
Here’s Dream-1 taking the wheel of a robot it’s never seen. Full details 👇
Ge Yan retweeted
Belated life update: I defended my PhD at UCSD earlier this year. Huge thanks to my advisors @xiaolonw @haosu_twitr, and to all my amazing collaborators!
I’m now a postdoc at NYU working with @LerrelPinto @ylecun on world models + robots ✨
Ge Yan retweeted
ASPIRE is now open-source!
Since our initial release, we've made exciting progress:
🤖 Real-world robot manipulation with GPT-6 Astra.
🔄 Continual skill refinement on physical robots.
📈 Saturated simulation benchmarks: 99% on LIBERO-Pro (80 tasks), 100% on Robosuite (7 representative tasks), and 100% on BEHAVIOR-1K (2 representative tasks).
Code: github.com/NVlabs/ASPIRE
Project: research.nvidia.com/labs/gea…
🧵 More results in the thread below.
Ge Yan retweeted
Towards an agent-native CERN of AI robotics
AI agents can now run the whole robotics research loop in hours: code, train, evaluate. But only in simulation.
So we built a robot farm agents can control through an API.
First test: can they replicate recent robotics papers? 1/17
Ge Yan retweeted
Roadster event livestream xitter.cf/i/broadcasts/1yKAPwNgg…
Ge Yan retweeted
Image augmentation is widely used to make learned vision algorithms more robust. MobileVISTA is the natural follow up to our previous work VISTA, exploring how to augment demos in the image space for greater robustness to changes in robot base positions.
Mobile manipulation policies can break from base pose errors of just a few cm, which are common after navigation. Can we get pose generalization without additional demos? Introducing MobileVISTA: collect demos at one base pose, get a policy that works from many 🧵👇
Website: sdwistreich.github.io/mobile…
Paper: arxiv.org/abs/2610.07511
Ge Yan retweeted
(1/5) Everyone says dexterous manipulation needs real-world data. These policies were trained with none. Our robots mesh gears, thread nuts, and insert pegs zero-shot from RGB, and the same recipe works for locomotion across terrains much harder than before.
The missing piece wasn't a better RL algorithm, but a small tweak to what the robot practices.
Today we're releasing Success-Guided Sampling (SGS), a simple change to the task sampler that unlocks new levels of dexterity and agility with large-scale, end-to-end sim2real RL.
No demonstrations, no per-task reward tuning. Just PPO at scale.
Check out the policies yourself at sgs-rl.github.io! All videos at 1x speed! 🧵 🔈
Ge Yan retweeted
A paradox in robot learning:
robot policies are spatially dumb and data-inefficient but their image features are spatially robust and semantically rich.
Image encoders are spatially smart — features embed semantics + geometry + are multiview-consistent ...
but robot policies we build upon them are spatially dumb — hundreds of demos to learn simple pick+place and a small camera bump breaks them
To better bridge this gap, check out BIND: a new action head that binds each candidate robot action to its projecting 2D image feature(s)
BIND basically offers the network the info of '‘choosing this candidate robot action would move the robot EEF to this image feature."
Result: policies much more data efficient and robust to OOD viewpoints and OOD object positions (same rgb input, no 3D sensors, dense EEF trajectories out) 🧵
Ge Yan retweeted
A new negative-result investigation: closer-to-target (non-Gaussian) action priors did not improve the fine-tuning of Large Behavior Models (LBMs), i.e., robot policies pretrained on large, diverse datasets.
Prior work found that replacing the Gaussian prior of diffusion and flow-matching policies with a closer-to-target prior substantially improves policies trained from scratch. We hypothesized that this benefit would carry over to fine-tuning pretrained LBMs.
Across 100K+ simulation rollouts and 1250 hardware rollouts, we found that it does not.
Website: cxu-tri.github.io/non_gaussi…
Paper: arxiv.org/abs/2609.27070
🧵 (1/10)
Ge Yan retweeted
A video encoder trained only on colour turns out to compress 3D geometry almost perfectly.
Flex-π by @GeYan_21 @Jesse_Y_Zhang et all (UW, AI2) builds a policy on that.
We made it a lesson you can read 📖, listen to 🎧, watch 👀 or teach back 👨🏫👩🏫
engineermaxxing.com/veanors/…
Ge Yan retweeted
World-action models typically imagine the future in RGB – but are pixels really the right representation for robotics?
Our bet is no: RGB spends capacity on fine-grained details and variation that are often irrelevant for robot policies.
DINO features, point tracks, and depth capture more useful features like semantics, motion, and geometry.
But no single modality captures everything — how can we effectively combine them?
We introduce ✨ModAR✨, which predicts the future one modality at a time, with each prediction informing the next. We train from scratch and find that this formulation performs best.
Our 30.1M scratch-trained model even outperforms a 6B video-model-initialized model finetuned on the same data!
🧵 [1/8]
Ge Yan retweeted
Flex-π: One model, multiple ways to think about the future
Most World-Action Models (WAMs) learn to predict future RGB latents and actions.
But manipulation isn’t just about pixels.
For precise interaction, a robot also needs to reason about 3D geometry and what objects are.
Flex-π asks: can we learn all of these together?
Ge Yan retweeted
NEW RESEARCH: What if you apply JEPA's principles to predict robot actions?
This project involves @GeYan_21, @jinghao0115, @Jesse_Y_Zhang, and others.
Called Flex-π, A Multi-Stream World-Action Model with Compute Flexibility, it is a 6-billion-parameter robot manipulation policy from the University of Washington and the Allen Institute for AI (Ai2).
And it allows a robot to repair its own gripper!
Unlike world-action models that predict only future RGB image latents, it jointly denoises three streams in a shared latent space:
- RGB appearance
- 3D geometry (pointmaps)
- and object-centric DINO semantics
It makes use of a single 5B trunk plus a 1B action expert on top of frozen encoders (a Wan-2.2 video VAE for both RGB and pointmaps, and frozen DINOv3).
Per-stream dropout with cross-modality forcing lets one checkpoint run 56 input/output combinations, from an action-only fast path to full joint world-and-action generation.
It is evaluated on real bimanual YAM-workcell tasks (plate racking, utensil sorting, kitchen organization, gripper self-repair, soft-bag zipping) plus RoboTwin and LIBERO, against π0.5, ManiFlow, Fast-WAM and RDT.
The intuition is that world-action models predict the wrong thing, RGB pixels, and the fix is to predict 3D geometry and object semantics instead.
The paper's argument is that pixel-reconstruction latents carry no explicit signal for the 3D geometry or object semantics that manipulation needs, so Flex-π adds pointmap and DINO-semantic streams alongside RGB.
This is the JEPA and LeVJEPA critique (do not reconstruct pixels, predict in a semantic latent space) applied to robot action, and it is the same discarded-channel logic as FACT, ADEPT and Patch Policy in this file, except the thrown-away channel here is geometry-plus-semantics rather than force, touch or spatial resolution.
Ge Yan retweeted
People making grand claims about #robotics really need to spend time with actual robots. Debug the controller yourself. Deal with contact, latency, calibration, failures. And before declaring old problems obsolete or rebranding long-standing robotics ideas around foundation models, engage seriously with the field you are making claims about. #ECCV2026
Ge Yan retweeted
Last time I posted here, I was still at UCSD, fresh out of grad school and about to take my robotics research into industry.
A year later, I’m writing from China, building @Action_Int_AI with my co-founders @scy994 & @JingkangY.
It’s our shared bet on how to make intelligence act in the real world.
The story will unfold chapter by chapter, as we work our way toward robotics’ “aha” moment.
Our first chapter is Continuo.
Intelligence that stays with you.
We’re starting with human experience, and saying hello to the Ego community at #ECCV.
Come meet the team and experience Continuo!
Sep 10–12 · Booth #44 · @eccvconf
Ge Yan retweeted
this is all a long way to say: i don’t believe it’s “undertrained” in action only mode. it’s more likely that extra test time compute + being able to “cheat” by looking at future representations helps produce better actions
Ge Yan retweeted
thanks for the shoutout!
regarding whether action only being undertrained causes it to perform worse or if making better use of test time compute is the main factor:
Early versions of the model were trained *without* randomly dropping inputs and output predictions.
In these settings, we basically had 1 model per generation mode, so they got equivalent training steps exactly matched with inference modes.
Even there, action only was also outperformed by all other additional modality checkpoints.
We then realized it made more sense to make the model able to just use 1 model checkpoint for all 56 possible input/output combinations, resulting in basically the same performance but without all the extra checkpoints!