Building general purpose robotics at Amazon FAR. Previously Covariant CEO and Co-Founder, @OpenAI, @UCBerkeley PhD.
Joined December 2017
- Tweets164
- Following2.7K
- Followers4.2K
- Likes1.4K
Peter Chen retweeted
Common belief: Robot policies need continuous action heads; discrete actions are 𝐥𝐞𝐬𝐬 𝐧𝐚𝐭𝐢𝐯𝐞 𝐚𝐧𝐝 𝐥𝐞𝐬𝐬 𝐩𝐫𝐞𝐜𝐢𝐬𝐞.
❗We find policies w/ the simplest discrete representations 𝐦𝐚𝐭𝐜𝐡 𝐨𝐫 𝐛𝐞𝐚𝐭 continuous ones—in sim & real, in single-step & diffusion/flow settings.
Here's why 🧵
This is one of the most surprising projects FAR took on recently: (naive) discrete actions can be more precise than continuous actions for robot control. Check out this cool work by @ZejiYi et al
4/12 So, 𝐰𝐡𝐲 𝐝𝐨 𝐝𝐢𝐬𝐜𝐫𝐞𝐭𝐞 𝐚𝐜𝐭𝐢𝐨𝐧𝐬 𝐰𝐨𝐫𝐤? 𝐒𝐡𝐨𝐮𝐥𝐝𝐧'𝐭 𝐭𝐡𝐞𝐲 𝐬𝐮𝐟𝐟𝐞𝐫 𝐟𝐫𝐨𝐦 𝐩𝐫𝐞𝐜𝐢𝐬𝐢𝐨𝐧 𝐢𝐬𝐬𝐮𝐞𝐬?
Surprisingly, 𝐧𝐨. Overall action error is about the same for all four heads. On fine motions (expert moves under 5 mm per step), discrete is 𝐞𝐯𝐞𝐧 𝐦𝐨𝐫𝐞 𝐩𝐫𝐞𝐜𝐢𝐬𝐞: 3.2 vs. 5.4 mm single-step, 3.7 vs. 4.6 mm with flow.
So binning is 𝐧𝐨𝐭 𝐚 𝐩𝐫𝐞𝐜𝐢𝐬𝐢𝐨𝐧 𝐛𝐨𝐭𝐭𝐥𝐞𝐧𝐞𝐜𝐤. Then what makes the difference?
ALT Discrete is even more precise on fine motions. Violin plots of end-effector error per step (log scale) for continuous regression, continuous flow, discrete regression, discrete flow, grouped by expert motion per step. Fine motion (0 to 5 mm), medians: 5.4, 4.6, 3.2, 3.7 mm. Static chunks: all under 1 mm, discrete at a 0.4 mm quantization floor. Larger motions: all heads similar.
Peter Chen retweeted
Your navigation model already found the goal. It just doesn't know when to stop.
Introducing StopWatch, a navigation agent that grounds the goal explicitly before every action. It sets a new state of the art on six object-goal navigation benchmarks.
Watch the flower vase episode below. Both agents reach the goal. StopWatch stops there. LightNav-0 turns away and keeps going.
Code, models and data are open source 👉 stopwatch-nav.github.io/
Work done at Amazon FAR w/ @yunshengtian @QiuYuhengQiu @pabbeel @peterxichen @GuanyaShi @simonforrest
🧵 [1/n]
Peter Chen retweeted
Happy to announce the full ABC release! We’re also excited that ABC was accepted to CoRL 2026!
Check out our website for code, 400+ hours of sim data on 24 tasks, and 5,850 labeled policy-evaluation episodes.
@arthurallshire @Cinnabar233 @ritvik_singh9 @redstone_hong @davidrmcall
Peter Chen retweeted
6/9 ↳Possibly because I2T loss reflects overall joint training progress. And since every model predicts within the same text token space, I2T loss stays comparable across tokenizers, whereas T2I loss does not.
I2T loss remains informative after SFT, correlating with both image generation quality and general VQA performance.
Peter Chen retweeted
1/9 Image Tokenizers define Visual Languages. 🎨🔤
Image tokenizers do more than compress and reconstruct pixels. In unified multimodal models, they define the visual language a single model must learn, align with text, and use for both understanding and generation — and the best compressor may not produce the easiest language to learn.
We study image tokenizers through the lens of unified AR multimodal training. 🧵
Peter Chen retweeted
Personal Update: I'll be splitting my responsibilities between CMU and Amazon FAR over the next year. I'm convinced we need a unified understanding of how the axes of scale, data, optimization and control interact for robot learning, and am excited for my lab + FAR to push towards uniting the "science of robot learning" with the "science of deep learning" at scale.
Please reach out if you are looking for a full-time position building powerful multimodal AI models for deployment on hardware, and if you are intern looking to tinker and do good science on architectures, optimizers, data, tokenization, VLAs, WAMs and more!
Peter Chen retweeted
WAMs are popular because of their promise of better generalization. Is that true? We started playing with Video-Action-Model (VAMs) and realized a gap: video model backbones can compositionally generalize but VAMs often do not.
We coin this the Video-Action-Generalization (VAG) gap and present a study on how to explain and improve it. More details: umishra.me/temporal-ratio/
🧵 below
Peter Chen retweeted
🤖 How can we scale up humanoid robot learning?
Introducing 🌟VLK🌟: generating large-scale synthetic data with paired egocentric observations, text, and full-body G1 kinematics for learning humanoid loco-manipulation. No teleoperation needed!
Website: vision-language-kinematics.g…
Peter Chen retweeted
Humanoids excel in free space but struggle with real-world contact. Meet SceneBot 🤖 the first unified RL framework for ALL free-space locomotion, terrain traversal and object interaction!
By conditioning on per-link contact labels, it masters complex, interaction-rich tasks like carrying a box upstairs. Code & data open-sourcing soon! 📦 🪜
Paper: arxiv.org/abs/2606.27581
Website: ericcsr.github.io/scenebot/
Peter Chen retweeted
A huge team effort! Fully open data, infrastructure and eval for the community to build on
Introducing ABC: open data, training, and infrastructure for robotics.
We release the largest teleop dataset to date, and extensively investigate design decisions, pretraining, and post-training techniques.
@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall
Peter Chen retweeted
Check out our paper, more videos, and our more than 3.5K hour dataset here
abc.bot
Peter Chen retweeted
🪜 What if humanoids could climb ladders and work on them straight out of simulation?
Meet LadderMan: a perceptive system for zero-shot sim-to-real ladder climbing and on-ladder manipulation.
Watch the humanoid climb, stabilize, and manipulate—all in one system. 🤖👇
Peter Chen retweeted
World models are heavy. They don't need to be.
Each frame is encoded as 1024 spatial tokens. What if it were just 1?
In our #CVPR2026 Highlight from Amazon FAR, we compress frames into "delta" tokens for efficient generative world modeling.
Paper, code & models below ↓
(1/7)
ALT Outline of DeltaWorld. Unlike large existing generative world models that require many forward passes and represent each frame with many spatial tokens, our small DeltaWorld generates multiple futures in a single forward pass by using a single delta token to encode the difference between consecutive frames.
Peter Chen retweeted
🤖We are hiring multiple Summer'26 Research Interns at @amazon FAR to work on open-world navigation and robot foundation models, especially in neural rendering & simulation/predictive world models/reasoning & agency/real-world evaluation/long-term autonomy!
Peter Chen retweeted
Can humanoids perform agile, autonomous, long-horizon parkour—based on what they see in the world?
We present 𝗣𝗲𝗿𝗰𝗲𝗽𝘁𝗶𝘃𝗲 𝗛𝘂𝗺𝗮𝗻𝗼𝗶𝗱 𝗣𝗮𝗿𝗸𝗼𝘂𝗿 (𝗣𝗛𝗣): a framework that chains dynamic human skills using onboard depth perception for long-horizon traversal.
1/6
Peter Chen retweeted
New project! Flow Policy Gradients for Robot Control
tldr; a simple online RL recipe for training and fine-tuning flow policies for robots
co-led w/ @redstone_hong: hongsukchoi.github.io/fpo-co…
Peter Chen retweeted
We introduce a framework that enables robust, long-horizon bi-directional locomotion over complex terrains, by effectively leveraging a single policy with dual-depth camera streams, without the need for LiDAR-based elevation maps.
Check out @Yuanhang__Zhang's tweet for architecture and implementation details, and the website for full, uncut rollouts.
Robust humanoid perceptive locomotion is still underexplored. Especially when different cameras see different terrains, paths get narrow, and payloads disturb balance...
Introduce RPL, tackling this with one unified policy:
• Challenging terrains (slopes, stairs and stepping stones);
• Multiple directions;
• Payloads;
Trained in sim. Validated long-horizon in the real world.
Watch the robot walk it all🦿
Details below👇
Peter Chen retweeted
Robust humanoid perceptive locomotion is still underexplored. Especially when different cameras see different terrains, paths get narrow, and payloads disturb balance...
Introduce RPL, tackling this with one unified policy:
• Challenging terrains (slopes, stairs and stepping stones);
• Multiple directions;
• Payloads;
Trained in sim. Validated long-horizon in the real world.
Watch the robot walk it all🦿
Details below👇
