🤖 AI/Robotics PhD @ UPenn GRASP Lab 💡 Previously @ TikTok AI Lab & Microsoft Research Asia
United States
Joined October 2019
- Tweets24
- Following128
- Followers14
- Likes45
Wei Wang - 汪炜 retweeted
One thing I've learned working on dexterous manipulation is how much a few millimeters can matter. A tiny change in finger position can lead to completely different contact dynamics.
Egocentric video tells us a lot about how humans interact with the world. But there's so much happening underneath that is hard to capture from pixels alone.
As the field move toward a million hours of human video, I've been thinking about what we should scale next.
I think a big part of the answer is richer, sensorized human data, with the sensing and robot hardware designed together from the start.
S1 is our first step in that direction. It's been really fun building it, and even more exciting to think about how much further we can take this.
A copilot flies the plane for you. A coach teaches you to fly on your own. What does it take to build an AI coach?
We formalize AI coaching as a non-cooperative game where the coach targets the learner's independent competence, then use model-free RL to train coaching policies.
8/9 🔍
We also inspect how L2C adapts assistance based on the estimated learner skill and evolving physical context.
The blending magnitude reveals two patterns: across learners, novices get more help; within a trajectory, assistance spikes near the gates, the make-or-break states for passing or colliding.
9/9 🔗
Project page: ai-coaching-drone-racing.git…
Paper: arxiv.org/abs/2606.25337
Joint work with Haimin Hu (@HaiminHu), Enlin Gu, Antonio Loquercio (@antoniloq), and Rahul Mangharam (@rahulmangharam).
Wei Wang - 汪炜 retweeted
“AI agents will outperform humans at almost all jobs by 2026–2027.” - The forecast is everywhere.
So we built the exam to test that claim, on real labor-market aligned work. On the hardest tier, top agents pass 2.6%.
Meet Agents' Last Exam (ALE), a rolling benchmark measuring whether agents can actually do real jobs. 🧵👇
Wei Wang - 汪炜 retweeted
Agreed: Latency is now low enough to support robot inference in the cloud, and edge is where embodiment transforms and safety checks should be performed: arxiv.org/abs/2205.09778
Wei Wang - 汪炜 retweeted
GEN-1 plays the 🐚 shell game, trained on just 1 hr of robot data. It also generalizes to unseen objects, like @BerkayAntmen 's car keys.
Physical AI models should be capable of benchmark tasks like this one. It's interesting for the all the reasons @RhodaAI calls out -- requires visual memory, and the model must track the cups from the very start, at high frame rates.
Interestingly, GEN-1 appears to exhibit a degree of "active perception." It's subtle; the hands can sometimes appear to "follow" the cups, using its own movements to help attend to where it thinks the object should be.
Read more about GEN-1 in our blog post in the comments below ↓
So making a hand-like design is not only for easier data collection but also for natural force feedback. 🤔
For Sunday Robotics, VR teleoperation is a dead end.
The reason? Your hand is numb.
@tonyzzhao explains the physics: In VR, you miss "force closure." You accidentally apply infinite force to soft objects because you get no feedback.
Sunday's solution? Ditch the headset for a glove. 👇
👍
What if robots could dream inside a video generative model? Introducing DreamGen, a new engine that scales up robot learning not with fleets of human operators, but with digital dreams in pixels. DreamGen produces massive volumes of neural trajectories - photorealistic robot videos paired with motor action labels - and unlocks strong generalization to new nouns, verbs, and environments. Whether you’re a humanoid (GR1), an industrial arm (Franka), or a cute little robot (HuggingFace SO-100), DreamGen enables you to dream.
Video generation models like Sora & Veo are neural physics engines. By compressing billions of internet videos, they learn a multiverse of plausible futures, i.e. superpositions of how the world could unfold from any initial image frame. DreamGen taps into this power with a simple 4-step recipe:
1. Fine-tune a SOTA video model on your target robot;
2. Prompt the model with diverse language prompts to simulate parallel worlds: how your robot would have acted in new scenarios. Filter out the bad dreams (ha!) that don’t follow instructions;
3. Recover pseudo-actions using inverse dynamics or latent action models;
4. Train robot foundation models on the massively augmented dataset of neural trajectories.
That’s it. Just more data, and plain old supervised learning. Simple, right?
What’s remarkable is how far this goes. Starting with just a single-task dataset of pick-and-place, our humanoid robot learns 22 new behaviors, such as pouring, folding, scooping, ironing, and hammering, despite never seeing those verbs before. Better yet, we can take the robot out of the lab and drop it into the NVIDIA HQ Cafe, and let DreamGen work its magic. We show true zero-to-one generalization: from 0% success to over 43% for novel verbs, and 0 -> 28% in unseen environments.
Compared to a traditional graphics engine, DreamGen doesn’t care if the scene involves deformable objects, fluids, translucent materials, contact-rich interactions, or crazy lighting. Good luck engineering those by hand. For DreamGen, every world is just a forward pass through a diffusion neural net. No matter how complex the dream is, it takes constant compute time to roll out.
Read our blog and paper today! We plan to fully open-source the entire pipeline in the next few weeks. Links in thread:
In 2022, I spent 12,418 hours training 12,319 models on @wandb! My #longestrun, copper-silence-1, trained for 72 hours. Here's its collector card, with an image generated by DALL·E 2 from @OpenAI: wandb.ai/year-wrapped-2022?u…