Yo, I’m a scientist! NYC➡️CS PhD➡️MPI➡️Body Labs (acq. by AMZN)➡️BK dad➡️🤖🤖🤖@theworldlabs
Brooklyn, New York
Joined April 2008
- Tweets7K
- Following1.5K
- Followers2.9K
- Likes11.5K
Pinned Tweet
Job update! I'm now part of @theworldlabs team helping train real robots in simulated environments.
Since joining Scenix last year, I've had a blast getting back to my roots in computer vision. Along the way, I've learned A TON about the substantial challenges you face when training policies that run on real hardware in realtime.
Having the opportunity to continue this work at World Labs gives our team the opportunity to move faster and think bigger.
In short, we're pumped!
@YunzhuLiYZ, @fast_sploosh, and @xhsonny have built a remarkable team that is training and evaluating robots in high-fidelity simulation – proven not only in lab demos, but in live deployments on real hardware.
Last month, we wrote that the boundaries between rendering, simulation, and planning are beginning to blur. Bringing our world models together with SceniX's simulation and robotics expertise is an important step in our quest for spatial intelligence.
Welcome to the team. Read the full announcement:
worldlabs.ai/blog/scenix
This new Tesla Roadster is basically just an excuse to get out in front of designing a vehicle that can drive around on the moon, right?
You’d need different tires/suspension but in low gravity all the thrust craziness actually makes a lot of sense.
I was not expecting to go through two acquisitions in 2026, but here we are!
12 months ago I was looking to get back to my roots in 3D computer vision after parting ways with an accounting startup I cofounded.
Since then I've had the opportunity to deep dive into robotics and land at a world class AI company.
I couldn't be more grateful for how things worked out. Huge shout out to all various founders of Scenix and World Labs for inviting me along for the ride!
We are excited to announce that World Labs is joining @AMD.
The research and technical breakthroughs we have achieved since our founding in 2024 have given us a clear vision for AI’s potential to solve problems in the spatial and physical world.
Accelerating the future of spatial and physical intelligence requires scaling our efforts, scaling our reach, and getting closer to the hardware.
Second, evaluating policies in the real world is typically manual and time consuming. With Astra, you have a few appealing new options.
First, even if you are simply recording a policy rollout (or you ask Astra to), you can then have Astra tally up and categorize the successes and failures.
This is particularly appealing if (as suggested above) you've also used Astra to construct a simulated environment of the scene. Why? Because in this case Astra can be prompted to recreate the failure in sim, and then correct the failed behavior.
Even though this is potentially time consuming, it's offline and gives you data you can feed back into the policy.
Third, if you are trying to use Astra to control a real world robot, and frustrated by its speed good news, you can potentially ask Astra to do things faster.
At present I don't expect this to work for complex tasks, but for simple pick and place tasks, there's no need for Astra to figure out everything on the fly.
Instead Astra can create a collection of action primitives that it can then control based on the state of the scene. This gives you a more efficient way to generate real world data offline for training either end-to-end policies or a collection of learned actions.
An important point from @DJiafei for anyone thinking that Astra provides a practical real-time solution for controlling a robot in production.
In practice if you give Astra control of a robot, it is 1) very slow and 2) pricy!
That said, there are still multiple ways to leverage Astra to help train and evaluate robot policies.
People watching GPT-6 Astra demos on X may not realise how slow it actually is in real time. Here’s a rollout of GPT-6 Astra controlling the MolmoAct2 YAMs to complete this task in real-time.
There is potential, but i don’t think it’s practical for real-world robotics deployment yet.
I applaud your patience if you make it through the whole video.
First, as discussed somewhat extensively in this fantastic post by @Ken_Goldberg is to use Astra offline to reconstruct scenes/tasks in simulation.
Remarkably, Astra can not just align geometry, but also tune physics parameters.
From there, Astra is able to generate novel scene configurations and use the sim scenes full state information (instead of relying on images) to script out programmatic solutions that succeed with high reliability.
This, in turn, gives you a way to generate training data that can used to train/tune an online policy.
xitter.cf/Ken_Goldberg/status/21…
One of the biggest bottlenecks to training robot policies in simulation is getting the simulation right.
At least for visual appearance, I'm confident that bottleneck is about to go away.
I suspect one reason OpenAI is putting a lot of resources into 3D/4D/robotics is because these problems are both challenging and relatively easy to evaluate.
The sweet spot for frontier labs right now is sufficiently challenging-but-useful problems where AI can chug away for minutes-to-hours (and soon days), and then we can easily understand "is this correct"
Evaluating whether a 3D or 4D scene matches collection of reference images is super quick, whereas solving the problem requires all sorts of learning.
Arguably this is even more so with robotics. Getting a robot to say, make a chair, is still fully out of reach, but I'll be able to tell if they did a good job in a few seconds.
This is a big part of what drew me to robotics. After years of looking into challenging to evaluate, highly domain-specific task (e.g. filing a tax return), it's been delightful to get back to tasks that I and others can evaluate at a glance.
Eric Rachlin retweeted
We're hiring in robot learning at @theworldlabs! Join me, @drfeifei, and the team to define and scale the next generation of world models for robot learning!
Atlas for Robotics: worldlabs.ai/blog/atlas#robo…
Real-to-Sim-to-Real: worldlabs.ai/blog/real-to-si…
Apply: jobs.ashbyhq.com/worldlabs/8…
The first Millennium Problem was solved by a reclusive genius working largely in isolation for the better part of a decade.
The second appears to have been solved by 10,000 highly coordinated AI agents working together for under a week.
Both are a testament to human achievement and the universe’s seemingly innate commitment towards understanding.
Goosebumps.
We’re sharing a solution to the Navier-Stokes Millennium Prize Problem, one of the deepest problems at the frontier of mathematics.
The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra.
The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navier-Stokes equations can break down. It has remained unresolved for roughly 90 years.
Took this delightful photo today at the Montreal Insectarium.
Seems noteworthy I wouldn’t be nearly as pleased with it had I generated it with AI.
For context, here’s someone else’s photo of the room the beetles are in
share.google/ii5FzOt8T3TupIg…
So crazy! Just used Atlas to generate this 3D reconstruction of my childhood home from only 3 images I grabbed from google street view.
Details are spot on, looks just like how I remember!
⬇️workflow below
Step 3:
Take one of the novel camera views (in this case one toward the end of the trajectory) and modify it in your AI image editor of choice.
From there, you can feed that image back into Atlas so that the edited image is baked into the reconstruction.
