Hanming Ye (@DozenDucc) profile picture

@theWaddleLabs | math @ harvard | rsi '23 | https://teapaw.org/t.co/j1Od9XUBZT

Cambridge, MA
Joined July 2024
Photo by Hanming Ye (@DozenDucc)Photo by Hanming Ye (@DozenDucc)Photo by Hanming Ye (@DozenDucc)Photo by Hanming Ye (@DozenDucc)Photo by Hanming Ye (@DozenDucc)
Introducing Waddle Labs: Claude Code for robots. Connect our API to your robot and enter a prompt, then our agents write code to achieve the task in 20 minutes. @yiding_song @theWaddleLabs
204
229
136
2,464
711,490
Will be giving a talk on research that @yiding_song and I have been working on!
YC Paper Club is coming to Cambridge, MA! YCPC brings together people from academia and industry to discuss the latest research. Our inaugural Cambridge event on November 18 will explore the state of the art in robot learning and control. If you’re a student or faculty at a local university, or a researcher in industry interested in staying up to date with the latest literature, request a spot: events.ycombinator.com/yc-pa…
3
18
789
Hanming Ye retweeted
Two YC companies led the way in igniting a revolution in how the robotics community sees robot control via LLMs. Tune in to hear more about the last five years of research that gave them the conviction that we were heading this way and some predictions on what comes next.
One of the biggest surprises in AI over the last few years has been how well coding agents generalize beyond software. In a recent essay, MIT professor Philip Isola argued that we may be entering the era of robot-use agents: general-purpose models that can control different robots, write policies, and learn new physical tasks with little or no robot-specific training. In this episode of Decoded, we're joined by the founders of @theWaddleLabs and @Robocurve, two of the startups whose work helped drive this realization. They're working at the frontier of using general-purpose models to control robots, and their recent demos helped inspire the growing conversation around robot-use agents. Together, we dig into the research behind that idea, from code-as-policies and vision-language-action models to the harnesses and evals needed to make these systems work in the real world. 00:00 — What Are Robot-Use Agents? 02:11 — From Vision-Language-Action Models to General LLMs 04:07 — The Bitter Lesson for Robotics 07:04 — How Coding Agents Learned to Control Robots 10:41 — How Robots Learn From Experience 14:22 — The Harness as a Form of Robot Intelligence 16:14 — Watching Astra Control a Robot 20:55 — Why General Models May Win in Robotics 26:05 — How Close Are General-Purpose Robots?
13
11
3
97
22,172
We talk about robot-use agents with @agupta and @FrancoisChauba1! Some thoughts: 1) robot-use agents should be able to control robots with either ode or VLAs or direct actions depending what's optimal. 2) more evals are needed to compare robot-use agents! I often find myself asking @chooi_jeq to benchmark new models.
One of the biggest surprises in AI over the last few years has been how well coding agents generalize beyond software. In a recent essay, MIT professor Philip Isola argued that we may be entering the era of robot-use agents: general-purpose models that can control different robots, write policies, and learn new physical tasks with little or no robot-specific training. In this episode of Decoded, we're joined by the founders of @theWaddleLabs and @Robocurve, two of the startups whose work helped drive this realization. They're working at the frontier of using general-purpose models to control robots, and their recent demos helped inspire the growing conversation around robot-use agents. Together, we dig into the research behind that idea, from code-as-policies and vision-language-action models to the harnesses and evals needed to make these systems work in the real world. 00:00 — What Are Robot-Use Agents? 02:11 — From Vision-Language-Action Models to General LLMs 04:07 — The Bitter Lesson for Robotics 07:04 — How Coding Agents Learned to Control Robots 10:41 — How Robots Learn From Experience 14:22 — The Harness as a Form of Robot Intelligence 16:14 — Watching Astra Control a Robot 20:55 — Why General Models May Win in Robotics 26:05 — How Close Are General-Purpose Robots?
10
10
64
15,245
"Robot-use agents" has been defined now. @yiding_song showed me Prof Isola's platonic representation paper last summer. All data—code, computer-use, VQA—should help models learn representations of the physical world. This motivated us to experiment with LLMs for robot-use. It's likely that the best robot models will be generalist agents.
Recently, there have been a lot of impressive demos of AI agents, like Claude, controlling robots. I wrote a short blog post with my thoughts on the advent of these "robot-use agents." web.mit.edu/phillipi/www/wri… I think it's an important change in the trajectory of robotics!
6
1
32
4,579
I agree with Phil's take here: The progress of LLMs on controlling robots is quite interesting. Intuitively, this makes sense: controlling a robot is not so different from computer use, and an agent that is good at computer use is probably also good at controlling a robot and vice versa - credit to my student @RyuHyunwoooo to pointing that out to me originally!
Recently, there have been a lot of impressive demos of AI agents, like Claude, controlling robots. I wrote a short blog post with my thoughts on the advent of these "robot-use agents." web.mit.edu/phillipi/www/wri… I think it's an important change in the trajectory of robotics!
8
21
3
191
43,149
1. Robot data isn't the only way models can learn to control robots. Coding, math, computer-use all contain information about the physical world. I bet that generalist LLMs will work better than robot-specific models for this reason 2. The output format (eg. joint angles, code-as-policy, tool calls as in Gemini ER) is up to debate. But why not just let an LLM choose?
GPT-6 Astra scored 95% on a robot control task, up from Fable 5.1's 40%, with 6.2x fewer output tokens at 2.3x lower cost. 🧵
5
6
1
123
12,736
Hanming Ye retweeted
Introducing @robocurve, a Public Benefit Corporation to measure and report frontier robotics capabilities. We build real-world evaluations for robots and publish results as a neutral third party.
39
47
16
460
75,523
This is really interesting! Very surprising that LLMs can control robots well
We took part in the research preview of MHS from @AnthropicAI Here is Claude Code running a real SO-ARM101. Nothing in this was trained - no policy, no teleoperation, no demonstrations. The agent measured the workspace itself and wrote the motion. The calibration is the interesting part. No checkerboard, no camera intrinsics: the arm is its own ruler. Torque drops, a human rests the closed gripper on 16 dots the software draws in the camera view, and the robot reads back where they are. 4.1 mm position accuracy, 3.0 mm placement. Best run so far: 12 bricks placed with all four colour groups formed. A full hands-off run, start to finish, is what's next. Research preview today, open source coming soon.
5
32
6,220
I work on code as policy but never expected raw joint control by LLMs to be capable of this. Some criticism around speed, but there can be so much future work on post training or distilling smaller LLMs just for robots.
Opus 5 can do this with zero demonstrations despite not being trained for robots. On the flip side Opus took 7 minutes while GEN-1.5 took 7 seconds. 🧵
4
1
37
6,550
Hanming Ye retweeted
Introducing our tech roadmap at @cosmic_robotics to start autonomous city construction on Mars in the mid-2030s. Our team includes senior talent from SpaceX, Tesla, Google, Amazon, Zoox, Nuro, and Autodesk. We're already deploying robots on large-scale construction sites. Link to roadmap in the thread below:
154
244
102
2,031
696,030
Hanming Ye retweeted
Sourccey is coming this August We’ve spent the past year building an affordable, capable, and reliable mobile manipulator designed for robotics developers. • @LeRobot compatible • Open-source hardware and software • Laundry folding AI model, training data, and autonomous navigation out of the box • Built to help developers advance physical AI from day one Over the next few weeks we’ll be unveiling the engineering advances, open-source resources, and vision behind Sourccey. America needs accessible developer robots to lead the physical AI revolution.
Meet Sourccey! Your Personal Robot Coming August 2026
7
12
1
56
9,796
Hanming Ye retweeted
Robotics felt different this month. Google DeepMind and Anthropic made the same question hard to ignore: can a reasoning model decide what a whole body/robot should do next? Gemini Robotics 2 put an embodied reasoning model above a VLA for whole-body control. Anthropic tested general-purpose reasoning models at several levels of control. - deepmind.google/blog/gemini-… - anthropic.com/research/claud… The bad news is clear. General-purpose reasoning models still cannot control a humanoid on their own. They need a pretrained policy underneath. Still, I think this might actually work. A strong model (w/ excellent multimodal and reasoning capabilities) could really do it one day. HumanCLAW is our attempt to get an early glimpse of that future.
🦾HumanCLAW: Can Vision-Language Models Act Through a Body? A VLM can spot the sofa in a second. But can it get a body there and sit down? We call it Action Intelligence, and our work decouples it from motor control so it can finally be measured. Today’s best models turn out to be surprisingly bad at it. 🔻human-claw.github.io/
10
31
1
230
44,644
Hanming Ye retweeted
We evaluated DeepMind's new Gemini Robotics-ER 2 model. It threw our robot arms off the table. Here's what happened 🧵
unavailable
22
26
10
286
133,083
Hanming Ye retweeted
Every AI company should own its data, its models, and the research lab that keeps making them better. Today, @thomasboser and I are launching @hiloopai to make that possible. The frontier advantage isn’t access to a model. It’s the research organization continuously improving it. Your product already generates the raw material for better intelligence: proprietary data, production feedback, evaluations, and domain expertise. Very few teams have the research capacity to turn those assets into better models. Hiloop builds and operates that capability with you. Bring us the model your product depends on, the data that makes it different, and an evaluation that defines success. We reproduce your baseline and run an autonomous research campaign against it. Agents pursue competing hypotheses in parallel, build on previous results, and promote only improvements that survive verification. Our researchers validate the winners and bring them into production. The work can run hosted or inside your environment. You retain control of your data, and the resulting models, evaluations, and research artifacts are yours. We’re starting with model training, post-training, and inference optimization, where progress is measurable and the value is immediate. But this isn’t a one-off model improvement. It’s a persistent research capability that begins each campaign with everything learned from the last. Over time, the lab accumulates research memory and improves its own tools, evaluations, and agents. The process used to create better intelligence gets better itself. That’s infrastructure for recursive self-improvement. If your product depends on a model and an important metric has stopped moving, bring us the model you can’t make better. We're much better at research than making videos! Reach out to us: [email protected]
37
29
4
171
146,013
Introducing Rook by @Hop_Aero (YC S26): an autonomous hypersonic cargo rocket that delivers 550 lbs up to 450 miles in ~15 minutes. It launches from a standard 40-foot shipping container and lands on unprepared surfaces—no runways or fixed infrastructure required.
303
268
256
3,872
1,220,122
Hanming Ye retweeted
Putting LLM brain on robots -> 4x SOTA with no extra training. I’ve been very surprised by how well this works. The time for agents running on robots is coming soon
Robot policies can move but can't think. LLMs can think but can't move. So we connected them. Real robot: 16.7% → 97.3% Sim (LIBERO-PRO): 12.8% → 53.3%
12
27
272
49,137
Ankit and YC have made us 1000% more ambitious. They've pushed us to turn our dorm room research project into the tech we have today.
one of the companies i'm working with this batch is launching today with probably the most mind-blowing demo I've seen this year. stay tuned
2
1
31
7,196