
@theWaddleLabs | math @ harvard | rsi '23 | https://teapaw.org/t.co/j1Od9XUBZT
Cambridge, MA
Joined July 2024
- Tweets144
- Following154
- Followers2.3K
- Likes603
Pinned Tweet
Introducing Waddle Labs: Claude Code for robots.
Connect our API to your robot and enter a prompt, then our agents write code to achieve the task in 20 minutes.
@yiding_song @theWaddleLabs
Will be giving a talk on research that @yiding_song and I have been working on!
YC Paper Club is coming to Cambridge, MA!
YCPC brings together people from academia and industry to discuss the latest research. Our inaugural Cambridge event on November 18 will explore the state of the art in robot learning and control.
If you’re a student or faculty at a local university, or a researcher in industry interested in staying up to date with the latest literature, request a spot: events.ycombinator.com/yc-pa…
Hanming Ye retweeted
Two YC companies led the way in igniting a revolution in how the robotics community sees robot control via LLMs.
Tune in to hear more about the last five years of research that gave them the conviction that we were heading this way and some predictions on what comes next.
One of the biggest surprises in AI over the last few years has been how well coding agents generalize beyond software.
In a recent essay, MIT professor Philip Isola argued that we may be entering the era of robot-use agents: general-purpose models that can control different robots, write policies, and learn new physical tasks with little or no robot-specific training.
In this episode of Decoded, we're joined by the founders of @theWaddleLabs and @Robocurve, two of the startups whose work helped drive this realization. They're working at the frontier of using general-purpose models to control robots, and their recent demos helped inspire the growing conversation around robot-use agents.
Together, we dig into the research behind that idea, from code-as-policies and vision-language-action models to the harnesses and evals needed to make these systems work in the real world.
00:00 — What Are Robot-Use Agents?
02:11 — From Vision-Language-Action Models to General LLMs
04:07 — The Bitter Lesson for Robotics
07:04 — How Coding Agents Learned to Control Robots
10:41 — How Robots Learn From Experience
14:22 — The Harness as a Form of Robot Intelligence
16:14 — Watching Astra Control a Robot
20:55 — Why General Models May Win in Robotics
26:05 — How Close Are General-Purpose Robots?
We talk about robot-use agents with @agupta and @FrancoisChauba1! Some thoughts:
1) robot-use agents should be able to control robots with either ode or VLAs or direct actions depending what's optimal.
2) more evals are needed to compare robot-use agents! I often find myself asking @chooi_jeq to benchmark new models.
One of the biggest surprises in AI over the last few years has been how well coding agents generalize beyond software.
In a recent essay, MIT professor Philip Isola argued that we may be entering the era of robot-use agents: general-purpose models that can control different robots, write policies, and learn new physical tasks with little or no robot-specific training.
In this episode of Decoded, we're joined by the founders of @theWaddleLabs and @Robocurve, two of the startups whose work helped drive this realization. They're working at the frontier of using general-purpose models to control robots, and their recent demos helped inspire the growing conversation around robot-use agents.
Together, we dig into the research behind that idea, from code-as-policies and vision-language-action models to the harnesses and evals needed to make these systems work in the real world.
00:00 — What Are Robot-Use Agents?
02:11 — From Vision-Language-Action Models to General LLMs
04:07 — The Bitter Lesson for Robotics
07:04 — How Coding Agents Learned to Control Robots
10:41 — How Robots Learn From Experience
14:22 — The Harness as a Form of Robot Intelligence
16:14 — Watching Astra Control a Robot
20:55 — Why General Models May Win in Robotics
26:05 — How Close Are General-Purpose Robots?
"Robot-use agents" has been defined now.
@yiding_song showed me Prof Isola's platonic representation paper last summer. All data—code, computer-use, VQA—should help models learn representations of the physical world. This motivated us to experiment with LLMs for robot-use.
It's likely that the best robot models will be generalist agents.
Recently, there have been a lot of impressive demos of AI agents, like Claude, controlling robots.
I wrote a short blog post with my thoughts on the advent of these "robot-use agents."
web.mit.edu/phillipi/www/wri…
I think it's an important change in the trajectory of robotics!
Hanming Ye retweeted
I agree with Phil's take here: The progress of LLMs on controlling robots is quite interesting. Intuitively, this makes sense: controlling a robot is not so different from computer use, and an agent that is good at computer use is probably also good at controlling a robot and vice versa - credit to my student @RyuHyunwoooo to pointing that out to me originally!
Recently, there have been a lot of impressive demos of AI agents, like Claude, controlling robots.
I wrote a short blog post with my thoughts on the advent of these "robot-use agents."
web.mit.edu/phillipi/www/wri…
I think it's an important change in the trajectory of robotics!
1. Robot data isn't the only way models can learn to control robots. Coding, math, computer-use all contain information about the physical world.
I bet that generalist LLMs will work better than robot-specific models for this reason
2. The output format (eg. joint angles, code-as-policy, tool calls as in Gemini ER) is up to debate. But why not just let an LLM choose?
Hanming Ye retweeted
Introducing @robocurve, a Public Benefit Corporation to measure and report frontier robotics capabilities.
We build real-world evaluations for robots and publish results as a neutral third party.
This is really interesting! Very surprising that LLMs can control robots well
We took part in the research preview of MHS from @AnthropicAI
Here is Claude Code running a real SO-ARM101. Nothing in this was trained - no policy, no teleoperation, no demonstrations. The agent measured the workspace itself and wrote the motion.
The calibration is the interesting part. No checkerboard, no camera intrinsics: the arm is its own ruler. Torque drops, a human rests the closed gripper on 16 dots the software draws in the camera view, and the robot reads back where they are. 4.1 mm position accuracy, 3.0 mm placement.
Best run so far: 12 bricks placed with all four colour groups formed. A full hands-off run, start to finish, is what's next.
Research preview today, open source coming soon.
I work on code as policy but never expected raw joint control by LLMs to be capable of this.
Some criticism around speed, but there can be so much future work on post training or distilling smaller LLMs just for robots.
Hanming Ye retweeted
Introducing our tech roadmap at @cosmic_robotics to start autonomous city construction on Mars in the mid-2030s.
Our team includes senior talent from SpaceX, Tesla, Google, Amazon, Zoox, Nuro, and Autodesk.
We're already deploying robots on large-scale construction sites.
Link to roadmap in the thread below:
Hanming Ye retweeted
Sourccey is coming this August
We’ve spent the past year building an affordable, capable, and reliable mobile manipulator designed for robotics developers.
• @LeRobot compatible
• Open-source hardware and software
• Laundry folding AI model, training data, and autonomous navigation out of the box
• Built to help developers advance physical AI from day one
Over the next few weeks we’ll be unveiling the engineering advances, open-source resources, and vision behind Sourccey.
America needs accessible developer robots to lead the physical AI revolution.
Robotics felt different this month. Google DeepMind and Anthropic made the same question hard to ignore: can a reasoning model decide what a whole body/robot should do next?
Gemini Robotics 2 put an embodied reasoning model above a VLA for whole-body control. Anthropic tested general-purpose reasoning models at several levels of control.
- deepmind.google/blog/gemini-…
- anthropic.com/research/claud…
The bad news is clear. General-purpose reasoning models still cannot control a humanoid on their own. They need a pretrained policy underneath.
Still, I think this might actually work. A strong model (w/ excellent multimodal and reasoning capabilities) could really do it one day.
HumanCLAW is our attempt to get an early glimpse of that future.
🦾HumanCLAW: Can Vision-Language Models Act Through a Body?
A VLM can spot the sofa in a second. But can it get a body there and sit down?
We call it Action Intelligence, and our work decouples it from motor control so it can finally be measured. Today’s best models turn out to be surprisingly bad at it. 🔻human-claw.github.io/
Hanming Ye retweeted
We evaluated DeepMind's new Gemini Robotics-ER 2 model. It threw our robot arms off the table.
Here's what happened 🧵
unavailable
Hanming Ye retweeted
Every AI company should own its data, its models, and the research lab that keeps making them better.
Today, @thomasboser and I are launching @hiloopai to make that possible.
The frontier advantage isn’t access to a model. It’s the research organization continuously improving it.
Your product already generates the raw material for better intelligence: proprietary data, production feedback, evaluations, and domain expertise. Very few teams have the research capacity to turn those assets into better models.
Hiloop builds and operates that capability with you.
Bring us the model your product depends on, the data that makes it different, and an evaluation that defines success. We reproduce your baseline and run an autonomous research campaign against it.
Agents pursue competing hypotheses in parallel, build on previous results, and promote only improvements that survive verification. Our researchers validate the winners and bring them into production.
The work can run hosted or inside your environment. You retain control of your data, and the resulting models, evaluations, and research artifacts are yours.
We’re starting with model training, post-training, and inference optimization, where progress is measurable and the value is immediate.
But this isn’t a one-off model improvement. It’s a persistent research capability that begins each campaign with everything learned from the last.
Over time, the lab accumulates research memory and improves its own tools, evaluations, and agents. The process used to create better intelligence gets better itself.
That’s infrastructure for recursive self-improvement.
If your product depends on a model and an important metric has stopped moving, bring us the model you can’t make better. We're much better at research than making videos!
Reach out to us: [email protected]
Hanming Ye retweeted
Introducing Rook by @Hop_Aero (YC S26): an autonomous hypersonic cargo rocket that delivers 550 lbs up to 450 miles in ~15 minutes.
It launches from a standard 40-foot shipping container and lands on unprepared surfaces—no runways or fixed infrastructure required.
Putting LLM brain on robots -> 4x SOTA with no extra training. I’ve been very surprised by how well this works. The time for agents running on robots is coming soon




















