Your friendly neighborhood PhD in Robotics @CMU. Dexterous Manipulation | Generative Control | Reinforcement Learning

Pittsburgh, PA
Joined November 2014
Interaction with the real world is the major bottleneck in robot learning. So what would robot RL look like if we didn’t need to limit compute per interaction? Our latest work, Off-Policy Generative Policy Optimization (OGPO, accepted to ICML26) embarks on answering this question (spoiler alert: when done correctly, it helps massively!). 🧵(1/N)
13
63
7
344
98,915
Sarvesh Patil retweeted
Hi all, I have a new result I'm excited to share. It turns out that *all existing implementations* of softmax and cross-entropy (that's JAX, PyTorch, textbooks, etc) have a subtle bug --- two bugs, actually. A simple rewrite fixes both, and eliminates "slingshots" when grokking:
134
769
141
6,318
743,976
Sarvesh Patil retweeted
A million hours of what? If we’re scaling human demonstrations for robot learning, what should each hour contain? We built S1, our skill capture glove, alongside our robot hand to record joint movements that map to robot targets, with touch and video on the same timeline.
39
66
30
328
49,869
Sarvesh Patil retweeted
This is one of my favorite projects I’ve been a part of this year. It’s ostensibly about robotics, but I feel that I learned something very fundamental about how AI models should represent all modalities that (superficially!) seem like they are best suited to being represented in continuous-valued (e.g. float) vectors. TL;DR discrete parametrizations are awesome, learn geometry from data, but aren’t constrained by it, and can perform better as a result. Exactly what the bitter lesson tells us we should look for.
Common belief: Robot policies need continuous action heads; discrete actions are 𝐥𝐞𝐬𝐬 𝐧𝐚𝐭𝐢𝐯𝐞 𝐚𝐧𝐝 𝐥𝐞𝐬𝐬 𝐩𝐫𝐞𝐜𝐢𝐬𝐞. ❗We find policies w/ the simplest discrete representations 𝐦𝐚𝐭𝐜𝐡 𝐨𝐫 𝐛𝐞𝐚𝐭 continuous ones—in sim & real, in single-step & diffusion/flow settings. Here's why 🧵
5
9
122
5,914
Sarvesh Patil retweeted
Common belief: Robot policies need continuous action heads; discrete actions are 𝐥𝐞𝐬𝐬 𝐧𝐚𝐭𝐢𝐯𝐞 𝐚𝐧𝐝 𝐥𝐞𝐬𝐬 𝐩𝐫𝐞𝐜𝐢𝐬𝐞. ❗We find policies w/ the simplest discrete representations 𝐦𝐚𝐭𝐜𝐡 𝐨𝐫 𝐛𝐞𝐚𝐭 continuous ones—in sim & real, in single-step & diffusion/flow settings. Here's why 🧵
9
47
11
275
27,718
Probably the cutest home robot I've seen so far! 🥹
Introducing Voxel, a droid butler made by Tuesday Labs.
2
11
363
Sarvesh Patil retweeted
I read more AI output than anything else, all in a language I already know. That felt like a waste, so now about 1 in 8 words is French.
30
15
18
449
50,486
Today, we introduce Priming Models, a new paradigm for training one-step generative models from scratch. No diffusion teacher. No flow map. No CFG. No JVP. We believe this work will challenge how we think about building one-step models. Joint work with @sitanch, @du_yilun, @parkwbin03, Bohyung Han. 🔥 1.19 FID one-step on ImageNet 256×256 while using substantially less compute, e.g., 12X less than Drifting-B, 6X less than Drifting-L 🎨 Scales to text-to-image generation (GenEval 0.82) 🤖 World modeling on RT-1 / Bridge-V2 Priming Model is built on one core message: [Reconstruction pretraining --> Distributional post-training].
38
130
24
1,462
106,120
Video model encodings with recurrent architectures (e.g. GDNs) for latent memory compression (like workspace models) seem to be scaling with the use of world models at scale in multi-task settings! The most impressive tasks for me are the seemingly innocuous ones, like wiping a table. Purely inferring actions from states that look more or less the same can render traditional MDP formulations unsuccessful. Using latent memory states to steer policy actions seems neat to convert the POMDP to an MDP!
What to remember in the past to act—spatial, temporal or motion details? We often assume what’s useful, but that’s hard to know in advance. Meet IMPRINT 🧠: retain rich memory first, then learn to use it for control. Compact latents keep inference fast as history grows. 🧵 1/10
1
9
794
Very cool work on distilling deliberative supervision from powerful but slow reasoning models into a "latent agentic harness" during training time! These embedding spaces can then be used at inference time for policy steering with minimal latency, e.g. as latent memory modules for learned robot policies to recall and/or react to environment!
There’s lots of buzz around agentic harnesses for robots, particularly for long horizon tasks that require complex reasoning and memory. But what will it really take to turn a reasoning agent into a reactive, reliable and low-latency robot policy? In our new paper, workspace models, we design a new memory architecture that acts as a latent harness for stronger reasoning models. Here’s why we think this might be the way forward: (1/9)
9
909
Sarvesh Patil retweeted
Surprisingly a robot can sometimes do highly-dynamic manipulation by just copying the demonstration 🤠. Without any learning, cracking a whip or lassoing a cleat can be done directly from human motion capture. And amazingly this process can be nearly automatically implemented by GPT-6 Astra Check out the full blog post with fun visuals: krishnasuresh.org/blog/2026/… Turn on sound for full effect 🔈 Quick side note: the whips shown in this post are called signal/stock whips. These whips are built to generate the loud cracking noise when the tip breaks the sound barrier and is a popular art form: youtube.com/watch?v=ietBtr7s….
33
41
39
408
113,081
Sarvesh Patil retweeted
1/ We introduce TrackEverything: a 3D point tracker that tracks all points across all frames of long videos (1000+ frames). Our key idea is to tie computation to unique 3D scene content rather than redundant 2D pixels in a video. trackeverything.github.io
25
183
14
1,662
139,260
Sarvesh Patil retweeted
I am teaching a class this Fall on Embodied AI with Web-Scale Video Data embodied-ai-with-videos.gith… Excited to dive into my favorite (and some not-so-favorite but popular) keywords in AI! Class lecture slides and materials will be all public for anyone to follow-along!
🤖 Made with AI
5
18
213
7,166
Sarvesh Patil retweeted
Robot WAMs typically predict future RGB latents trained for pixel reconstruction...but why stop there? We propose Flex-π, a WAM that jointly denoises multiple visual streams jointly, enabling the WAM to inherit strong 3D, object-centric, AND spatio-temporal priors for action generation. It's more demo-efficient than WAM and VLA baselines and can infer faster than π0.5. (1/3)
Are VLAs dead? No, and you don't have to choose between VLAs and WAMs. Introducing Flex-π: a multi-stream world-action model (WAM) that jointly predicts future RGB, 3D pointmaps and DINO semantics with actions in training, then deploys as a VLA, a full WAM, or anything in between — all from a single checkpoint. Website: flex-pi.github.io/
5
23
1
159
27,198
"""once someone launches an attack with an open-weights model whose guardrails they've removed, there will be many more defenders enabled by open-weights models to guard against it."""
Anthropic treats attacker access to open models as decisive while ignoring defender access to the same models. That is not a complete offense-defense analysis. Open weights also scale detection, hardening, attribution, and coordinated response. A coalition is the stable game-theoretic equilibrium which defends against such attacks, and is *enabled* by open models. Instead, they seem fixated on the single-step use case of an attacker removing guardrails from an open-weights model, but they seem to forget what naturally happens afterwards: once someone launches an attack with an open-weights model whose guardrails they've removed, there will be many more defenders enabled by open-weights models to guard against it.
8
1,440
Sarvesh Patil retweeted
Anthropic treats attacker access to open models as decisive while ignoring defender access to the same models. That is not a complete offense-defense analysis. Open weights also scale detection, hardening, attribution, and coordinated response. A coalition is the stable game-theoretic equilibrium which defends against such attacks, and is *enabled* by open models. Instead, they seem fixated on the single-step use case of an attacker removing guardrails from an open-weights model, but they seem to forget what naturally happens afterwards: once someone launches an attack with an open-weights model whose guardrails they've removed, there will be many more defenders enabled by open-weights models to guard against it.
There’s been a lot of speculation about where we stand on open-weights models. We’ve outlined our views in full here: anthropic.com/news/position-…
5
3
2
43
5,545
Sarvesh Patil retweeted
No one: Today's NeurIPS reviewers if it was 2015: "ResNet is incremental"
3
14
2
299
27,319
Introducing MotionForesight: A very simple approach re-purposing video models for forecasting future 3D scene flow given a short observed video context Trained with just 40k RGB human videos, it generalizes well to completely new home/office scenes! 1/n
6
33
4
119
16,676
Sarvesh Patil retweeted
I'm attending #RSS2026 for a couple papers/talks and as an RSS Pioneer! Please reach out to chat and/or come to the following: - Monday 9:40 AM: Spotlight pres on detecting side effects at the Rethinking Safety Workshop by @ryanlindeborg (sites.google.com/andrew.cmu.…) - Tues 4-5PM: Come visit my RSS Pioneers Poster! @RSSPioneers - Wed 3:15PM: Robometer Reward Fxn talk + poster after by @yigitkkorkmaz and @aliangdw (robometer.github.io/) - Thurs 3:15PM: TMRL pre-training for post-training talk + poster after by @matthewh6_ (weirdlabuw.github.io/tmrl/) - Friday 9:30AM: Giving an invited talk at SemRob Workshop (semrob.github.io/) - Friday 5:30PM: Invited talk at the Diffusion Workshop (rss2026-diffusion-robot-lear…) on TMRL
1
9
52
4,076
Sarvesh Patil retweeted
Presenting PTLD: an approach to learning tactile dexterous policies without ever simulating the tactile sensor. Tactile is essential for performing highly dexterous manipulation. However, collecting tactile observations reliably has been a fundamental bottleneck: (1) teleoperating a multi fingered hand for dynamic tasks is challenging, making sim-to-real imperative, (2) one can’t realistically simulate tactile today
6
46
10
188
38,775