Your friendly neighborhood PhD in Robotics @CMU. Dexterous Manipulation | Generative Control | Reinforcement Learning
Pittsburgh, PA
Joined November 2014
- Tweets911
- Following649
- Followers663
- Likes24.3K
Sarvesh Patil retweeted
Hi all, I have a new result I'm excited to share. It turns out that *all existing implementations* of softmax and cross-entropy (that's JAX, PyTorch, textbooks, etc) have a subtle bug --- two bugs, actually. A simple rewrite fixes both, and eliminates "slingshots" when grokking:
Sarvesh Patil retweeted
A million hours of what?
If we’re scaling human demonstrations for robot learning, what should each hour contain?
We built S1, our skill capture glove, alongside our robot hand to record joint movements that map to robot targets, with touch and video on the same timeline.
Sarvesh Patil retweeted
This is one of my favorite projects I’ve been a part of this year. It’s ostensibly about robotics, but I feel that I learned something very fundamental about how AI models should represent all modalities that (superficially!) seem like they are best suited to being represented in continuous-valued (e.g. float) vectors. TL;DR discrete parametrizations are awesome, learn geometry from data, but aren’t constrained by it, and can perform better as a result. Exactly what the bitter lesson tells us we should look for.
Sarvesh Patil retweeted
Common belief: Robot policies need continuous action heads; discrete actions are 𝐥𝐞𝐬𝐬 𝐧𝐚𝐭𝐢𝐯𝐞 𝐚𝐧𝐝 𝐥𝐞𝐬𝐬 𝐩𝐫𝐞𝐜𝐢𝐬𝐞.
❗We find policies w/ the simplest discrete representations 𝐦𝐚𝐭𝐜𝐡 𝐨𝐫 𝐛𝐞𝐚𝐭 continuous ones—in sim & real, in single-step & diffusion/flow settings.
Here's why 🧵
Sarvesh Patil retweeted
Today, we introduce Priming Models, a new paradigm for training one-step generative models from scratch.
No diffusion teacher. No flow map. No CFG. No JVP.
We believe this work will challenge how we think about building one-step models.
Joint work with @sitanch, @du_yilun, @parkwbin03, Bohyung Han.
🔥 1.19 FID one-step on ImageNet 256×256 while using substantially less compute, e.g., 12X less than Drifting-B, 6X less than Drifting-L
🎨 Scales to text-to-image generation (GenEval 0.82)
🤖 World modeling on RT-1 / Bridge-V2
Priming Model is built on one core message: [Reconstruction pretraining --> Distributional post-training].
Sarvesh Patil retweeted
Surprisingly a robot can sometimes do highly-dynamic manipulation by just copying the demonstration 🤠. Without any learning, cracking a whip or lassoing a cleat can be done directly from human motion capture. And amazingly this process can be nearly automatically implemented by GPT-6 Astra
Check out the full blog post with fun visuals: krishnasuresh.org/blog/2026/…
Turn on sound for full effect 🔈
Quick side note: the whips shown in this post are called signal/stock whips. These whips are built to generate the loud cracking noise when the tip breaks the sound barrier and is a popular art form: youtube.com/watch?v=ietBtr7s….
Sarvesh Patil retweeted
1/ We introduce TrackEverything: a 3D point tracker that tracks all points across all frames of long videos (1000+ frames).
Our key idea is to tie computation to unique 3D scene content rather than redundant 2D pixels in a video.
trackeverything.github.io
Sarvesh Patil retweeted
I am teaching a class this Fall on Embodied AI with Web-Scale Video Data
embodied-ai-with-videos.gith…
Excited to dive into my favorite (and some not-so-favorite but popular) keywords in AI!
Class lecture slides and materials will be all public for anyone to follow-along!
🤖 Made with AI
Sarvesh Patil retweeted
Robot WAMs typically predict future RGB latents trained for pixel reconstruction...but why stop there?
We propose Flex-π, a WAM that jointly denoises multiple visual streams jointly, enabling the WAM to inherit strong 3D, object-centric, AND spatio-temporal priors for action generation.
It's more demo-efficient than WAM and VLA baselines and can infer faster than π0.5.
(1/3)
Are VLAs dead?
No, and you don't have to choose between VLAs and WAMs.
Introducing Flex-π: a multi-stream world-action model (WAM) that jointly predicts future RGB, 3D pointmaps and DINO semantics with actions in training, then deploys as a VLA, a full WAM, or anything in between — all from a single checkpoint.
Website: flex-pi.github.io/
Sarvesh Patil retweeted
Anthropic treats attacker access to open models as decisive while ignoring defender access to the same models. That is not a complete offense-defense analysis. Open weights also scale detection, hardening, attribution, and coordinated response.
A coalition is the stable game-theoretic equilibrium which defends against such attacks, and is *enabled* by open models.
Instead, they seem fixated on the single-step use case of an attacker removing guardrails from an open-weights model, but they seem to forget what naturally happens afterwards: once someone launches an attack with an open-weights model whose guardrails they've removed, there will be many more defenders enabled by open-weights models to guard against it.
There’s been a lot of speculation about where we stand on open-weights models. We’ve outlined our views in full here: anthropic.com/news/position-…
Sarvesh Patil retweeted
Introducing MotionForesight:
A very simple approach re-purposing video models for forecasting future 3D scene flow given a short observed video context
Trained with just 40k RGB human videos, it generalizes well to completely new home/office scenes!
1/n
Sarvesh Patil retweeted
I'm attending #RSS2026 for a couple papers/talks and as an RSS Pioneer! Please reach out to chat and/or come to the following:
- Monday 9:40 AM: Spotlight pres on detecting side effects at the Rethinking Safety Workshop by @ryanlindeborg (sites.google.com/andrew.cmu.…)
- Tues 4-5PM: Come visit my RSS Pioneers Poster! @RSSPioneers
- Wed 3:15PM: Robometer Reward Fxn talk + poster after by @yigitkkorkmaz and @aliangdw (robometer.github.io/)
- Thurs 3:15PM: TMRL pre-training for post-training talk + poster after by @matthewh6_ (weirdlabuw.github.io/tmrl/)
- Friday 9:30AM: Giving an invited talk at SemRob Workshop (semrob.github.io/)
- Friday 5:30PM: Invited talk at the Diffusion Workshop (rss2026-diffusion-robot-lear…) on TMRL
Sarvesh Patil retweeted
Presenting PTLD: an approach to learning tactile dexterous policies without ever simulating the tactile sensor.
Tactile is essential for performing highly dexterous manipulation. However, collecting tactile observations reliably has been a fundamental bottleneck: (1) teleoperating a multi fingered hand for dynamic tasks is challenging, making sim-to-real imperative, (2) one can’t realistically simulate tactile today
Sarvesh Patil retweeted
Do we still need Uncertainty Quantification for robotics? What does uncertainty mean in the modern robotics paradigm?
💡 We are organizing a workshop “Rethinking Uncertainty for Modern Robotics Paradigms” @ IROS 2026
Submit your work by Aug 23! 👇 (rethinking-uncertainty.githu…)
Sarvesh Patil retweeted
gov keeps handing anthropic marketing Ws
The US government, citing national security authorities, has issued an export control directive to suspend all access to Fable 5 and Mythos 5 by any foreign national, whether inside or outside the United States, including foreign national Anthropic employees.
The net effect of this order is that we must abruptly disable Fable 5 and Mythos 5 for all our customers to ensure compliance.
Access to all other Claude models is not affected.
We apologize for this disruption to our customers. We believe this is a misunderstanding and are working to restore access as soon as possible.
Read our full statement: anthropic.com/news/fable-myt…
Sarvesh Patil retweeted
💥Introducing FACTR 2, learning external force sensing on commodity robot arms without needing dedicated sensors.
We show that learned force signals enable force-feedback teleop on low-cost arms and improve BC policies.
FACTR 2 consists of:
1. Neural External Torque (NEXT): learns external forces without needing dedicated force sensors.
2. Force-Informed Re-Sampling Training (FIRST): uses the learned force signal to identify task-critical regions and upsample them during training.
w/ @StevenOh_ @_tonytao_
🧵(1/N)
Sarvesh Patil retweeted
终于有人把LLM的推理黑箱拆开了：只看最终答案和token数根本不知道模型是在真推理还是瞎绕路，准确率相同的模型推理结构可能天差地别，这才是评估的盲区。
论文从根源下手——构建逻辑谜题基准LRM，把原始推理链条解析成有向无环图，节点是论断，边是依赖关系，再定义推理效率指标量化逻辑流的集中程度。
分析DeepSeek-R1等模型发现：同等准确率下推理图复杂度竟能相差数倍，哪个更可靠一目了然。
arxiv.org/abs/2606.03883
Sarvesh Patil retweeted
We release Diamond Maps💎 unlocking accurate and efficient guidance for diffusion models. Our experiments show that our methods scale incredibly well. Excited to see what people will build with this!
Accurate guidance has been a notoriously hard problem, but in this work, we’re bringing TWO (!) solutions to the table. The recipe for success:
1️⃣ Speed: Use distilled models (flow maps, mean flows, consistency models).
2️⃣ Exploration: Inject stochasticity to properly explore your search space.
Because this fundamentally improves anything using flow matching and diffusion, we see a lot of potential for applications across audio, robotics, molecules, and beyond.
Paper: arxiv.org/abs/2602.05993
Code: github.com/PeterHolderrieth/…
Huge thanks to an amazing team: Douglas Chen, @LucaEyring, @ishin_shah, Giri Anantharaman, @electronickale, @zeynepakata, Tommi Jaakkola, @nmboffi, and @max_simchowitz. It was awesome bringing this to life together!