
Previously: Research Scientist @UCBerkeley • Postdoc @MIT_CSAIL @MIT • PhD from @SCSatCMU @CarnegieMellon • Generative AI and RL 🤖
Berkeley, CA
Joined March 2010
- Tweets388
- Following647
- Followers1.2K
- Likes2.3K
Pinned Tweet
Why is action chunking so important for robot behavior cloning? Even when open-loop execution reduces the policy’s reactivity? 🤖 Our latest work identifies the most important reason: **action chunking helps short-memory policies imitate long-memory experts**. More precisely, action chunking is needed when expert demonstrations are *non-Markovian* with respect to the robot’s observation context.
We provide two broad pieces of evidence. We either (1) imitate Markovian experts that only use the single most-recent observation, or (2) we just increase the policy's context length (memory). Under either intervention, closed-loop single-action policies outperform open-loop action chunking! The second result is particularly actionable: it says we should be designing policies with long context lengths and short action chunks since they are more reactive AND more performant!
I like to think about this phenomenon via my favorite Chris Nolan movie: Memento (2000). In Memento, the protagonist suffers from anterograde amnesia — he forgets anything outside a 15 minute window, much like our robot policies! But he needs to execute an elaborate plan to catch the bad guy. To do so, he keeps writing his plans down on notes, photos, and even tattoos across his body, so that he doesn’t forget his own thoughts. Similarly, it seems that our amnesiac robot policies use action chunks as a *scratchpad* to remember what to do. But if we can fix our policy’s memory loss, we no longer need the scratchpad!
Action chunking — especially executing long action sequences open-loop — is widely used in imitation learning for robotic manipulation. Why is it so effective and do we really need it? We find a key reason:
Long open-loop execution helps short-context policies imitate non-Markovian experts.
With this insight, we show how to move beyond open-loop execution: extending policy context restores reactivity while achieving even higher task performance.
🧵(1/5)
The disappointing performance of the new Astra/Sol 6.2 models is really to boost Anthropic’s upcoming IPO. And I’m sure Anthropic will return the favor before OpenAI’s IPO. It’s a win-win strategy!
8
533
Siddharth Ancha retweeted
Replying to @Nicolas_Keller @josermedinahdez @GaoyueZhou @jeffacce @ylecun @LerrelPinto @carlotapares @leto__jean @agiachris @RohanSinhaSU @contactrika @drmapavone @liu730chaoqi @xshenhan @WinstonGu_ @du_yilun @GeYan_21 @RanjayKrishna @imankitgoyal @xiaolonw
13/17 — Open-loop
Revisiting Open-Loop Execution in Robotics: Toward Reactive, Higher-Performing Policies
With real-time chunking, longer history came out ahead on our robots.
@michzeng_, Abhinav Agarwal, Ajay Bati, Brian Lee, @siddancha, @RussTedrake
This is beautiful and inspiring @octi_zhang! Keep up the cool work!
zoctipus.github.io/#/about
These guys want to match the intelligence of OpenAI/Anthropic and the creepiness of Elon Musk.
7
1,474
Siddharth Ancha retweeted
Replying to @suchenzang
There should not be one evaluator. There should be a market and there should be financial incentives to uncover misalignment.
Also, Dandelion Chocolate is hands down the best chocolate in the Bay area! 🍫🌉
A video from a Pi robot deployed at Dandelion Chocolate, fully autonomous w/ no interventions. 🤖
Deploying robots has taught us surprising lessons about the gap between proof-of-concept (i.e. building one box) and real-world utility (productively building boxes for hours).
I expected that the hard part is building the box, since it’s the most dexterous, but that wasn't what we found. Counterintuitively, the hardest part was reliably stacking the boxes. Our original table-mounted robot had poor visibility of the stack without special separately-mounted cameras. Plus, stacking requires more generalization (each box is placed in a different location), and an imprecisely-placed box can lead the entire stack to collapse many boxes later.
We recently switched this deployment over to a mobile robot, and it's fun to watch the robot being completely self sufficient for multiple hours. 🙂 Data and feedback from real world use-cases like these are quite valuable for the π pre-trained model as we scale!
6
657
We live in interesting times! Previously we would painstakingly collect a labeled dataset for each task, train models and hope they would generalize (they wouldn’t). Now we can ask agents (with appropriate harnesses) to do just about anything. Token heavy but works zero shot!
In the old days we used to talk about "graduate student descent" or "researcher gradient descent" as referring to the phase in research where one tries various techniques, changes parameters in the hope of getting to some desired output. Now we can get AI agents to do it. The classic CV paradigm of analysis by synthesis becomes considerably more powerful this way.
6
818
Siddharth Ancha retweeted
When we first started working on Diffusion Policy, which felt like an "unlock" for fitting human demonstration data, the explanation I gave myself was that "human demonstration data was subtly much more multimodal than we thought, so we need the modeling power of denoising diffusion to fit it properly."
What Michael's paper brings to light is that some of this multimodality was probably exacerbated (more than I realized) by the fact that we've only been giving our models incredibly short observation context lengths.
For me, this is pretty compelling evidence that we should be pushing for policies with long-context lengths and short action prediction horizons.
Action chunking — especially executing long action sequences open-loop — is widely used in imitation learning for robotic manipulation. Why is it so effective and do we really need it? We find a key reason:
Long open-loop execution helps short-context policies imitate non-Markovian experts.
With this insight, we show how to move beyond open-loop execution: extending policy context restores reactivity while achieving even higher task performance.
🧵(1/5)
Really elegant work on fixing variational RL with diffusion models using self-distillation! 👏👏
*Problem*: Likelihood gradients ∇ log p(a) are needed for RL but are hard to compute for diffusion models. A useful surrogate is ∇ ELBO(a), where the ELBO is a supervised diffusion loss such as denoising score-matching loss (DSM) or conditional flow-matching loss (CFM); e.g. the latter is used in FPO (arxiv.org/abs/2507.21053). However, the ELBO suffers from a KL gap: which causes the gradients of ELBO(a) and logp(a) to be misaligned:
ELBO(a) = log p(a) - D_KL(a)
*Idea*: Drive the KL gap down to zero after each policy gradient step, reminiscent of the EM algorithm for training GMMs. The E-step drives the KL gap to zero by computing the posterior distribution over latents *without changing the generative model.* Then, the M-step optimizes the ELBO; since the KL gap is already zero and can only increase, the M-step guarantees that the log-likelihood will also increase.
*Solution*: The rough equivalent of the "E-step" in diffusion models is self-distillation. You sample action labels from the current diffusion model and just do supervised learning (minimize ELBO) to match its own actions. This might seem useless because self-distillation does not modify the terminal action distribution at all! *The generative model and log-likelihoods log p(a) are unchanged*. But this drives the diffusion KL gap to zero! The KL gap in diffusion models is the distributional gap between the noising and denoising paths; a perfectly trained diffusion model that exactly reverses the noising process has zero KL gap. When this happens, D_KL(a) = 0. But zero is the global minimum for KL divergence, and so ∇D_KL(a) = 0. Which means that ∇ log p(a) = ∇ ELBO(a). The true and surrogate policy gradients are exactly aligned after self-distillation! Takeaway: Interleave policy update steps with self-distillation steps!
Cool work on refining coarse VLM actions using a flow matching policy π(a₀ | o) → a₁ where a₀ ∼ N(0, 1) by first reversing (inverting) the given coarse action a₁ via â₀ = π⁻¹(a₁ | o) and then reconstructing it in the forward direction i.e. â₁ = π(â₀ | o).
The interesting bit is that this method crucially **relies** on Euler integration error! If you use a vanishingly small integration step size (which is usually supposed to be a good, albeit expensive, thing), there is no refinement i.e. â₁ = a₁!
RIP Prof. Dimitri Bertsekas! 🙏 I learned so much from his seminal textbooks on optimal control, dynamic programming and RL. Unfortunately I never got to meet him in person, but I corresponded over email about some typos I found in his books.
He cared very deeply about the quality of his books, and would reply **instantly** to any issues I sent his way. I fondly recall a long back-and-forth with him about whether something I flagged in his book was indeed a mistake. Rather than getting annoyed at being pestered by some random graduate student, he engaged with me very patiently and generously -- and eventually thanked me for my perseverance. I will always remember his kindness and generosity!
42
5,805
Siddharth Ancha retweeted
boston dynamics uses mjlab btw :)















