Ok, but aren’t you throwing away geometry with binned representations? No How does the policy know that nearby bins are “similar”?  Discrete policies learn geometry in their embeddings, rather than having them imposed from the continuous-vector representation of actions. This is peak bitter lesson.

Why do discrete policies do better than their NFE-matched continuous counterparts? These policies are much more reactive, because token-based representations enable better “switching” between actions, both in response to observation conditioning, and faster “switching” over the course of the chunk. We can see this by plotting Jerk ratios and Lipschitz constants.

The real challenge with discrete representations is predicting consistent actions across bins. Because we use a softmax predictor for each quantized bin, these probabilities are independent for each forward pass of the model. So we need multiple steps of prediction to make sure the probabilities factorize correctly (e.g. AR or discrete diffusion). Discrete Diffusion works better and has less generation error.)

We find that discrete flow models match and even can outperform continuous flow policies, and the same for discrete regression v.s. continuous regression. Quantization error (which we had all assumed was a problem) is not an issue.

Nope! Actually, this is the coolest thing. OGPO *increases exploration*. Magically (we don’t enitrely understand why), it pushes variances orthgonal to Q gradients. This means that exploration squeezes the exploration distribution along axes the matter, but preserves exploration across axes where the critic is indifferent . Said otherwise: expressive updates can explore more! (the cool part / n)

Results: OGPO, and its improved variants work really well, particular on complex long horizon tasks with sparse reward. It can complete certain tasks where alternatives stall (e.g. robomimic suite), has 10x the sample efficiency of DPPO, and can even learn from extremely limited data. (13/n)

Even more fun: conservative advantages. We optimize an ensemble of advantages, and take the smallest if all postive, largest if all negative, and 0 if the signs disagree. In the plot below, this kills the dip in offline-to-online adaptation, and beats regularization and cal-QL. Note you can only do this with a zero-order optimization through the denoising because it relies on advantages, *not critic gradient*(12/n)

We propose off-policy generative policy optimization (OGPO). It’s a pretty natural algorithm template. Like DPPO, we treat each denoising step as one step in a “denoising MDP.” We then use this formulation to learn a policy which maximizes a Q critic as a terminal reward. This effectively severs the DPPO formulation after denoising, which has tons of cool benefits (5/n)

some tunes for that last NeurIPS push - Delorean Dynamite (Todd Terje, 2014) youtube.com/watch?v=LUOIvT9h…

🧐🧐 Why do we pretrain LLMs with log likelihood? Why does action chunking work so well in robotics? Why is EMA so ubiquitous? And could their be a mathematical basis for Moravec’s paradox? 🤖🤖 Come check out our NeurIPS  2025 Tutorial “Foundations of Imitation Learning” with @canondetortugas and Adam Block, Tuesday 130-4pm, to find out! (🧵for details)