Taken together, these results suggest (but do not yet prove!) a broader principle in AI: that discrete representations are really really really powerful, and enable greater policy expressivity. Stay turned for some more ambitious work coming out that will exploit these insights in VLA design and RL ;)

Ok, but aren’t you throwing away geometry with binned representations? No How does the policy know that nearby bins are “similar”?  Discrete policies learn geometry in their embeddings, rather than having them imposed from the continuous-vector representation of actions. This is peak bitter lesson.

In other words, continuous policies are constrained by representing actions in ambient Euclidean geometry, which necessarily imposes these (overly strong) inductive biases towards smoothness.

Why do discrete policies do better than their NFE-matched continuous counterparts? These policies are much more reactive, because token-based representations enable better “switching” between actions, both in response to observation conditioning, and faster “switching” over the course of the chunk. We can see this by plotting Jerk ratios and Lipschitz constants.

The real challenge with discrete representations is predicting consistent actions across bins. Because we use a softmax predictor for each quantized bin, these probabilities are independent for each forward pass of the model. So we need multiple steps of prediction to make sure the probabilities factorize correctly (e.g. AR or discrete diffusion). Discrete Diffusion works better and has less generation error.)

We find that discrete flow models match and even can outperform continuous flow policies, and the same for discrete regression v.s. continuous regression. Quantization error (which we had all assumed was a problem) is not an issue.

In detail. train robot policies that predict discretized actions: each action chunk consists of some number of bins for each action dimension and time step. We then train policies that generate actions either in a single forward pass (discrete regression, DR), and multiple passes both autoregressive generation, and discrete flow matching (DF). We compare these to popular continuous-action parametrizations like flow (CF) and L_2 regression (CR)

This is one of my favorite projects I’ve been a part of this year. It’s ostensibly about robotics, but I feel that I learned something very fundamental about how AI models should represent all modalities that (superficially!) seem like they are best suited to being represented in continuous-valued (e.g. float) vectors. TL;DR discrete parametrizations are awesome, learn geometry from data, but aren’t constrained by it, and can perform better as a result. Exactly what the bitter lesson tells us we should look for.

super cool work! definitely updated my priors. Thanks @jayden_teoh_ for pointing this out

There seems to be fear reasoning models will trivialize research on robot learning architectures, algorithms, etc... I disagree. Innovation comes from constraints, and robotics imposes a ton of then. Our new paper, led by @DashoraNitish , suggests a really promising new direction for meeting the constraints of real robot deployment whilst capturing the "intelligence" of agentic reasoning models. Check it out!

well done!! v timely.

thanks ill read it!!

everyone talks about AI model alignment, but what about AI oligopoly alignment? Suppose a super powerful A̵I̵ ̵m̵o̵d̵e̵l̵ AI company thought it was acting in the public interest but was actually undermining it. How would we stop i̵t̵ them?

why?

shoot me an email!

Personal Update: I'll be splitting my responsibilities between CMU and Amazon FAR over the next year. I'm convinced we need a unified understanding of how the axes of scale, data, optimization and control interact for robot learning, and am excited for my lab + FAR to push towards uniting the "science of robot learning" with the "science of deep learning" at scale. Please reach out if you are looking for a full-time position building powerful multimodal AI models for deployment on hardware, and if you are intern looking to tinker and do good science on architectures, optimizers, data, tokenization, VLAs, WAMs and more!

backprop through the whole model, and find a way to scale more flops (BoN, GRPO, advantage subtraction) per update

thoughts on robot/generative RL: the best regularization is more expressive updates, not less. What we've learned from the LLM community is that data and model scale lead to the most informative and stable gradients, and we should do the same in robotics.