apaz

I've joined Anthropic! I'm excited to get back to (frontier) research and work on the models that I fell in love with a decade ago. I’m very grateful for my time at @AMD ♥️ Starting with the people: I'm honored to have had the opportunity to work with the one and only @LisaSu! She's an amazing leader, manager, and friend who allowed me so much agency in my work -- they say don't meet your heroes, but she definitely lives up to the pedestal that I've put her on. And thank you to my team: dedicated, resilient folks who've been brave enough to jump on a rocket ship together, in some cases several times to ever ambitious projects. For those who came with me from my startup Lamini - thank you also for bringing your passion, wonder, kindness, and (yes) silliness with me, as we found a home at AMD. And now the work: I built and ran an AI research org that led our frontier lab partnerships, where I served as executive sponsor. We were among the first in the AI research world to take RL kernel optimization really seriously last year, and it showed me that pushing a frontier capability forward takes far fewer people than you’d think (100x, and I'm now updating my priors to 1000x fewer, than just a year ago). That work has also had massive implications for the compute that labs and companies can now use out-of-the-box with models like Claude. That is to say, it’s always time to make some GPUs go brrr.

Releasing mlsweep, a sweep scheduler and visualizer for distributed ML training. It aims to make launching runs across groups GPUs frictionless and achieve near feature-parity with wandb. But you can use it with whatever frameworks or loggers you like, wandb included.

We're announcing the public beta of datavendor.ai We spent the last two years building the best solution for people buying and selling data, whether you're a researcher in a lab or a company trying to sell data. Data is the fundamental bottleneck for AI progress, and DataVendor will be the infra that solves it.

what if you just shot your infra engineers

Conventional zeroth-order optimizers such as MeZO estimate gradients through random parameter perturbations, wasting evaluations on directions unrelated to meaningful updates. AGZO from ICML26 exploits a simple but powerful geometric property: the weight gradient of a linear layer lies in the subspace spanned by its input activations. Rather than sampling perturbations across the entire parameter space, it extracts low-rank activation subspaces during forward computation and restricts gradient estimation to those directions. Under spectral concentration assumptions, the authors prove better expected alignment with the true gradient than isotropic perturbations. On Qwen3-0.6B, AGZO achieves 89.2% accuracy on CB versus 80.3% for MeZO, while enabling Qwen3-4B fine-tuning in a hardware setting where full first-order training runs out of memory. The deeper insight is that avoiding backpropagation does not require treating neural networks as structureless black boxes. Forward activations already reveal useful information about optimization geometry, connecting zeroth-order learning with randomized linear algebra and low-dimensional representation structure. However, the method remains substantially more iteration-intensive than first-order training, and its advantages have not been established at frontier pretraining scale. The next question is whether adaptive activation subspaces can reduce both gradient-estimation noise and the number of forward evaluations required for effective learning. Perhaps the next generation of optimizers will use forward computation not merely to evaluate loss, but to determine where parameter updates should occur. arxiv.org/abs/2601.17261

Looped transformers often save compute by truncating backpropagation to the last few loops. Is it worth it? We trained ~600 models to find out 👀 Short answer: only if the loop count is fixed. Otherwise, run fewer loops and train them all! 📄 arxiv.org/abs/2610.26030 🧵👇

Backprop has been the only credit assignment algorithm capable of training large neural nets. Introducing Dust, the first zeroth-order method to pretrain transformers to approach and sometimes even *exceed* backprop with large amounts of computation (bitter lesson!) - Dust is on the order of 1,000 to 10,000x more compute efficient than EGGROLL, the state-of-the-art ES method, for training transformers. - We chose pretraining transformers because it's arguably the hardest task possible for zeroth-order optimization. - Search in high dimensions is really misunderstood: zeroth-order usually gets better with overparameterization, not worse, i.e. at a fixed population size, larger models (up to 120x) reach a lower loss than smaller ones. - Backprop-like gradients emerge from a large population of activation perturbations, and the alignment holds up at every scale we tested, up to 1B tokens, which is critical for scaling. The core idea is a *virtual population*, which helps scale to large populations in transformers by perturbing activations in parallel. w/ @bishmdl76, @cs_serdar, @akshayvegesna