I think the last month might have been the biggest month in (zeroth-order) optimization history. In the 10 days since we released the Noise speedrun, the time to pretrain a transformer went from 1.76 hours to 1.6 seconds, a 4,000x (!!!) speedup. Dust itself was up to 10,000x more compute-efficient than the previous state-of-the-art. Not only can we pretrain a transformer without backprop, it's much more efficient than anyone would've expected. Search fking works! Now, scaling is the biggest open question. Some of the tricks that brought the time down to 1.6 seconds will simply not scale. To discard those, we'll be scaling the dataset every time it gets overfit, starting with 30M tokens instead of 10M. Huge thanks to @jayden_teoh_, @spicey_lemonade, @RobertTLange, @EmanueleRodola, @ChunyuanDeng, @LuochengHuman, @Alaxarr, @pavleculafic, @IsaacYong13, Euraba and areu01or00 (GitHub), for the contributions. And @cs_serdar for gracefully maintaining the speedrun and @bishmdl76 for helping out. Expect a lot more from us over the coming weeks.

"A painting that is well composed is half finished." — Pierre Bonnard #Arts #QOTD

Inspired by Nick Cammarata’s interpretability essay “Neural Networks Are Already Clean”, I turned a couple of its ideas into speedups for pretraining without backpropagation. Still in the works. 2.3x speedup, cool xitter.cf/industriaalist/status/…

the post derives all of this step by step, with interactive figures where you can watch memories pile up yourself plus pointers to the latest work on memory like recent works! alexwang.ai/posts/associativ…