Ruoyu Sun (@RuoyuSun_UI) profile picture

Ruoyu Sun

@RuoyuSun_UI

Associate Prof at CUHK-Shenzhen. Prev: assistant prof @UofIllinois; postdoc @Stanford; visitor @AIatMeta Work on optimization of machine learning, DL, LLM.

 Shenzhen, China
ruoyus.github.io/
Joined December 2010
  • Tweets139
  • Following634
  • Followers1.3K
40Media
Photo by Ruoyu Sun (@RuoyuSun_UI)Photo by Ruoyu Sun (@RuoyuSun_UI)Photo by Ruoyu Sun (@RuoyuSun_UI)Photo by Ruoyu Sun (@RuoyuSun_UI)Photo by Ruoyu Sun (@RuoyuSun_UI)
Cool to see that on vision problems with unpaired data, relativistic GAN is still SOTA openaccess.thecvf.com/conten…
1
8
1
80
14,113
We did a very careful study of 10 optimizers with no horse in the race. Despite all the excitement about Muon, Mars, Kron, Soap, etc., at the end of the day, if you tune the hyperparameters rigorously and scale up, the speedup over AdamW diminishes to only 10% :-( Experiments are made possible by Marin (github.com/marin-community/m…); anyone developing new optimizers: please come try your method on this benchmark!
Fantastic Pretraining Optimizers And Where to Find them · Issue #1290 · marin-community/marin

Fantastic Pretraining Optimizers And Where to Find them · Issue #1290 · marin-community/marin

Description We evaluate a suite of optimizers on Transformer-style language models (130 M–1.2 B parameters) trained on up to 16× Chinchilla-optimal data. The goal is to quantify real speedups under...

github.com
Kaiyue Wen (@wen_kaiyue) profile pictureKaiyue Wen
@wen_kaiyue4 Sep 2025
(1/n) Check out our new paper: "Fantastic Pretraining Optimizers and Where to Find Them"! >4000 models to find the fastest optimizer! 2× speedups over AdamW? Unlikely. Beware under-tuned baseline or limited scale! E.g. Muon: ~40% speedups <0.5B & only 10% at 1.2B (8× Chinchilla)!
Image from Kaiyue Wen (@wen_kaiyue)
17
78
21
685
182,325
Ruoyu Sun retweeted
Durk Kingma (@dpkingma) profile picture
Durk Kingma@dpkingma3 Dec 2022
“Adam can converge without any modification on update rules” arxiv.org/abs/2208.09632 Proves that (vanilla) Adam is theoretically justified without any modification. Presented at NeurIPS'22.
7
71
3
381
Load more