Research Scientist @ Google Deepmind
San Jose, CA
Joined October 2011
- Tweets118
- Following514
- Followers437
- Likes248
Sai Bi retweeted
Hi all, I have a new result I'm excited to share. It turns out that *all existing implementations* of softmax and cross-entropy (that's JAX, PyTorch, textbooks, etc) have a subtle bug --- two bugs, actually. A simple rewrite fixes both, and eliminates "slingshots" when grokking:
Sai Bi retweeted
Releasing the model weights and technical report of Kimi K3.
Kimi K3 is our most capable model: a 2.8T MoE model with native visual understanding and a 1M-token context window.
New model architecture: 2.5x the intelligence per unit of compute, not just more params.
Alongside Kimi K3, we're opening up more of the stack behind it — high-performance attention kernels, MoE communication library, and infrastructure for running agent environments at scale.
Model weights: huggingface.co/moonshotai/Ki…
Tech report: github.com/MoonshotAI/Kimi-K…
Tech blog: kimi.com/blog/kimi-k3
Sai Bi retweeted
Research ideas you can't be outscaled on.
Many important problems in ML now demand compute no university lab can match, and junior students feel the constant pressure of getting scooped. I wanted to share a different bet: research directions where insight may still matter more than scale.
These are deliberately unconventional — grounded in mechanisms rarely seen in the usual scaling playbook. Because they sit in the long tail, they're unlikely to show up in AI-generated research idea lists. No guaranteed success, but real open questions with enough material to get started.
If you find these useful, PRs to add more are welcome. I'll keep updating too.
antheali.github.io/open-rese…
Sai Bi retweeted
New paper: AsymFlow🔥
JiT x0-prediction is not enough for pixel generation. Better keep velocity in a low-rank subspace:
- 1.57 FID on ImageNet (best pixel flow model)
- Finetunes FLUX.2 klein into pixel space, beats the original on HPSv3/DPG/GenEval (#1 overall on HPSv3)
1/7
Sai Bi retweeted
This is what we've been coking for the last 9 months: make MoEs training goes ~2x faster and ~2x less memory! Highlights:
- MoE typically takes the most time and memory in modern models. Turns out one can mathematically rewrite the MoE backward pass to reduce the activation mem you need to store in the fwd by ~2x, resulting in the same gradients with no extra matmul recomputation. I really like this result, as it combines both algorithmic and systems insights.
- Analyzing bottlenecks in MoE layer leads to a natural optimization stragegy: reduce mem reads/writes as much as possible! Gathering the input for fwd and output grad for bwd can sometimes take as much time as the grouped GEMMs. We fuse gather with grouped GEMM + overlap mem access and compute to make the whole layer goes ~2x faster.
- Computing top-k for expert routing can take surprisingly long, ~15-20% of the whole MoE layer! Standard top-k impl uses radix top-k algo, great for large k but suboptimal for small k. We rewrote top-k using bitonic top-k algo, and it's sometimes 20-30x faster than pytorch's top-k!
All the main kernels are written in Cute-DSL so they should be easy to extend (and install :D). Hopper kernels are out, Blackwell kernels are just about ready. MoE models used to be 2x less hardware-efficient to train, hopefully Sonic-MOE will change that.
🚀SonicMoE🚀: a blazingly-fast MoE implementation optimized for NVIDIA Hopper GPUs. SonicMoE reduces activation memory by 45% and is 1.86x faster on H100 than previous SOTA😃
Paper: arxiv.org/abs/2512.14080
Work with @MayankMish98, @XinleC295, @istoica05, @tri_dao
Sai Bi retweeted
(1/N) Will this be the BERT/GPT moment for 3D vision？
Finally, unsupervised pre-training for 3D works.
Led by @qitao_zhao , we present E-RayZer — a fully self-supervised 3D reconstruction model that:
🔥Matches or surpasses supervised methods like VGGT
👀Learns transferable 3D representations, outperforming CroCo, VideoMAE, and DINO
📈Scales with more unlabeled data
A new recipe for scalable 3D foundation models.
Sai Bi retweeted
We've been running @radixark for a few months, started by many core developers in SGLang @lmsysorg and its extended ecosystem (slime @slime_framework , AReaL @jxwuyi). I left @xai in August — a place where I built deep emotions and countless beautiful memories. It was the best place I’ve ever worked, the place I watched grow from a few dozen people to hundreds, and it truly felt like home. What pushed me to make such a hard decision is the momentum of building SGLang open source and the mission of creating an ambitious future, within an open spirit that I learnt from my first job at @databricks after my PhD.
We started SGLang in the summer of 2023 and made it public in January 2024. Over the past 2 years, hundreds of people have made great efforts to get to where they are today. We experienced several waves of growth after its first release. I still remember the many dark nights in the summer of 2024, I spent with @lm_zheng , @lsyincs , and @zhyncs42 debugging, while @ispobaoke single-handedly took on DeepSeek inference optimizations, seeing @GenAI_is_real and the community strike team tag-teaming on-call shifts non-stop. There are so many more who have joined that I'm out of space to call out, but they're recorded on the GitHub contributor list forever. The demands grow exponentially, and we have been pushed to make it a dedicated effort supported by RadixArk. It’s the step-by-step journey of a thousand miles that has carried us here today, and the same relentless Long March that will lead us into the tens of thousands of miles yet to come.
The story never stops growing. Over the past year, we’ve seen something very clear:
The world is full of people eager to build AI, but the infrastructure that makes it possible is not shared. The most advanced inference and training stacks live inside a few companies. Everyone else is forced to rebuild the same schedulers, compilers, serving engines, and training pipelines again and again — often under enormous pressure, with lots of duplicated effort and wasted insight.
RadixArk was born to change that. Today, we’re building an infrastructure-first, deep-tech company with a simple and ambitious mission:
"Make frontier-level AI infrastructure open and accessible to everyone."
If the two values below resonate with you, come talk to us:
(1) Engineering as an art.
Infrastructure is a first-class citizen in RadixArk. We care about elegant design and code that lasts. Beneath every line of code lies the soul of the engineer who wrote it.
(2) A belief in openness.
We share what we build. We bet on long-term compounding through community, contribution, and giving more than we take.
A product is defined by its users, yet it truly comes alive the moment functionality transcends mere utility and begins to embody aesthetics.
Thanks to all the miles (the name of our first released RL framework; see below).
radixark.ai
Check out the amazing work by our intern @AntheaYLi on applying Evolution Strategies to improve reasoning efficiently with LoRA and quantization. Yichen is at NeurIPS, and please reach out to her and chat if you are interested.
We look at how Evolution Strategies can be effective to improve reasoning under small population size and low-rank perturbations:
• How population size, noise scale, step size, and LoRA rank interact
• A trust-region + spectral norm lens on the stability of rank
• Forward-only allows for smooth quantization: alleviates training-inference mismatch in RL
Blog(WIP): antheali.notion.site/eses
Sai Bi retweeted
We look at how Evolution Strategies can be effective to improve reasoning under small population size and low-rank perturbations:
• How population size, noise scale, step size, and LoRA rank interact
• A trust-region + spectral norm lens on the stability of rank
• Forward-only allows for smooth quantization: alleviates training-inference mismatch in RL
Blog(WIP): antheali.notion.site/eses
Check out the cool work by our intern @HanshengCh on policy-based distillation for few-step generation.
Excited to announce a new track of accelerating Generative AI:
pi-Flow: Policy-Based Few-Step Generation via Imitation Distillation
github.com/Lakonik/piFlow
Distill 20B flow models now using just an L2 loss via imitation learning for SOTA diversity and teacher-aligned quality.
Sai Bi retweeted
We found that visual foundation encoder can be aligned to serve as tokenizers for latent diffusion models in image generation!
Our new paper introduces a new tokenizer training paradigm that produces a semantically rich latent space, improving diffusion model performance🚀🚀.
Sai Bi retweeted
Wrapped up Stanford CS336 (Language Models from Scratch), taught with an amazing team @tatsu_hashimoto @marcelroed @neilbband @rckpudi. Researchers are becoming detached from the technical details of how LMs work. In CS336, we try to fix that by having students build everything:
I am going to give a talk on scalable 3D reconstructions today at the 3D-LLM/VLA workshop at CVPR at 10:55am today at Room 106A. Welcome to attend! 3d-llm-vla.github.io/
Sai Bi retweeted
🚀 Excited to announce our CVPR 2025 Workshop:
3D Digital Twin: Progress, Challenges, and Future Directions
🗓 June 12, 2025 · 9:00 AM–5:00 PM
📢 Incredible lineup: @rapideRobot, Andrea Vedaldi
@Oxford_VGG,@richardzhangsfu,@QianqianWang5,Dr. Xiaoshuai Zhang @Hillbot_AI, @xiaolonw, @shuangz, @MilosHasan, James Fort @meta_aria.
🔗Details👉projectaria.com/events/CVPR2…
Join us to explore photorealistic, functional & physically-accurate #3DdigitalTwins for Spatial & Contextual AI, Robotics, AR/VR, Digital Content Creation!
#CVPR2025 #3DdigitalTwins #3DVision #ContextualAI #SpatialAI #Robotics #Humanoidrobots #Metaverse #Gen3d
🖼👇
Sai Bi retweeted
Bored of linear recurrent memories (e.g., linear attention) and want a scalable, nonlinear alternative?
Our new paper “Test-Time Training Done Right” propose LaCT (Large Chunk Test-Time Training) — a highly efficient, massively scalable nonlinear memory with:
💡 Pure PyTorch (no custom kernels)
🚀 10× GPU FLOPs utilization compared to previous nonlinear test-time training(ttt) methods.
🧠 Huge memory size (up to 40% of model params)
Project page with code: tianyuanzhang.com/projects/t…
(videos generated with our AR video diffusion)
1/9
Sai Bi retweeted
Excited to attend #ICLR2025 in person this year! I’ll be presenting two papers:
1. LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias
🔹 Oral Presentation: Session 3C (Garnet 216-218) — Apr 25 (Fri), 11:06–11:18 a.m.
🔹 Poster: Hall 3 + Hall 2B, Poster #593 — Apr 25 (Fri), 3:00–5:30 p.m.
🔹 Website: haian-jin.github.io/projects…
2. RelitLRM: Generative Relightable Radiance for Large Reconstruction Models (led by @tianyuanzhang99)
🔹 Poster: Hall 3 + Hall 2B, Poster #531 — Apr 26 (Sat), 3:00–5:30 p.m.
🔹 Website: relit-lrm.github.io/
Feel free to drop by—looking forward to chatting with you!
Check out the fantastic work by our intern @HanshengCh at Adobe Research. The code and model are publicly available!
Excited to share our work:
Gaussian Mixture Flow Matching Models (GMFlow)
github.com/lakonik/gmflow
GMFlow generalizes diffusion models by predicting Gaussian mixture denoising distributions, enabling precise few-step sampling and high-quality generation.
