CS PhD student @PrincetonCS, Currently Intern @nvidia, Previously CS MEng + BS @CornellCIS
Joined November 2021
- Tweets75
- Following201
- Followers1.1K
- Likes159
Pinned Tweet
🚀SonicMoE🚀now runs at peak throughput on NVIDIA Blackwell GPUs 😃
54% & 35% higher fwd/bwd TFLOPS than the DeepGEMM baseline and 21% higher fwd TFLOPS than the triton official example. SonicMoE still maintains its minimum activation memory footprint: the same as a dense model with equal activated parameters and independent of expert granularity. We wrote a blogpost on how we leveraged Blackwell features and the software abstraction on QuACK:
Work with @MayankMish98, @XinleC295, @istoica05, @tri_dao
Wentao Guo retweeted
🚀 Kernel Design Agents to Optimize Kimi Delta Attention (upto 2.96x)
When the KDA (Kernel Design Agents) project was initiated, its name happened to coincide with another KDA (Kimi Delta Attention). More than once, people asked us , "Can we use KDA to write a version of KDA?" Today, right at a moment with a full moon good for shooting, we are thrilled to share the latest results of KDA(gent)-v0.6: using KDA(gent) to optimize KDA(ttn)!
KDA-v0.6 achieves this through
* Humanize2 Flow & LLM Collaboration Optimization: Introducing multi-round iterative evolution with Flame Chase, GPT-5.6-Sol, and Fable-5, paired with an automated Workspace cleanup mechanism to significantly broaden the operator search space.
* Multilingual Domain-Specific Skill Stack Support: Fully supporting CuTe-DSL, CUDA C++, as well as recent Agent-Native CAKE IR and TIRx. Combined with customized diagnostic tools (such as CuTe's dedicated IKET pipeline analyzer and TIRx's CPU-side numerical simulation and static checks), achieving deep performance bottleneck diagnostics.
* Self-Evolving Kernel Wiki: Building a dynamically updated knowledge base that automatically cleans up invalid/erroneous implementations, corrects classification tags, and streamlines search indexing, significantly improving the Agent's knowledge retrieval quality and code generation accuracy.
KDAgent ultimately optimized the KDAttn kernel to 2.96x of FlashKDA, and we face a lot of unexpected hacks during the optimization. More details shared in our blogs nvlabs.github.io/kda/blog/20…
Release results of KDA for KDA: github.com/NVlabs/kda/tree/2…
Community & Production Deployment:
All kernels are open-sourced for community verification! For deploying NVIDIA Agentic CUDA acceleration in production environments, we recommend using the production-level version rigorously verified by FlashInfer.
By the Team: Dongyun Zou, Yixin Dong, Hongyi Jin, Junxian Guo, Yahui Cui, Avery Huang, Zihao Ye, Junru Shao, Changye Li, Zijian Zhang, Sihao Liu, Song Bian, Ligeng Zhu.
Wentao Guo retweeted
⚡ Attention kernels are well-optimized on GPU. But what about TPU?
🎯 We release Flywheel Attention: TPU-native softmax attention (MHA / GQA) + Gated DeltaNet kernels. Flywheel is 1.56× faster than the best open-source MHA/GQA kernel and 3.61× faster than the best open-source GDN kernel we could find. 👇
Wentao Guo retweeted
We pretrained a 2.3B MoE (360M active) Hybrid Mamba-2 that lands within a few points of Llama-3.2-3B using <1% of its pretraining FLOPs.
No dedicated cluster. The run hopped between H100s, A100s, V100s (yes, V100s) and TPU v5p/v6e on a single codebase.
Meet Rigel 🧵
Wentao Guo retweeted
Open-source video generation is now faster than playback without compromising quality.
Introducing Video Delta Net (VDN): hybrid attention for live text-to-video with near-lossless quality.
VDN accelerates Minimax-H3 by 75 - 90 x, generating 14 seconds of 768p video in 11 seconds on 8× NVIDIA B200 GPUs.
Checkpoints + training/inference code + Technical Blog ⬇️
(1/6)
Wentao Guo retweeted
🎉 Thrilled to share that our paper, “Thought-Level Beam Search for Reasoning,” has been accepted to COLM 2026!
Different from Self-Consistency and Pruning-only approaches, Gambit actively reallocates test-time compute while generation is in flight — pruning low-scoring reasoning traces and branching from the strongest prefixes. See thread 🧵
Wentao Guo retweeted
Robot policies can move but can't think.
LLMs can think but can't move.
So we connected them.
Real robot: 16.7% → 97.3%
Sim (LIBERO-PRO): 12.8% → 53.3%
Wentao Guo retweeted
This was a large cross-domain effort, made possible by collaborators across systems, language models, robotics, AI for Science, trustworthy learning, and more.
Huge thanks to everyone who helped build, validate, and test MLS-Bench:
Bohan Lyu, Yucheng Yang, Siqiao Huang @KnightNemo_, Jiaru Zhang, Qixin Xu, Xinghan Li @XinghanLi66, Xinyang Han, Yicheng Zhang, Huaqing Zhang @zhqwqwq, Runhan Huang @RunhanH, Kaicheng Yang, Zitao Chen, Wentao Guo @WentaoGuo7, Junlin Yang @junlin45300, Xinyue Ai @KeelyAi04, Wenhao Chai @wenhaocha1, Yadi Cao @YadiCao, Ziran Yang @__zrrr__, Kun Wang @KunWang0129, Dapeng Jiang, Huan-ang Gao, Shange Tang @sangertang1999, Chengshuai Shi @chengshuai_shi, Simon S. Du @SimonShaoleiDu, Max Simchowitz @max_simchowitz, Jiantao Jiao @JiantaoJ, Dawn Song @dawnsongtweets, and Chi Jin @chijinML. (8/9)
Wentao Guo retweeted
Why do we store the SSM state at all?
More and more models are hybrids (Nemotron-3, Qwen3.5), so SSM decode speed matters.
We only write it back every step so the next step can read it.
ReplaySSM caches the recent inputs instead and rebuilds the state on the fly.
Same outputs, half the memory traffic
→ ~2x on spec decode at large batch sizes, which barely even helped SSMs before
→ up to 1.43x standard decode on large hybrids (up to Nemotron-Ultra-550B)
Work with @tri_dao
Blog + Code👇
Wentao Guo retweeted
Excited to share that LessIsMore has been accepted to ICML 2026! 🚀
LessIsMore is a training-free sparse attention for efficient long-horizon reasoning. By enforcing cross-head unified token selection, it brings up to 1.6x E2E speedup while preserving reasoning accuracy under practical workloads.
Huge thanks to my amazing co-authors and mentors @Jackfram2, @JiaZhihao, Ravi!
Paper: arxiv.org/abs/2508.07101
Code: github.com/DerrickYLJ/LessIs…
#ICML2026 #LLM #EfficientAI
Again, great thanks to NVIDIA CUTLASS team @nvidia, PLI @PrincetonPLI and AI Lab, SkyLab @BerkeleySky and Together AI @togethercompute
[7/N]
A detailed blogpost of our approach describing how we use Blackwell's TMEM double-buffering, 2CTA MMA, CLC tile scheduling, and gather fusion to hide IO costs behind MMA compute. We also walk through QuACK's software abstraction, and a few ablation studies on Blackwell GPUs.
Blogpost: dao-lab.ai/blog/2026/sonicmo…
[6/N]
All SonicMoE Grouped GEMM kernels are built on QuACK. Each kernel overrides a single function. The most complex kernel (dH backward) only adds ~200 LoC for SonicMoE on top of QuACK, and the same code runs on Hopper and Blackwell GPUs.
[5/N]
On Hopper, we leverage Ping-Pong warpgroup scheduling to overlap the heavy epilogue IO with the tiled GEMM computation.
On Blackwell, the hardware does it differently but in the same spirit: we have a dedicated on-chip tensor memory (TMEM) split into two accumulator stages. The MMA warp fills one stage while epilogue warps handle the other stage, then they swap. The heavy epilogue IO is overlapped with the tiled GEMM again, which we elaborate on in the blogpost.
[4/N]
SonicMoE is designed for fine-grained MoEs. It achieves greater relative speedup over existing MoE baselines when we increase the expert granularity on B300 GPUs.
[3/N]
On B300 GPUs, SonicMoE achieves 54% & 35% higher fwd/bwd TFLOPS than the DeepGEMM baseline and 21% higher fwd TFLOPS than the triton official example across 6 open-source MoE configs (7B to 685B). SonicMoE often doubles the achieved TFLOPS over ScatterMoE and MoMoE.
[2/N]
Modern MoEs are scaled towards the fine-grained regime where we have more smaller experts to activate. However, the activation memory footprint of existing MoE kernels will linearly increase and drain the VRAM resources. We instead compute the MoE backward pass in a different but math-equivalent way. Activation memory usage is now the same as a dense model with equal activated parameters, with 45% reduction from community MoE kernels. SonicMoE’s activation memory usage is also independent of expert granularity.
[1/N]
Blogpost: dao-lab.ai/blog/2026/sonicmo…
Pip: pypi.org/project/sonic-moe/
Run pip install --upgrade sonic-moe to try it out!
We updated the paper to include Blackwell results: arxiv.org/abs/2512.14080
Code: github.com/Dao-AILab/sonic-m… - Feedback and contribution are welcome!
Wentao Guo retweeted
We made Muon run up to 2x faster for free!
Introducing Gram Newton-Schulz: a mathematically equivalent but computationally faster Newton-Schulz algorithm for polar decomposition.
Gram Newton-Schulz rewrites Newton-Schulz such that instead of iterating on the expensive rectangular X matrix, we iterate on the small, square, symmetric XX^T Gram matrix to reduce FLOPs. This allows us to make more use of fast symmetric GEMM kernels on Hopper and Blackwell, halving the FLOPs of each of those GEMMs.
Gram Newton-Schulz is a drop-in replacement of Newton-Schulz for your Muon use case: we see validation perplexity preserved within 0.01, and share our (long!) journey stabilizing this algorithm and ensuring that training quality is preserved above all else.
This was a super fun project with @noahamsel, @berlinchen, and @tri_dao that spanned theory, numerical analysis, and ML systems! Blog and codebase linked below 🧵
Wentao Guo retweeted
Introducing M²RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling
We bring back non-linear recurrence to language modeling and show it's been held back by small state sizes, not by non-linearity itself.
📄 Paper: arxiv.org/abs/2603.14360
💻 Code: github.com/open-lm-engine/lm…
🤗 Models: huggingface.co/collections/o…
