Building LLMs & trying to understand how they work. Google | Pinterest | PhD, CMU
Palo Alto, CA
Joined October 2010
- Tweets425
- Following754
- Followers537
- Likes1.3K
Po-Wei Wang retweeted
SSMs had the nice asymptotics but kept missing out on the inference goodies Transformers get for free like spec decode and quantization. Turns out we were just not caching the right thing. Same IO-aware trick behind FlashAttention, Mamba scan, paged KV. A lot of the big wins weren’t new architectures, just people taking memory IO seriously.
ReplaySSM now runs in production in GLM-5.3-Flash and Kimi-K3, and ships in TensorRT-LLM, vLLM, FlashInfer, and SGLang!
It speeds up standard decoding (up to 1.47x) and speculative decoding (up to 2.1x), and now also enables a low-precision SSM state whose rounding error never propagates through the recurrence.
Building on our earlier ReplaySSM work, we collaborated with @NVIDIA to pin down what the method really unlocks. Caching SSM inputs instead of the state is what ReplaySSM does. What it changes is that computing the output and updating the state can be separated during decoding. All three gains follow from that separation.
Joint blog post, paper, and code with @NVIDIA and @tri_dao 👇
Po-Wei Wang retweeted
@jon_barron's talk really resonates. 3D was the cool field 5 years ago and progress was largely driven by flashy research demos. As AI models commercialized and went industry scale, vision/graphics folks realized that 3D representations offer true commercial value only within highly specific use cases. Once that became clear, the momentum pivoted toward view synthesis (implicit 3D) and then to video models (more implicit 3D) for robotics. Standalone 3D now stays mostly within the interactive media and manufacturing realms.
My personal bitter(sweet) lesson is to really understand the clear deliverables and viable target applications first. Technological excitement comes after that.
Po-Wei Wang retweeted
Got my Monday night reading! ☕️📚
One of the most fun parts of research is building connections across papers and fields. That's one of the reasons I've enjoyed @sedielem's blog post series a lot since the beginning of my PhD.
And.. For anyone not familiar with self-conditioning, plz do read this paper: arxiv.org/abs/2208.04202
It's recurrent computation striking back — in diffusion, and, perhaps more broadly, in AR too. It's really interesting to see how fundamental a difference it can make *in certain cases*.
(Btw, been cooking something for a while... let me get a blog post up soon on lessons learned from tons of training & inference tokens 👀)
(human added em dashes)
New blog post: continuous diffusion for language is back!
This research direction receded into the background for a while, but as of this year, it is once again a hot topic. I wrote down a historical perspective and some thoughts on the recent revival.
sander.ai/2026/08/24/continu…
Po-Wei Wang retweeted
Apodex 1.1 is out. Weights and harness are fully open:
▸ Apodex 1.1 Mini — 35B FrontierLevel model for Professional Work, Finance, and Scientific Research tasks.
▸ FrontierAgent — Native TUI, ReAct and Agent Team modes.
🐙 github.com/ApodexAI/Frontier…
🤗 huggingface.co/collections/a…
Po-Wei Wang retweeted
A must-read from @_selebou, @SwayStar123, @skywalkeryxc, and collaborators.
I really want to put the people behind this work front and center!
Po-Wei Wang retweeted
In our work led by Shuvo, we designed a domain-specific harness around GPT-5.6 Sol to automate much of first-order optimization research. Shuvo's thread covers the paper itself, so I want to offer some meta-commentary on where optimization research goes from here. (1/7)
What role remains for human researchers as frontier LLMs offer remarkable mathematical intelligence? With Heechang Kim and @ErnestRyu, we add a domain-specific harness to general-purpose LLMs, giving them capabilities unavailable on their own: arxiv.org/abs/2608.07407 (1/7)
Po-Wei Wang retweeted
I'm excited to announce a first development release of nanobind 3.0. It introduces *split mode*, a new distribution model where a single Python wheel runs across all Python versions, while achieving outstanding performance and gaining rather than losing features! 🧵
Po-Wei Wang retweeted
Presenting my grand unified theory of ML researcher impact: Your impact is directly proportional to how much pain you cause to infra.
Fundamentally, you can only inflict pain upon infra if your approach actually works. And the better your approach works the more pain infra is forced to endure.
So, to give some examples:
- MoE's add a ton of data-dependent computation => pain (shazeer++)
- GDN/KDA are the most complex architecture I've been forced to care about and a very annoying matrix inversion => pain (sonta++)
- Muon is much more annoying than Adam and causes annoying restrictions on parallelism => pain (keller/jeremy++)
- RL scaling forced many researchers to care about LLM inference and RL infra as a category => pain (tworek++)
Even papers like Attention Is All You Need have lead to significant pain! Before transformers were invented everyone was running small jobs and I never needed to think about kv-caches or 6D parallelism.
Po-Wei Wang retweeted
New blog: "PyTorch: a reference language" docs.pytorch.org/devlogs/com…
Po-Wei Wang retweeted
I would too!
I have written very few papers solo in the 21st century and I’m quietly pleased with how all of them have aged. 💅
Probabilistic syntax
2003
nlp.stanford.edu/~manning/pa…
Local Textual Inference: It’s hard to circumscribe, but you know it when you see it – and NLP needs it
2006
nlp.stanford.edu/~manning/pa…
Part-of-speech tagging from 97% to 100%: Is it time for some linguistics?
2011
nlp.stanford.edu/pubs/CICLin…
Last words: Computational linguistics and deep learning
2015
aclanthology.org/J15-4006.pd…
Human language understanding & reasoning
2022
amacad.org/sites/default/fil…
I would encourage everyone to re-read @chrmanning 's own opinion piece from nearly 10 years ago where he warns about the "Deep learning tsunami" and NLP being sort of a rabbit in the headlights. In 2015 we were discussing context-free word embeddings. Now we have usable coding agents.
direct.mit.edu/coli/article/…
Po-Wei Wang retweeted
We’re entering a data-scarce regime; but is less data always worse? Our #ICML work shows that small-set training (e.g., with repetition) can be actively leveraged as a useful optimization bias for structured reasoning tasks. Come chat with Jingwen and Ezra in Seoul to know more!
Po-Wei Wang retweeted
If you're looking for questions no one has studied before in the science of deep learning (specifically, about memory & geometry of representations in language models) find @ShNoroozi, @ElanRosenfeld and me tomo at <insert #ICML poster session here>
xitter.cf/_vaishnavh/status/2009…
Po-Wei Wang retweeted
Excited to share new work from our group, Visored, led by @dixiyu1 !
We can now verify math proofs written in Latex! We let LLMs write latex proofs in a controlled natural language, which can be compiled and automatically translated into Lean.
Paper: arxiv.org/abs/2606.17581
We introduce Visored, a dependent-type-based prover designed around how LLMs (and humans) tend to write math, complementing Lean.
Paper: arxiv.org/abs/2606.17581
Code + Lean output: github.com/xiyuzhai-husky-la…
[1/n]
Visored is a semantically rich intermediate representation that carries the full content of a proof. Going Visored to LaTeX and Visored to Lean are easy projections. The hard part of autoformalization is the upward arrow, parsing LaTeX into Visored, where the LLM supplies the prior over what the prose actually means.
Po-Wei Wang retweeted
AI has now solved a major open problem -- one of the best known Erdos problems called the unit distance problem, one of Erdos's favourite questions and one that many mathematicians had tried.
openai.com/index/model-dispr…
Po-Wei Wang retweeted
I wrote a blog post about a Codex harness/workflow I built to autonomously prove a new mathematical result after 3 days of continuous work producing ~60k lines of Lean, where the input is a Lean theorem statement and output is a fully formalized proof.
1/4
chenyang.co/blog/automatic-t…
Po-Wei Wang retweeted
The dynamical system view gives very clean conditions for looped transformer to be stable
We’ve been thinking a lot about scaling laws, wondering if there is a more effective way to scale FLOPs without increasing parameters.
Turns out the answer is YES – by looping blocks of layers during training. We find that predictable scaling laws exist for layer looping, allowing us to use looping to achieve the quality of a Transformer twice the size.
Our scaling laws suggest that for a fixed parameter budget, data and looping should be increased in tandem!
🧵👇
Po-Wei Wang retweeted
1/ We found that deep sequence models memorize atomic facts "geometrically" -- not as an associative lookup table as often imagined.
This opens up practical questions on reasoning/memory/discovery, and also poses a theoretical "memorization puzzle."
Po-Wei Wang retweeted
(1/8)🍎A Galileo moment for LLM design🍎
As Pisa Tower experiment sparked modern physics, our controlled synthetic pretraining playground reveals LLM architectures' true limits. A turning point that might divide LLM research into "before" and "after." physics.allen-zhu.com/part-4…
ALT Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers. Joint work with Alberto Alfarano