@MicrosoftAI | Prev Microsoft Research, PhD-ing CMU, Intern at Meta, Apple, AFRL, Amazon
NYC
Joined March 2024
- Tweets43
- Following515
- Followers207
- Likes131
Pinned Tweet
I'm always a fan of better information utilization in LLMs to improve efficiency/performance. So much potential in features of deeper layers and an efficient way to extract them.
Congrats to @xidulu and team on the amazing work!
1/ Sharing a new, interesting project I did during my internship at Microsoft AI Frontiers w/ @JohnCLangford
TL;DR: At decoding time, we feed **previous hidden state** into the input together with token embedding, and it boosts performance for free.
arxiv.org/abs/2608.08888
Harry Dong retweeted
Agents can often fix a mistake by retrying, then forget what they just learned! Next session, same mistake 🥲
Introducing Self-Direction: each failure becomes a small, procedural adapter the agent trains on its own reflection and reuses later 💪🏽
🌐 self-direction.github.io/
🧵1/n
Harry Dong retweeted
Banger paper from Microsoft and Cornell.
If you have looked at thinking tokens and decided you cannot afford the context, read this one.
(bookmark it)
A pause token buys the model extra compute for each next-token prediction, and it pays for that compute with a sequence position. Free pause tokens carry the same compute in a parallel prediction stream over a weight-shared backbone, riding an existing position instead of adding one.
At inference it adds nothing to context length, leaves the KV cache unchanged, and costs essentially no extra latency, since the additional flops are not the throughput bottleneck. On a 1B model it improves next-token prediction by 2 to 3 centinats.
The cost moves to training, where the overhead against an optimized pretraining pipeline comes down to about 1.14x while keeping most of the benefit.
Paper: arxiv.org/abs/2609.03807
Chat with Paper: academy.dair.ai/papers/free-…
Harry Dong retweeted
A quick fun paper: We ran State Prediction Separation arxiv.org/pdf/2607.01218 through an optimization process arxiv.org/abs/2609.03807 finding that it works at higher op and yielding "free pause tokens" which allow a transformer to make higher quality inference for (near) free.
Harry Dong retweeted
“Full-bandwidth Transformer”
Transformers usually pass just the sampled token into the next step, even though the model just computed a much richer hidden state.
This paper feeds that hidden state back in too, so the model can keep working on ideas internally instead of spelling everything out token by token.
At 1B scale, it improves math, coding, and LM evals with <1% extra decoding cost, and can match models trained on ~2x more data.
alphaxiv.org/abs/2608.08888
Harry Dong retweeted
TLDR: we integrate many computational and statistical improvements for muon-style updates here: github.com/microsoft/dion .
Comp: Gram Newton Schulz, Row Selection, Megabatching, Cudagraph Capture, Symmetric Kernels
Stat: Joint learning rate+fraction, row norm, fused rounding
Newton-Schulz is what makes Muon work, but it's also a massive pain compared to Adam. *Dion3* makes it ~6x faster and more parallelizable too.
Combines our work on row-filtered gradients, Gram-NS alg, and GPU kernels in one package. Try pip install dion!
arxiv.org/abs/2608.11612
Harry Dong retweeted
another very interesting paper with the shape I like of things looping somehow in latent space
1/ Sharing a new, interesting project I did during my internship at Microsoft AI Frontiers w/ @JohnCLangford
TL;DR: At decoding time, we feed **previous hidden state** into the input together with token embedding, and it boosts performance for free.
arxiv.org/abs/2608.08888
Harry Dong retweeted
Video generation models are improving fast—real-time autoregressive models now deliver high quality at low latency, and they’re quickly being adopted for world models and robotics applications. So what’s the problem? They’re still too slow on consumer hardware.
🚀 What if we told you that we can get true real-time 16 FPS video generation on a single RTX 5090? (1.5-12x over FA 2/3/4 on 5090, H100, B200)
Today we release MonarchRT 🦋, an efficient video attention that parameterizes attention maps as (tiled) Monarch matrices and delivers real E2E gains.
📄 Paper: arxiv.org/abs/2602.12271
🌐 Website: infini-ai-lab.github.io/Mona…
🔗 GitHub: github.com/Infini-AI-Lab/Mon…
🧵1/n
Harry Dong retweeted
RL is notoriously unstable under actor–policy mismatch 😥 — a common reality caused by kernel differences, MoE randomness, FP8 rollouts, or asynchronous pipelines.
But here’s a crazy thought 🤔
👉 What if you could RL-train a large model using rollouts generated only by a weaker, faster, and completely different model?
Sounds doomed from the start? 💩
We are releasing Jackpot 🎰.💡 enabling training Qwen3-8B-Base using only Qwen3-1.7B-Base generated rollouts
✨ Jackpot is surprisingly powerful:
• Enables cheap, fast rollouts to train stronger models
• Dramatically changes the cost–performance tradeoff of RL training
We release Jackpot 🎰 in the following format:
🌔Paper: arxiv.org/abs/2602.06107
🌕Code: github.com/Infini-AI-Lab/jac…
🌖Blog: infini-ai-lab.github.io/jpt_…
[1/n]
Very neat work led by @RJ_Sadhukhan to make LLMs more efficient, sparse, and interpretable!
Lookup memories are having a moment 😄
The whale 🐋 #deepseek dropped engram… and we dropped up-projections from our FFNs…perfect timing 😅
🥳 Introducing STEM: Scaling Transformers with Embedding Modules 🌱
A scalable way to boost parametric memory with extra perks:
✅ Stable training even at extreme sparsity
✅ Better quality for fewer training FLOPs (knowledge + reasoning + long-context gains)
✅ Efficient inference: ~33% FFN params removed + CPU offload & async prefetch
✅ More interpretable → seamless knowledge editing 🔧🧠
Looking forward to DeepSeek v4… feels like we’ve only scratched the surface of embedding-lookup scaling 👀
📄Paper: arxiv.org/abs/2601.10639
🌐 Website: infini-ai-lab.github.io/STEM
🔗 GitHub: github.com/Infini-AI-Lab/STE…
At NeurIPS all week! Swing by the Efficient Reasoning workshop at 10:45-11:00 on Saturday to hear my oral presentation about our work on interdependent sampling for parallel generation!
1/🧵
🎉Introducing Bridge🌉, our parallel LLM inference scaling method that shares info between all responses to an input prompt throughout the generation process! Bridge greatly improves the quality of individual responses and the entire response set!
📜arxiv.org/pdf/2510.01143
Harry Dong retweeted
This will be presented as an oral talk at NeurIPS Workshop Efficient Reasoning! Stop by to listen to Harry if you plan to attend NeurIPS. Harry is also on the job market this year! @Real_HDong
1/🧵
🎉Introducing Bridge🌉, our parallel LLM inference scaling method that shares info between all responses to an input prompt throughout the generation process! Bridge greatly improves the quality of individual responses and the entire response set!
📜arxiv.org/pdf/2510.01143
Harry Dong retweeted
Excited to release our new preprint - we introduce Adaptive Patch Transformers (APT), a method to speed up vision transformers by using multiple different patch sizes within the same image!
Harry Dong retweeted
New paper! I had a lot of fun working on scaling up RL with @agarwl_ @Devvrit_Khatri @louvishh and others. Check it out!
Harry Dong retweeted
🤔Can we train RL on LLMs with extremely stale data?
🚀Our latest study says YES!
Stale data can be as informative as on-policy data, unlocking more scalable, efficient asynchronous RL for LLMs.
We introduce M2PO, an off-policy RL algorithm that keeps training stable and performant even when using data stale by 256 model updates.
🔗 Notion Blog: m2po.notion.site/rl-stale-m2…
📄 Paper: arxiv.org/abs/2510.01161
💻 GitHub: github.com/Infini-AI-Lab/M2P…
🧵 1/4
1/🧵
🎉Introducing Bridge🌉, our parallel LLM inference scaling method that shares info between all responses to an input prompt throughout the generation process! Bridge greatly improves the quality of individual responses and the entire response set!
📜arxiv.org/pdf/2510.01143
8/🧵
✨Key takeaway: By treating LLM features for parallel scaling as a single tensor unit instead of independent slices, each response can give/take info to/from other responses to improve individual response AND response set quality while maintaining total parallelism.
9/🧵
More details in the full paper (Generalized Parallel Scaling with Interdependent Generations): arxiv.org/pdf/2510.01143
Big thank you to all my collaborators @brandfonbrener, @ErykHelenowski, Yun He, Mrinal Kumar, @Han_Fang_, @yuejiec, @karthikabinav