📚 Journal club on discrete diffusion models 🎥 Replays available on YouTube! Contact: [email protected] Hosted by @ssahoo_, @jdeschena, @zhihanyang_
Joined August 2025
- Tweets281
- Following0
- Followers2.4K
- Likes263
Discrete Diffusion Reading Group retweeted
Hot take: the community keeps asking whether diffusion LLMs can beat AR. That's the wrong question. Diffusion is a sampling method, and it can sit on top of an AR model to make it faster instead of replacing it.
Speculative decoding already speeds up AR, but it has two long-standing problems. Training a drafter takes a long time, and you have to build a complete pretraining corpus for it. Even then, token acceptance drops sharply on out-of-domain data.
Uno @ssahoo_ offers an elegant way out: it decouples sampling training from capability training. There's no separate drafter. The model itself, plus one LoRA, drafts several tokens in a single forward pass, and the LoRA learns to match the model's own next-token distribution, so there's no drafter corpus to build and no labels to collect.
We applied this to the Looped Transformer Nanbeige4.2-3B: 1 hour of LoRA training gives a lossless 2× speedup on agentic task. We haven't optimized anything yet, and we believe the acceleration can go much further.
The takeaway: diffusion doesn't need to beat AR. Any model that can score its next token can get a fast diffusion mode in an hour.
Page: fengsxy.github.io/LoopUno
Lora Adapters: huggingface.co/liddlefish/Lo…
Discrete Diffusion Reading Group retweeted
This is so cool! With less than an hour of training on a single A100 node, Uno delivered a 2x speedup on a looped model (Nanbeige) for agentic tasks.
In the coming weeks, we’ll see diffusion drafters like Uno replace traditional speculative decoding drafters across frontier models.
Great work, @Loy004Yu!
Hot take: the community keeps asking whether diffusion LLMs can beat AR. That's the wrong question. Diffusion is a sampling method, and it can sit on top of an AR model to make it faster instead of replacing it.
Speculative decoding already speeds up AR, but it has two long-standing problems. Training a drafter takes a long time, and you have to build a complete pretraining corpus for it. Even then, token acceptance drops sharply on out-of-domain data.
Uno @ssahoo_ offers an elegant way out: it decouples sampling training from capability training. There's no separate drafter. The model itself, plus one LoRA, drafts several tokens in a single forward pass, and the LoRA learns to match the model's own next-token distribution, so there's no drafter corpus to build and no labels to collect.
We applied this to the Looped Transformer Nanbeige4.2-3B: 1 hour of LoRA training gives a lossless 2× speedup on agentic task. We haven't optimized anything yet, and we believe the acceleration can go much further.
The takeaway: diffusion doesn't need to beat AR. Any model that can score its next token can get a fast diffusion mode in an hour.
Page: fengsxy.github.io/LoopUno
Lora Adapters: huggingface.co/liddlefish/Lo…
RT @JCJesseLai: 🎉 Excited to extend our book, The Principles of Diffusion Models, into a tutorial at @NeurIPSConf 2026!
This time, it’s al…
This quoted post is unavailable.
Discrete Diffusion Reading Group retweeted
🎊 Excited to give a tutorial on diffusion LLMs at @NeurIPSConf 2026 with @StefanoErmon @JCJesseLai @mittu1204
📆 Mon, Dec 7, 9 AM AEDT
📍 TikTok Ent Centre (Sydney, Australia)
Topics covered:
> A beginner friendly introduction to diffusion-LLMs
> Designing frontier diffusion-LLMs
> Using Continuous diffusion for LLMs
Discrete Diffusion Reading Group retweeted
Missed the presentation yesterday? We just uploaded the recording on YouTube: youtube.com/watch?v=J8rcc00l…
I especially tried to give some intuition for the sampler, which can look surprising on a first read!
This Monday, Oct 19, Justin Deschenaux (@jdeschena) will present his paper:
Simplex Diffusion Models
Discrete Diffusion Reading Group retweeted
Join our reading group next Monday!
Topic: Simplex Diffusion Models
Presenter: Justin Deschenaux
This Monday, Oct 19, Justin Deschenaux (@jdeschena) will present his paper:
Simplex Diffusion Models
Discrete Diffusion Reading Group retweeted
Really excited to see Uno-adapters incorporated into @nvidia -Nemo
Thanks @huiying_lii @krasul @Chonghan_Liu
Learn more about Uno: xitter.cf/ssahoo_/status/2095741…
We added Uno training to NeMo AutoModel.
The neat idea: let the model draft for itself, then verify with the original AR model — no separate draft model needed.
Our Qwen3-0.6B run gets ~2 committed tokens per forward on GSM8K and 1.84 on HumanEval.
PR: github.com/NVIDIA-NeMo/Autom…
cc @IFM_AI @ssahoo_
Thanks @krasul and @Chonghan_Liu for getting this in 🙏
Discrete Diffusion Reading Group retweeted
Join our reading group next Monday!
Topic: Gumbel Straight Flow: Distilling Autoregressive Models into One-Step Flow Maps
Presenter: Yeongmin Kim
Discrete Diffusion Reading Group retweeted
Excited to share my work during summer internship at NVIDIA! We build the first large-scale continuous dLLM, reaching comparable and even better performance compared with masked dLLM trained under aligned budgets, and discover interesting properties its continuous dynamics!
💫 Large Language Continuous Diffusion Models
Presenting Sigma, the first large-scale (3B/8B) continuous diffusion LM (dLM) trained with likelihood.
📈 Comparable with discrete dLMs and AR on math and coding
🎛️ Enables meaningful study of inference-time interventions
Jointly led with Wei Guo (@WeiGuo01), Jean-Marie Lemercier (@jm_lemercier), Simon Welker (@_cobalamin)
Team: @YongganFu @mmkamani7 Sajad Norouzi @julberner @tomasgeffner @karsten_kreis @YongxinChen1 @MoleiTaoMath @jwthickstun @PavloMolchanov Ante Jukic @ArashVahdat @MardaniMorteza
Discrete Diffusion Reading Group retweeted
💫 Large Language Continuous Diffusion Models
Presenting Sigma, the first large-scale (3B/8B) continuous diffusion LM (dLM) trained with likelihood.
📈 Comparable with discrete dLMs and AR on math and coding
🎛️ Enables meaningful study of inference-time interventions
Jointly led with Wei Guo (@WeiGuo01), Jean-Marie Lemercier (@jm_lemercier), Simon Welker (@_cobalamin)
Team: @YongganFu @mmkamani7 Sajad Norouzi @julberner @tomasgeffner @karsten_kreis @YongxinChen1 @MoleiTaoMath @jwthickstun @PavloMolchanov Ante Jukic @ArashVahdat @MardaniMorteza
Discrete Diffusion Reading Group retweeted
I’ll be speaking at the EDPM workshop about our recent paper:
“Unlocking Lossless Speedups in LLMs via Discrete Diffusion.”
📅 October 7, 10:50–11:30 AM
In NYC until October 7. Reach out if you’d like to meet up and talk shop!
Register for the Zoom Link: simonsfoundation.zoom.us/web…
Diffusion LLMs have two limitations relative to AR models:
(1) Lower quality, and
(2) Slower inference at large batch sizes.
We address this "Uno"
> Retains the AR architecture of LLMs
> Each layer has two sets of weights: AR weights and Diffusion weights
> Diffusion weights enable parallel sampling from the AR distribution losslessly
Results:
💥Faster than all speculative decoding methods: DFlash and EAGLE-3
🔥 Beats ALL diffusion LLMs: Mercury 2, Diffusion Gemma, Llada
Links to the paper, models, code below
Discrete Diffusion Reading Group retweeted
⭐️Continuous Diffusion Scales Competitively with Discrete Diffusion for Language
Has been accepted at NeurIPS 2026!
See you at the conference!
📢Excited to share our new paper:
Continuous Diffusion Scales Competitively with Discrete Diffusion for Language
We introduce RePlaid 🌊, a continuous diffusion language model (DLM) with
🏅Discrete likelihood bound
🏅Scaling laws competitive with SOTA discrete DLMs
How? Dive in👇[🧵1/12]
Paper: arxiv.org/abs/2605.18530
Work done with my amazing collaborators: @WeiGuo01 @ShuibaiZ69721 @ssahoo_ @YongxinChen1 @ArashVahdat @MardaniMorteza @jwthickstun
Discrete Diffusion Reading Group retweeted
Join our reading group next Monday!
Topic: DFlash: Block Diffusion for Flash Speculative Decoding
Presenter: Jian Chen (@jianchen1799)
📢Sep 21 (Mon): DFlash: Block Diffusion for Flash Speculative Decoding
🤔 Speculative decoding is meant to fix the sequential bottleneck in autoregressive LLMs — but the drafter is usually autoregressive too. The cure inherits the disease:
1️⃣ Drafting stays sequential, which caps the practical speedup
2️⃣ Diffusion LLMs can draft in parallel, but on their own they still underperform autoregressive models
🔑 The insight: speculative decoding is exactly the setting where a diffusion model shines. The target model verifies every draft, so the drafter's parallelism is pure upside — and the output stays lossless.
🔧 DFlash uses a lightweight block diffusion model as the drafter, with two pieces:
(1) Draft tokens are produced in a single forward pass — drafting stops being sequential
(2) The drafter is conditioned on context features extracted from the target model, which lifts draft quality and acceptance rates
📈 Over 6× lossless acceleration across a range of models and tasks — and up to 2.5× higher speedup than EAGLE-3, the state-of-the-art speculative decoding method.
This Monday, the lead author (@jianchen1799) will present DFlash.
Discrete Diffusion Reading Group retweeted
Today’s LLMs still write like typewriters: one token at a time. This sequential process creates a hard inference bottleneck.
We're introducing Uno, a diffusion-augmented LLM that delivers autoregressive quality at diffusion speed. It’s a lossless speedup method that accelerates generation without degrading response quality.
With Uno, K2-Horizon-7B outperforms state-of-the-art diffusion methods in both quality and throughput, delivering up to a 2.2× speedup with no loss in quality.
Paper: arxiv.org/abs/2609.04010
Model available at: huggingface.co/IFM/K2-Horizo…
Discrete Diffusion Reading Group retweeted
You should use diffusion adapters to speed up post-training.
It’s almost impossible to get speculative decoding to speed up RL, but with diffusion adapters, you can easily get into 30-40% speedup
Happy to chat more
Diffusion LLMs have two limitations relative to AR models:
(1) Lower quality, and
(2) Slower inference at large batch sizes.
We address this "Uno"
> Retains the AR architecture of LLMs
> Each layer has two sets of weights: AR weights and Diffusion weights
> Diffusion weights enable parallel sampling from the AR distribution losslessly
Results:
💥Faster than all speculative decoding methods: DFlash and EAGLE-3
🔥 Beats ALL diffusion LLMs: Mercury 2, Diffusion Gemma, Llada
Links to the paper, models, code below
Discrete Diffusion Reading Group retweeted
I’ll share new experimental results in this talk, including:
⚡️ Faster RL training
📊 Comparisons with a retrained DFlash model
📅 Sep 18, Friday (10am PT)
This week, @ssahoo_ presents diffusion-augmented LLMs:
🔥 AR-level Quality
⚡️ Diffusion-level speed
✨ A drop-in replacement for speculative decoding:
>> No separate draft model to train, plus
>> Faster RL post-training.
The method, Uno, beats all diffusion models by a wide margin in system throughput, agentic evals, and long-context reasoning.
📅 Sep 18, Friday
🕒 10am PT / 1PM ET
Discrete Diffusion Reading Group retweeted
Uno shows a simple way to speed up existing LLMs without changing their output distribution: keep the original model in charge, and use diffusion only to draft multiple tokens in parallel.
Keep the autoregressive model for quality and add diffusion for parallel drafting
On Qwen3-8B, Uno delivered 2.5X higher per-request throughput and 1.6X higher system throughput at the largest tested batch size.
Normal autoregressive LLMs generate 1 token at a time, making long responses, agents, and RL rollouts slow.
Uno adds lightweight diffusion adapters that propose several future tokens at once, while the original model verifies them.
That avoids a separate draft model and preserves the base model’s sampling behavior.
It also sped up end-to-end RL training by up to 40% in the reported runs.
Discrete Diffusion Reading Group retweeted
Thanks AK, for sharing our work.
Project page: s-sahoo.com/uno/
Github: github.com/ifm-ai/uno
HF: huggingface.co/collections/s…
Discrete Diffusion Reading Group retweeted
The 2B MDLM, Duo, Eso-LMs models are up on HF 🤗
(finally)
📢 New Paper: Scaling Beyond Masked Diffusion Language Models
Takeaways👇
🚫 Masked Diffusion ≠ endgame
🔥 Uniform-state beats AR on GSM8K
📉 Perplexity is a bad metric for comparing diffusion families
📰 arxiv.org/abs/2602.15014
🖥️ github.com/s-sahoo/scaling-d…
🌐 s-sahoo.com/scaling-dllms/
collaborators: @jm_lemercier @zhihanyang_ @jdeschena @Jingyu227 @jwthickstun Ante Jukic
(1/5)
