rishi (@rishiiyer01) header
rishi (@rishiiyer01) profile picture

squeezing water out of stone

Joined January 2017
Photo by rishi (@rishiiyer01)Photo by rishi (@rishiiyer01)Photo by rishi (@rishiiyer01)Photo by rishi (@rishiiyer01)Photo by rishi (@rishiiyer01)Photo by rishi (@rishiiyer01)Photo by rishi (@rishiiyer01)Photo by rishi (@rishiiyer01)Photo by rishi (@rishiiyer01)Photo by rishi (@rishiiyer01)
Go try ZONOS3-Mini out on SGL!
Zonos3-Mini from @ZyphraAI runs on SGLang-Omni on day 0! It’s an 800M MoE voice-cloning TTS model with 80M params active per token. On one H100, first audio streams back in 57 ms. Serve it today with SGLang-Omni: sgl-project.github.io/sglang…
1
4
19
875
rishi retweeted
At the moment H-Net seems to still scale worse than BPE as the compute scale increases from these small-scale experiments. The scaling laws crossover will happen at ~4e24 flops, well within modern pretraining budgets. A scaling ladder comparison would still have likely rejected such treatment. From using simple fitting of L = E + A·C^(−α) using data from the blog, BPE has a better exponent α at 0.1366 compared to H-Net's 0.1338. If we plot out the compute multipliers needed for baseline to match, you can see trend more clearly that the H-Net's lead shrinks (though with high uncertainty given the few data points). The effect is likely from either of: 1/ the flops of the BPE baseline get dominated by the last layer compute due to large vocab size; 2/ suboptimal vocab size for the BPE baseline ladder. It's a well-known effect that larger models scale better with smaller vocab. Nonetheless, H-Net is definitely worth more research!
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data. Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less? Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
5
10
156
15,980
rishi retweeted
Tinker’s Dimuon is Muon with delayed diagonal conditioning, Polar Express and a rank-dependent update norm. I recovered its update rule through the API, borrowing @mrtnm’s approach to StableReinforce. Here’s how 🧵
Dimuon is a new optimizer we built for LoRA fine-tuning. In our SFT tests, it often reaches lower loss in fewer steps than AdamW. We're still experimenting with Dimuon; you can Tinker with it today and share your feedback! See our optimizers guide at tinker-docs.thinkingmachines…
14
31
2
355
20,789
ecp edcp tsp are my favorites and missing 😔 could also do tdsp
I'm doing DP. I'm doing TP. I'm doing SP. I'm doing PP. I'm doing ZP. I'm doing FSDP. I'm doing 1F1B. I'm doing CP. I'm doing ETP. I'm doing DPA. I'm doing VPP. I'm doing P/D. I'm doing ZBPP. I'm doing DPA. I'm doing HSDP. I'm doing DCP. I'm doing DWDP. I'm doing AFD.
1
5
462
Traditional LLMs write one token at a time. Diffusion models don't. At the London PyTorch Meetup I gave an intro to text diffusion and how DiffusionGemma gets ~4x faster text generation. If you know LLMs but diffusion is a black box, this one's for you! youtu.be/3n5RN7i8h2Y
15
1
29
4,048
i guess the story that everyone in os compares to everyone in os once you are above 200B param count still holds true
5
463
As if we needed any more evidence that 2026 is the year of continuous diffusion for language🤭 (I recently wrote a blog post about my thoughts on this topic, in case you missed it: sander.ai/2026/08/24/continu…)
💫 Large Language Continuous Diffusion Models Presenting Sigma, the first large-scale (3B/8B) continuous diffusion LM (dLM) trained with likelihood. 📈 Comparable with discrete dLMs and AR on math and coding 🎛️ Enables meaningful study of inference-time interventions Jointly led with Wei Guo (@WeiGuo01), Jean-Marie Lemercier (@jm_lemercier), Simon Welker (@_cobalamin) Team: @YongganFu @mmkamani7 Sajad Norouzi @julberner @tomasgeffner @karsten_kreis @YongxinChen1 @MoleiTaoMath @jwthickstun @PavloMolchanov Ante Jukic @ArashVahdat @MardaniMorteza
6
24
233
15,867
On first pass this is amazing progress for continuous diffusion lms
💫 Large Language Continuous Diffusion Models Presenting Sigma, the first large-scale (3B/8B) continuous diffusion LM (dLM) trained with likelihood. 📈 Comparable with discrete dLMs and AR on math and coding 🎛️ Enables meaningful study of inference-time interventions Jointly led with Wei Guo (@WeiGuo01), Jean-Marie Lemercier (@jm_lemercier), Simon Welker (@_cobalamin) Team: @YongganFu @mmkamani7 Sajad Norouzi @julberner @tomasgeffner @karsten_kreis @YongxinChen1 @MoleiTaoMath @jwthickstun @PavloMolchanov Ante Jukic @ArashVahdat @MardaniMorteza
1
1
33
2,298
💫 Large Language Continuous Diffusion Models Presenting Sigma, the first large-scale (3B/8B) continuous diffusion LM (dLM) trained with likelihood. 📈 Comparable with discrete dLMs and AR on math and coding 🎛️ Enables meaningful study of inference-time interventions Jointly led with Wei Guo (@WeiGuo01), Jean-Marie Lemercier (@jm_lemercier), Simon Welker (@_cobalamin) Team: @YongganFu @mmkamani7 Sajad Norouzi @julberner @tomasgeffner @karsten_kreis @YongxinChen1 @MoleiTaoMath @jwthickstun @PavloMolchanov Ante Jukic @ArashVahdat @MardaniMorteza
17
110
22
698
81,617
🔥Welcome 🔺Simplex Diffusion Models🔻! Standard discrete diffusion models suffer from "information collapse" because they discard uncertainty at intermediate steps by sampling categorical tokens. (1/5)
16
106
12
849
98,566
Excited to share my new result on unbounded Holevo additivity gaps, a long-sought open question since Hastings’ seminal discovery on addivitiy violation. We identify a sequence of channels where Holevo $\chi$ diminishes while classical capacity diverges. Surprisingly, the construction was hiding in plain sight: it simply takes a fresh look at earlier work by Benoît Collins et al. bridging random channel violations and free probability theory (, which is also a subject of my favorites, and more works coming soon, stay tuned!). arxiv.org/abs/2609.18222
2
3
138
i cant wait to release paper(s) (s00n) so i can release my unhinged drafts i keep them in the vault just so the snr of my twitter is preserved to some extent
1
1
21
603
i like to occupy the lower singular values that get boosted by ns anyway
1
5
226
-the second instance of extremely scaled engram -enc dec in this fashion has really good disagg properties, it seems as though they went in on optimizing for prefill efficiency with this arch, not sure if it will actually outperform not doing this type of enc dec -this is one of the most impressive archs I will have to think more on how much of this is needed under modern racks that we have available
Deepseek V4.1 Flash 552B total, 8/16B active with a new arch trained on 45T tokens, there are different active parameters for input/output tokens with the encoder/decoder arch, engram, new sparse attention, new mHC, native vision very high benchmarks (beating K3), insane efficiency, and as always amazing tech report this is probably the most novel arch i've seen in a while, pretty insane
2
1
27
2,244
Sacrifice everything for performance engineering
1
3
178
This month, I reverse-engineered Jane Street's ASIC puzzle from its chip layout. I reconstructed the circuit, solved the hidden 11 x 11 Star Battle, recovered `(* TWO STARS *)` and found @JaneStreetGroup's spiral logo drawn into the metal
1
2
11
251
rishi retweeted
Replying to @adavya_sharma
s00n
1
136
Additionally the throughput chart on the left of the original paper is incredibly misleading. The arbitrary choice of measuring throughput on a single h200 with much larger models in competition is unfair. Additionally the toks/nfe are quite low, likely due to lack of training tokens in the diffusion conversion phase.
Seems like interesting work, but "Beats ALL diffusion LLMs" is simply not true. Latency-quality is comfortably Pareto dominated by both Mercury 2 and DiffusionGemma. (I can only post the Uno-Qwen point because strangely the LCBv6 number for Uno is missing from the writeup.....)
1
1
27
2,523
How can your abstract ignore years of research? It has been repeatedly proven that SWA is much worse than linear attention when trained from scratch. See app. A of arxiv.org/pdf/2509.24552 or any other recent linear attention paper's SWA baseline. Is this just an honest mistake?
1
5
121
9,179
this paper does not deserve the time of your day.
unavailable
2
128
39,281