Leon (@leonlufkin) header
Leon (@leonlufkin) profile picture

Stats PhD @Stanford MTS @ZyphraAI

San Francisco
Joined September 2021
Photo by Leon (@leonlufkin)Photo by Leon (@leonlufkin)Photo by Leon (@leonlufkin)Photo by Leon (@leonlufkin)Photo by Leon (@leonlufkin)Photo by Leon (@leonlufkin)
Replying to @ZyphraAI
This project was a lot of fun! There is so much structure in these models, waiting to be understood and used. Many thanks to my amazing collaborators @cutterdawes, @AlxSp_, @kamesh_ai, and to @TomFigliolia, @BerenMillidge, and @ZyphraAI for their support.
2
5
29
2,086
We scaled Zyphra Hierarchical Attention to 2.4 trillion parameters. Across four open-weight models, we kept the backbones frozen and preserved performance on most reasoning, coding, and long-context benchmarks.
We’ve scaled @ZyphraAI’s subquadratic attention method - ZHA - to leading open source multi-trillion parameter models across Nemotron, Qwen, Mimo, and GPT-OSS! ZHA preserves reasoning and agentic performance while delivering 9-60x long-context speedups at 10M tokens.
1
3
15
699
ZHA is growing up!
We’ve scaled @ZyphraAI’s subquadratic attention method - ZHA - to leading open source multi-trillion parameter models across Nemotron, Qwen, Mimo, and GPT-OSS! ZHA preserves reasoning and agentic performance while delivering 9-60x long-context speedups at 10M tokens.
1
4
11
419
ZHA really does scale to the frontier! Fantastic work again from the team. It's shocking how fast and cheap the distillation is -- the evals are way more expensive to run than the full training process!
We’ve scaled @ZyphraAI’s subquadratic attention method - ZHA - to leading open source multi-trillion parameter models across Nemotron, Qwen, Mimo, and GPT-OSS! ZHA preserves reasoning and agentic performance while delivering 9-60x long-context speedups at 10M tokens.
3
4
33
1,561
Leon retweeted
We’ve scaled @ZyphraAI’s subquadratic attention method - ZHA - to leading open source multi-trillion parameter models across Nemotron, Qwen, Mimo, and GPT-OSS! ZHA preserves reasoning and agentic performance while delivering 9-60x long-context speedups at 10M tokens.
4
16
7
185
53,921
Zyphra's new paper proposes Zyphra Hierarchical Attention (ZHA), which organizes the KV-cache into a tree of compressed summaries. Queries search the tree from top to bottom, narrowing down which tokens deserve attention. With just 1B tokens of additional distillation training, ZHA preserves most of the performance of full attention on models up to 35B parameters. On RULER at 256K context, it achieves 93.3% vs 96% accuracy, while attending to just 1-6% of the KV-cache. alphaxiv.org/abs/2610.42112
9
28
2
212
10,090
Leon retweeted
Introducing Zyphra FlashAttention (ZYFA): faster, more accurate 8-bit floating-point (FP8) attention on AMD MI355X. Our prefill kernels deliver 1.16–1.76x the throughput of AMD’s inference kernel library, AITER, on real model inputs at 16K tokens under sustained load. 🧵
8
22
8
135
82,134
We’ve received a lot of attention and questions about our ZHA release yesterday and how this compares to other sparse attention methods. Some thoughts below. 🧵
Today we’re introducing Zyphra Hierarchical Attention (ZHA), a path to true subquadratic attention. ZHA searches a learned tree of compressed summaries, with projected speedups of 35× over dense attention and 5× over MSA at 10M tokens, increasing to ~400× and ~55× at 1B. 🧵
4
14
1
100
11,752
Along with the development of efficient kernels for ZHA’s wide use, we plan to keep improving ZHA by incorporating new insights. It’s very exciting to see the unique contribution of each new paper, and we hope that ZHA is a worthy contribution itself!
1
6
381
And thanks again to my amazing team @cutterdawes, @AlxSp_, @kamesh_ai, @TomFigliolia, and @BerenMillidge at @ZyphraAI!
5
410
As in the brain, "ultra-long contexts may require subquadratic attention. Attend to everything, and the signal gets lost in noise. Too sparse, and we lose the signal. Hierarchical attention may be essential not just for efficiency, but for attention to work at all." — Zyphra
Today we’re introducing Zyphra Hierarchical Attention (ZHA), a path to true subquadratic attention. ZHA searches a learned tree of compressed summaries, with projected speedups of 35× over dense attention and 5× over MSA at 10M tokens, increasing to ~400× and ~55× at 1B. 🧵
5
8
92
7,667
Leon retweeted
Today we’re introducing Zyphra Hierarchical Attention (ZHA), a path to true subquadratic attention. ZHA searches a learned tree of compressed summaries, with projected speedups of 35× over dense attention and 5× over MSA at 10M tokens, increasing to ~400× and ~55× at 1B. 🧵
44
147
54
2,493
245,595
Leon retweeted
Today we're releasing ZAYA1-8B, a reasoning MoE trained on @AMD and optimized for intelligence density. With <1B active params, it outperforms open-weight models many times its size on math and reasoning, closing in on DeepSeek-V3.2 and GPT-5-High with test-time compute. 🧵
98
280
127
2,417
1,268,658