Pinned Tweet
Replying to @ZyphraAI
This project was a lot of fun! There is so much structure in these models, waiting to be understood and used.
Many thanks to my amazing collaborators @cutterdawes, @AlxSp_, @kamesh_ai, and to @TomFigliolia, @BerenMillidge, and @ZyphraAI for their support.
Leon retweeted
We scaled Zyphra Hierarchical Attention to 2.4 trillion parameters. Across four open-weight models, we kept the backbones frozen and preserved performance on most reasoning, coding, and long-context benchmarks.
We’ve scaled @ZyphraAI’s subquadratic attention method - ZHA - to leading open source multi-trillion parameter models across Nemotron, Qwen, Mimo, and GPT-OSS!
ZHA preserves reasoning and agentic performance while delivering 9-60x long-context speedups at 10M tokens.
ZHA is growing up!
We’ve scaled @ZyphraAI’s subquadratic attention method - ZHA - to leading open source multi-trillion parameter models across Nemotron, Qwen, Mimo, and GPT-OSS!
ZHA preserves reasoning and agentic performance while delivering 9-60x long-context speedups at 10M tokens.
Leon retweeted
ZHA really does scale to the frontier! Fantastic work again from the team. It's shocking how fast and cheap the distillation is -- the evals are way more expensive to run than the full training process!
We’ve scaled @ZyphraAI’s subquadratic attention method - ZHA - to leading open source multi-trillion parameter models across Nemotron, Qwen, Mimo, and GPT-OSS!
ZHA preserves reasoning and agentic performance while delivering 9-60x long-context speedups at 10M tokens.
We’ve scaled @ZyphraAI’s subquadratic attention method - ZHA - to leading open source multi-trillion parameter models across Nemotron, Qwen, Mimo, and GPT-OSS!
ZHA preserves reasoning and agentic performance while delivering 9-60x long-context speedups at 10M tokens.
Leon retweeted
Zyphra's new paper proposes Zyphra Hierarchical Attention (ZHA), which organizes the KV-cache into a tree of compressed summaries. Queries search the tree from top to bottom, narrowing down which tokens deserve attention.
With just 1B tokens of additional distillation training, ZHA preserves most of the performance of full attention on models up to 35B parameters. On RULER at 256K context, it achieves 93.3% vs 96% accuracy, while attending to just 1-6% of the KV-cache.
alphaxiv.org/abs/2610.42112
Introducing Zyphra FlashAttention (ZYFA): faster, more accurate 8-bit floating-point (FP8) attention on AMD MI355X.
Our prefill kernels deliver 1.16–1.76x the throughput of AMD’s inference kernel library, AITER, on real model inputs at 16K tokens under sustained load. 🧵
We’ve received a lot of attention and questions about our ZHA release yesterday and how this compares to other sparse attention methods. Some thoughts below. 🧵
Along with the development of efficient kernels for ZHA’s wide use, we plan to keep improving ZHA by incorporating new insights. It’s very exciting to see the unique contribution of each new paper, and we hope that ZHA is a worthy contribution itself!
1
6
381
And thanks again to my amazing team @cutterdawes, @AlxSp_, @kamesh_ai, @TomFigliolia, and @BerenMillidge at @ZyphraAI!
5
410
Leon retweeted
As in the brain, "ultra-long contexts may require subquadratic attention. Attend to everything, and the signal gets lost in noise. Too sparse, and we lose the signal. Hierarchical attention may be essential not just for efficiency, but for attention to work at all." — Zyphra
Today we’re introducing Zyphra Hierarchical Attention (ZHA), a path to true subquadratic attention.
ZHA searches a learned tree of compressed summaries, with projected speedups of 35× over dense attention and 5× over MSA at 10M tokens, increasing to ~400× and ~55× at 1B. 🧵
Today we're releasing ZAYA1-8B, a reasoning MoE trained on @AMD and optimized for intelligence density.
With <1B active params, it outperforms open-weight models many times its size on math and reasoning, closing in on DeepSeek-V3.2 and GPT-5-High with test-time compute. 🧵











