We’ve scaled @ZyphraAI’s subquadratic attention method - ZHA - to leading open source multi-trillion parameter models across Nemotron, Qwen, Mimo, and GPT-OSS!
ZHA preserves reasoning and agentic performance while delivering 9-60x long-context speedups at 10M tokens.
Alex Speicher retweeted
We’ve received a lot of attention and questions about our ZHA release yesterday and how this compares to other sparse attention methods. Some thoughts below. 🧵
Had a great time exploring how to make attention, one of the core mechanisms in LLMs, more efficient!
Thanks to everyone who made this project happen: @leonlufkin, @cutterdawes, @kamesh_ai, @TomFigliolia, @BerenMillidge, and @ZyphraAI!
I annotated and reimplemented Norm-Agnostic Residual Networks (NAG), by Tomas Figliolia & @BerenMillidge at @ZyphraAI, and reproduced its core geometry: controlled norm growth, steadier rotation with depth, and fewer attention sinks.
Blog: alexanderspeicher.com/blog/n…
I explored @diyerxx's Discrete Distribution Networks in @__tinygrad__, a generative approach using discrete nodes instead of diffusion noise. Blog + code: alexanderspeicher.com/blog/d…
Paper: arxiv.org/abs/2401.00036
Alex Speicher retweeted
🚀 Excited to share the most inspiring work I’ve been part of this year:
"Learning to Reason without External Rewards"
TL;DR: We show that LLMs can learn complex reasoning without access to ground-truth answers, simply by optimizing their own internal sense of confidence. 1/n
Alex Speicher retweeted
Nous Research's RL Environments Hackathon recap thread!
Starting with the stars of the show, the winners!
Top 3 for the subjective track were:
1st - Pokemon Trainer - by @iyajainfinity & @AlexReibman
2nd - VR-CLImax by @JakeABoggs
3rd - DynastAI by David van Vliet and @SRacoon23
Top 3 for the objective track were:
1st - CyberMaxxing by @1999_karthik
2nd - HelpfulDoctors by @tsadpbb, Nilesh Shah, Max Phelps, and Alexander Speicher
3rd - Physical RL by @nullref0 and @venkatacrc
Another special shout out to our partners, @xai, @MistralAI, @nvidia, @tensorstax, @akashnet, @nebiusai, @runpod, @daytonaio, @morph_labs, @LambdaAPI and @Tesla
As well as our many judges from @arcee_ai, @axolotl_ai, @cursor_ai, @latentspacepod, @MIT, @togethercompute, @haizelabs, @SophontAI, @EdgeAGI, @Google, specifically:
@AlpayAriyak, @winglian, Samuel Barry, @tmm1, @keirp1, @swyx, @teknium, @karan4d, Meghana Puvvadi, @arattml, @brianlechthaler, Josh May, Alex Gu, @gordic_aleksa, @AlpayAriyak, @eraqian, @LukePiette, Rohan Rao, @chargoddard, @LoganGrasby, @xennygrimmato_, @zhangir_azerbay, @rogershijin, @max_paperclips, @theemozilla, and Abhinav Balasubramanian
Alex Speicher retweeted
One of my PhD students got their visa revoked. I know of other cases amongst my AI colleagues. This is not what investing in US leadership in AI looks like.
Alex Speicher retweeted
Introducing *ARC‑AGI Without Pretraining* – ❌ No pretraining. ❌ No datasets. Just pure inference-time gradient descent on the target ARC-AGI puzzle itself, solving 20% of the evaluation set. 🧵 1/4
Alex Speicher retweeted
Prediction: one day, the idea of training human-readable reasoning traces will seem bizarre.
(It's always seemed bizarre to me at least -- kinda like requiring a diffusion image model to output an image sequence that matches the strokes an artist would use.)
Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
We study a novel language model architecture
that is capable of scaling test-time computation by
implicitly reasoning in latent space. Our model works by iterating a recurrent block, thereby unrolling to arbitrary depth at test-time. This stands
in contrast to mainstream reasoning models that scale up compute by producing more tokens. Unlike approaches based on chain-of-thought, our
approach does not require any specialized training data, can work with small context windows,
and can capture types of reasoning that are not
easily represented in words. We scale a proof-of-concept model to 3.5 billion parameters and 800
billion tokens. We show that the resulting model
can improve its performance on reasoning benchmarks, sometimes dramatically, up to a computation load equivalent to 50 billion parameters.
Alex Speicher retweeted
Today OpenAI announced o3, its next-gen reasoning model. We've worked with OpenAI to test it on ARC-AGI, and we believe it represents a significant breakthrough in getting AI to adapt to novel tasks.
It scores 75.7% on the semi-private eval in low-compute mode (for $20 per task in compute ) and 87.5% in high-compute mode (thousands of $ per task). It's very expensive, but it's not just brute -- these capabilities are new territory and they demand serious scientific attention.



















