Alex Speicher (@AlxSp_) profile picture
Joined October 2017
Alex Speicher retweeted
We’ve scaled @ZyphraAI’s subquadratic attention method - ZHA - to leading open source multi-trillion parameter models across Nemotron, Qwen, Mimo, and GPT-OSS! ZHA preserves reasoning and agentic performance while delivering 9-60x long-context speedups at 10M tokens.
4
16
7
185
53,923
Alex Speicher retweeted
We’ve received a lot of attention and questions about our ZHA release yesterday and how this compares to other sparse attention methods. Some thoughts below. 🧵
Today we’re introducing Zyphra Hierarchical Attention (ZHA), a path to true subquadratic attention. ZHA searches a learned tree of compressed summaries, with projected speedups of 35× over dense attention and 5× over MSA at 10M tokens, increasing to ~400× and ~55× at 1B. 🧵
4
14
1
100
11,752
Had a great time exploring how to make attention, one of the core mechanisms in LLMs, more efficient! Thanks to everyone who made this project happen: @leonlufkin, @cutterdawes, @kamesh_ai, @TomFigliolia, @BerenMillidge, and @ZyphraAI!
Today we’re introducing Zyphra Hierarchical Attention (ZHA), a path to true subquadratic attention. ZHA searches a learned tree of compressed summaries, with projected speedups of 35× over dense attention and 5× over MSA at 10M tokens, increasing to ~400× and ~55× at 1B. 🧵
1
11
437
I explored @diyerxx's Discrete Distribution Networks in @__tinygrad__, a generative approach using discrete nodes instead of diffusion noise. Blog + code: alexanderspeicher.com/blog/d… Paper: arxiv.org/abs/2401.00036
2
7
1
72
9,903
🚀 Excited to share the most inspiring work I’ve been part of this year: "Learning to Reason without External Rewards" TL;DR: We show that LLMs can learn complex reasoning without access to ground-truth answers, simply by optimizing their own internal sense of confidence. 1/n
92
500
145
3,469
583,238
Nous Research's RL Environments Hackathon recap thread! Starting with the stars of the show, the winners! Top 3 for the subjective track were: 1st - Pokemon Trainer - by @iyajainfinity & @AlexReibman 2nd - VR-CLImax by @JakeABoggs 3rd - DynastAI by David van Vliet and @SRacoon23 Top 3 for the objective track were: 1st - CyberMaxxing by @1999_karthik 2nd - HelpfulDoctors by @tsadpbb, Nilesh Shah, Max Phelps, and Alexander Speicher 3rd - Physical RL by @nullref0 and @venkatacrc Another special shout out to our partners, @xai, @MistralAI, @nvidia, @tensorstax, @akashnet, @nebiusai, @runpod, @daytonaio, @morph_labs, @LambdaAPI and @Tesla As well as our many judges from @arcee_ai, @axolotl_ai, @cursor_ai, @latentspacepod, @MIT, @togethercompute, @haizelabs, @SophontAI, @EdgeAGI, @Google, specifically: @AlpayAriyak, @winglian, Samuel Barry, @tmm1, @keirp1, @swyx, @teknium, @karan4d, Meghana Puvvadi, @arattml, @brianlechthaler, Josh May, Alex Gu, @gordic_aleksa, @AlpayAriyak, @eraqian, @LukePiette, Rohan Rao, @chargoddard, @LoganGrasby, @xennygrimmato_, @zhangir_azerbay, @rogershijin, @max_paperclips, @theemozilla, and Abhinav Balasubramanian
23
40
11
387
65,259
Alex Speicher retweeted
One of my PhD students got their visa revoked. I know of other cases amongst my AI colleagues. This is not what investing in US leadership in AI looks like.
This is flying a bit under the radar. But in terms of damage to America’s innovation and knowledge supremacy, the chilling effect of these revocations on the country’s ability to attract and retain scientific talent likely dwarfs the impact of tariffs or other policies.
38
159
14
1,050
313,087
Alex Speicher retweeted
Introducing *ARC‑AGI Without Pretraining* – ❌ No pretraining. ❌ No datasets. Just pure inference-time gradient descent on the target ARC-AGI puzzle itself, solving 20% of the evaluation set. 🧵 1/4
35
176
38
1,293
226,610
Prediction: one day, the idea of training human-readable reasoning traces will seem bizarre. (It's always seemed bizarre to me at least -- kinda like requiring a diffusion image model to output an image sequence that matches the strokes an artist would use.)
Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach We study a novel language model architecture that is capable of scaling test-time computation by implicitly reasoning in latent space. Our model works by iterating a recurrent block, thereby unrolling to arbitrary depth at test-time. This stands in contrast to mainstream reasoning models that scale up compute by producing more tokens. Unlike approaches based on chain-of-thought, our approach does not require any specialized training data, can work with small context windows, and can capture types of reasoning that are not easily represented in words. We scale a proof-of-concept model to 3.5 billion parameters and 800 billion tokens. We show that the resulting model can improve its performance on reasoning benchmarks, sometimes dramatically, up to a computation load equivalent to 50 billion parameters.
50
58
9
668
87,438
Today OpenAI announced o3, its next-gen reasoning model. We've worked with OpenAI to test it on ARC-AGI, and we believe it represents a significant breakthrough in getting AI to adapt to novel tasks. It scores 75.7% on the semi-private eval in low-compute mode (for $20 per task in compute ) and 87.5% in high-compute mode (thousands of $ per task). It's very expensive, but it's not just brute -- these capabilities are new territory and they demand serious scientific attention.
198
1,533
587
8,529
2,228,721