AI safety and efficiency; current: @matsprogram 11.0 w/ @openai | CS PhD @JohnsHopkins prev: @qualcomm, @adobe, @microsoft, @siemens
Baltimore
Joined November 2013
- Tweets293
- Following49
- Followers321
- Likes178
Everyday we get closer to
P + AI = NP
The complexity theory result is actually huge.
P problems can be solved in polynomial time. It was widely believed that NP-complete problems require exponential time.
P = NP states that NP-complete problems can be solved in polynomial time. OpenAI didn't prove that. But, they proved that (some) NP-complete problems can be solved faster than exponential time, meaning:
- P = NP, or
- the complexity of NP problems is a "middle thing" between polynomial and exponential, eg. O(n^log n)
If it survives peer reviews, this could be one of the biggest CS breakthroughs in decades.
The modern problem of potentially dealing with the same sabotage that you're trying to detect.
(Isak is my MATS mentor)
they gave the Claude interface double ticks.
pretty sure, someone is going to be left on read by Claude soon.
Why are we not making all AI safety-related discussions/work privately available only to humans? Kind of like private benchmark test sets.
We don't want to give models ideas, or tell them our lines of defenses through public-available training data.
teapaw.org/ApolloResearch/status/…
Looped LMs promise dynamic computation by allowing easy tokens to exit early, yet Ouro computes all loops anyway. We fixed it:
1. Ragged KV: −25% inference FLOPs & memory on Ouro, same accuracy, no retraining.
2. Global exit objective: true difficulty-aware exits 🧵
3/ Our fix --Ragged KV: every token's exit-depth KV is final; deeper reads alias to it. Skipped work stays skipped.
On Ouro-2.6B-Thinking(no training), 6 reasoning+code benchmarks: 3.0 vs 4.0 avg loops end to end (−25% KV & FLOPs), 62.0 vs 62.1 pass@1, 79.4 vs 78.0 pass@4.
To make it work in practice, we use page tables. Depth-batching (@raymin0223) would help fill idle GPU lanes, and turn the FLOPs savings into increased throughput.
cc-ing: teapaw.org/Grad62304977/status/20…
and
teapaw.org/eliebakouch/status/209…
a few thoughts on "recurrent depth" transformers
the main question: why recurrent depth instead of just scaling depth? it's not faster at inference or training* since you still go through the full "effective depth", the advantage is storage (for kv cache storage btw you could do kv sharing) and a few other things i tried to list below
1) could be better in undertrained regimes (too few data or compute constraints) but i don't think any frontier model is in this regime? could also help convergence somehow, maybe it's just not possible to train a 10T effective param model (so maybe 2Tx5) without tricks like this, but i doubt it..
2) one positive aspect could be better adaptive depth/compute (which is a good direction imo): depending on the token, you skip some recurrent blocks. but you could also just skip layers (or skip moe routing like in longcat flash) and get something quite similar
you could also imagine co-training models this way, sol with 8 blocks, terra with 4, luna with 2 or something like that
3) another interesting thing (to verify?): you could simulate higher batch size at inference by overlapping the computation of depth_block=1 of request 1 and depth_block=t of request 2 on the same gpus, so effective batch size = batch size * number of recurrent blocks (and you could even overlap across models if you do the co-training i mentioned above!), see the picture attached for an illustration of what i mean
still quite skeptical overall, i think scaling depth is just more expressive when you are not compute/data bound but excited to see a potential new direction
*on "faster to train", could be if you only backprop through the last few loops or something like that
6/ local exits barely vary, so Ragged KV ≈ full depth (0.11 PPL gap), hence Ouro works out of the box. Dynamic global exits open a gap (0.63).
Ragged-aware training closes it: alternate dense and ragged steps, same training FLOPs. Ragged PPL ends 0.23 below its own full depth.
Spent an amazing summer in Europe working @Qualcomm AI research, with top-quality collaborators: @arnau_padres, Victor Conchello Vendrell, Jordi Ros Giralt, @behboodiarash & my awesome mentor: @fvmassoli.
📄 arxiv.org/abs/2610.09013v1
Cursed scale returns
Introducing Beam: a highly efficient agentic open model with 501B total parameters and 23B active.
- Frontier reasoning efficiency
- Advances the Western open frontier on coding & agentic tasks
- Trained end-to-end from scratch
Full weights release this month.
Learn more about Beam: reflection.ai/beam
Oof
So you report pass @ 1:1 which is a flawed setup for zero-shot evaluations. The pass @ 1: 5 results confirm this. For example
Chemistry pass@1:1 pass@1:5
SFT 61.8 62.33
Your work 66.0 64.20
So your gains go down from 4.2 to 1.9 which can change the conclusions.
Also the GPQA results that you just provided seem to still come from a single run. GPQA has 198 questions, and you should expect whole numbers of correct answers. So the average of 5 runs should be some count of 198x5 = 990, which is clearly not the case here.
Apart from this, I checked on the MATH500 setup and 268/500 problem is also present in the training set, and you can check it yourself too (see the quoted tweet about it below) and let me know if there's a mistake here.
I'd strongly suggest, you re-run your evaluations, and come clean about these discrepancies. It's ok if the gains might not be as strong as you have expected but at least they reflect something real.
Regarding model selection, there's a Qwen2.5 pretraining contamination issue for MATH500 and AMC which is documented in at least two papers, Pretraining contamination(Wu et al.) and Spurious rewards(Shao et al.). I don't think the results in your paper can support any generalization for math due to these issues.
Also related to this, after checking your code, you basically build the training set by picking whatever the base model rates as most likely. So basically you are amplifying Qwen2.5's existing habits which is the exact same issue that the Spurious rewards paper showed for Qwen2.5 unrealistic RL gains.
the keyword is sample efficiency.
I wonder if this new RL from scratch is what David Silver’s ineffable intelligence pitched to investors.
Was hoping to get all infinity stones (NeurIPS, ICML and ICLR first-author papers) during my PhD. But couldn’t catch NeurIPS.
Funny that the rejected paper got the best response on X of all my papers.
NLAs are claimed to verbalize model activations. But can they faithfully interpret steered activations?
In our latest paper, we show that steering moves activations into non-invertible regions; and almost surely, no prompt maps to steered activations!
NLAs fail to interpret steered activation states faithfully, supporting our results! ↓
@anqi_liu33 @DanielKhashabi
teapaw.org/AnthropicAI/status/205…
With papers being optimized for AI reviews, gems will probably tend to not be found in prestigious conferences soon.
I never understood the around 30% acceptance rate across ML conferences.
With resubmit inflation, this is even more problematic. Why don’t chairs just accept papers on the basis of merit, not top percentile of their batch? I guess too much work.
