Ollie Jaffe (@JaffeOliver) profile picture

{eval, safety}/acc

Joined October 2024
Photo by Ollie Jaffe (@JaffeOliver)Photo by Ollie Jaffe (@JaffeOliver)Photo by Ollie Jaffe (@JaffeOliver)
the gap is small for an RSI-relevant capability
We evaluated Kimi K3.1 Pro on TasteVal, and found that it has an experiment compute multiplier of 0.78x relative to our human experts, which puts it roughly 4.6 months behind the frontier and on par with Opus 4.8. We fit a separate trend to leading open-weight models. On average, these models lag the frontier by about 4.2 months, and their doubling time is similar to that of frontier models.
1
8
306
Ollie Jaffe retweeted
OpenAI fired me last week, along with two of my safety colleagues. I was given one reason: that I accessed an executive's email. I want to say this plainly, because too many of OpenAI’s history is smoke and mirrors when people disappear:
Two other safety researchers and I were fired from OpenAI last week. We wrote this letter to leadership. I believe we were fired for prioritizing safety over the near-term interests of OpenAI as a corporation.
343
1,330
203
9,781
1,872,146
Ollie Jaffe retweeted
Two other safety researchers and I were fired from OpenAI last week. We wrote this letter to leadership. I believe we were fired for prioritizing safety over the near-term interests of OpenAI as a corporation.
230
860
271
5,680
2,122,264
Ollie Jaffe retweeted
We are releasing the technical report for Limite 1B -Violetto. Violetto is our first model, designed to push mathematical capabilities per parameter to its limits, trained from scratch.
15
90
29
589
65,834
Proud to announce TasteVal: a benchmark measuring experimental research taste, evaluating models from GPT-3.5 to Opus 5.5. We find frontier models now have higher experimental research taste on our tasks than our human experts
How fast is AI's research taste improving? We find that the experimental research taste of frontier models has doubled every ~3 months since December 2025. The best model, Opus 5.5, now exceeds our expert human baseline. Our human experts are experienced researchers, but most haven’t worked at a frontier lab. Why measure research taste? In the AI Futures Model, it largely determines how quickly artificial superintelligence is reached once coding is fully automated.
4
3
61
3,321
We’ll be coming out with a v1.1 update to TasteVal in a few weeks. We have some final tasks and human baselines that are almost done. We designed the benchmark to be insanely challenging to saturate, so I expect we'll be maintaining it for a while
1
4
136
We're hiring! If you're keen to forecast AI R&D, understand its risks, and think about how to safely pace progress, you should think about joining! See roles here: jobs.ashbyhq.com/pzeroresear…
3
130
"~10% of ICLR 2025 papers have (significantly) incorrect results" is a fun paper begging to be written: 1. Scrape all ICLR codebases for papers (+check they reproduce paper) 2. Use CC/codex to automatically review+fix+re-run I researched this in Jan but no capacity to finish
1
3
244
Autonomous orgs/sub-teams are right around the corner - shocking diff in perf of multi-agent vs. best-of-N single-agent on >100h tasks. Just put multiple CC in the same slack Feels like there is massive underelicitation across many evals I would worry if your moat is SWE
5
263
GLM-4.6 performs about the same as SWE-1.5 on SWE-Bench Pro. I evaluated GLM-4.6 myself this morning, just giving it simple bash tools. If the rumours are true and SWE-1.5 is a descendant of GLM-4.6, this is embarrassing for @cognition.
Today we’re releasing SWE-1.5, our fast agent model. It achieves near-SOTA coding performance while setting a new standard for speed. Now available in @windsurf.
2
4
1,283
The SWE-1.5 release feels suspicious. Why did they only release scores on a single eval? Why didn't they report confidence intervals? Why didn't they disclose the model they finetuned from? And why does GLM-4.6 even outperform gpt-5-high on this benchmark?
1
2
310
Setup for evaluating GLM4.6. Reasoning enabled. 300 randomly selected tasks (verified this is representative of using full set). Tools for executing arbitrary bash commands, viewing files, editing files. Only one seed because that's what SWE-1.5 authors seemingly did :)
1
197