JunShern (@junshernchan) profile picture

Trying to make AI go well @AnthropicAI. Previously @OpenAI, @CHAI_Berkeley, @nyuniversity, autonomous vehicles @motionaldrive. 🇲🇾

Joined March 2018
Photo by JunShern (@junshernchan)Photo by JunShern (@junshernchan)Photo by JunShern (@junshernchan)Photo by JunShern (@junshernchan)Photo by JunShern (@junshernchan)Photo by JunShern (@junshernchan)Photo by JunShern (@junshernchan)Photo by JunShern (@junshernchan)Photo by JunShern (@junshernchan)Photo by JunShern (@junshernchan)
Wake up babe, new agent benchmark dropped! MLE-bench - or, as I like to say it, EMILY BENCH 🙆🏻‍♀️. Main summary on Neil’s thread, but adding some highlights of my own:
Proud to introduce MLE-bench: A benchmark of 75 real-life Kaggle competitions to test AI agents on ML engineering! When will we have our first AI Kaggle Grandmaster? 🥇🥈🥉
1
14
3
106
21,033
JunShern retweeted
Anthropic shared extensive evidence across its AI R&D pipeline, from pretraining to RL, including key details like estimates of algorithmic progress over time. We found this nonpublic evidence highly informative about AI R&D acceleration.
4
19
6
589
143,023
JunShern retweeted
This report shares our three subjective conclusions about risk: 1. We believe that, as of Oct 2026, the pace of capability growth on well-specified tasks is likely about 2x what it was from Mar 2024 to Oct 2025. We’re highly uncertain and would not be surprised by 1.3x–3x.
8
122
55
1,307
321,154
JunShern retweeted
We’re committing a portion of Anthropic’s shares to the Anthropic Public Benefit Fund, a new wholly owned entity dedicated entirely to the public benefit. It will start off with 5% of our shares. For every 50% increase in the company’s value over a set baseline, we can contribute up to an extra 2%, subject to our board’s discretion. In total, this means that roughly 20% of Anthropic’s shares are committed to the public good. Our co-founders have already pledged to donate 80% of their equity, and around half our employees are donating a portion of their equity as part of a donation matching program we’ve run since our founding. In the post below, we share more details about our new fund. anthropic.com/news/public-be…
2,293
1,744
1,688
21,519
7,808,871
JunShern retweeted
Remember that math is just the first domain of many to be exploded and transformed by a totally unprecedented 1000x-ing of cognitive labor.
We’re releasing a broad range of new mathematical results produced by an internal frontier model. We’ve been consulting with the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, and we have drawn on their advice and public recommendations to inform how we release these results. github.com/openai/math
10
13
1
194
5,384
ECI and its components ECI treats each benchmark as having a logistic distribution, so it's linear in log-odds. This implies if you plot log-odds of each benchmark the lines should all be increasing together, though each has a different x-position (difficulty) and different dispersion (slope).
5
4
2
86
4,827
Great work from @JaffeOliver and @danesherbs ! Clever setup and big results measuring "experimental research taste", which isolates experimental design efficiency from coding capability.
How fast is AI's research taste improving? We find that the experimental research taste of frontier models has doubled every ~3 months since December 2025. The best model, Opus 5.5, now exceeds our expert human baseline. Our human experts are experienced researchers, but most haven’t worked at a frontier lab. Why measure research taste? In the AI Futures Model, it largely determines how quickly artificial superintelligence is reached once coding is fully automated.
1
6
132
Imo this doesn't cover taste in the sense of conceptual or fuzzy research tasks, but depending on how much you think frontier AI development is just about optimizing a known setup, this may be more or less be the relevant indicator for AI R&D capability.
1
4
43
JunShern retweeted
I think now is an excellent time for many people to do independent research (in academia and outside academia) on the societal impacts of advanced AI. I am less certain that the historic format of a PhD is the right form for many today. But doing a "PhD" today with an adviser who is open-minded on how to define and achieve high-impact research is a great idea! With that said, I want to comment on something Phillip addresses quickly (in a way that I agree with). I am increasingly frustrated by many researchers (in CS working on AI or in other areas affected by AI) who are pessimistic about the "specter" of AI affecting their research. I personally find this deeply unscientific even if an understandable reaction to disruption. Today's AI can do incredible things. Next year's AI will do even more incredible things. As researchers, yes, that might affect *what* you do. But, to an extent unlike almost all other professions, we have great autonomy over the problems we work on. Most of the problems we work on continue to remain problems. We have a lot to do. Whether individual researchers use AI or not, we should be able to respond to technological progress with renewed commitment to making progress on pushing the frontiers of knowledge and understanding. Not fear, nor pearl-clutching over certain forms of human "expertise".
Students sometimes ask me if it still makes sense, in this accelerating age, to pursue a PhD in AI. Perhaps counterintuitively, I think it's a great time to do so. I wrote up some thoughts on this here: web.mit.edu/phillipi/www/wri…
3
7
113
10,029
JunShern retweeted
This post (from one of the independent investigators) is the best short thing I've seen on the new & crazy stuff the OpenAI-Hugging Face investigation found. Full post in screenshots. Wild that this is the level of crazy that was uncovered by an extremely limited-scope initial investigation of what happened in a single 6-day window. Imagine what would be turned up by a real investigation of root causes, organizational processes & culture, and everything else that goes into an incident investigation when you're serious about it. Screenshot batch 1/2:
New post: going into our investigation of the HF attack (before Black Hat), I was very wrong about what basically happened. This incident was far more serious than I expected, and far more serious than previous documented misalignment incidents. planned-obsolescence.org/p/t…
31
114
10
786
92,206
The performance of AI on some fixed set of tasks you consider important is very likely to accelerate, and this has nothing to do with progress in AI accelerating - it's just not possible to go from flat at 0% of that set to flat at 100% without there being apparent acceleration at some point. Typically that acceleration will look like a sigmoid - meaning you'll see continous acceleration at first. This does not mean AI is actually getting faster.
1
2
43
3,028
Excited to finally be able to tweet this: we're looking for a world class visionary to lead @AnthropicAI's cyberdefense research mission. Leading Glasswing and our cyberdefense mission this year has been an honor. It's now clear we need to scale up our ambition to help defend the world. Done well, Glasswing and Mythos will look like a small first step. Claude is + will be one of the world's best security researchers -- what should we do with that? What should we research? We're looking for a visionary, senior lead -- someone that can see past normal cybersecurity into the model-driven world beyond it. This is a senior position at Anthropic, and this person is a unicorn. You will have a team of already world-class researchers that produced the work on Mythos vulns & exploits, n-days, attacks on networks, and our cyber evals. We have started working on new research we'll share soon. This will be a massive lever on how AGI goes. Cyber will probably continue to drive the AI security and policy conversation. And we need cybersecurity in a world of extremely powerful models. Link below.
31
46
9
469
86,115
We are starting a new, nonprofit alignment organization, ⊢ Sequent Research, bringing together researchers previously on UK AISI’s Alignment Team, Timaeus, and elsewhere to research how to align superintelligence. We are hiring! 🧵
28
145
48
1,034
296,519
JunShern retweeted
Our internal data shows Claude is accelerating AI development—a possible path to recursive self-improvement, or AI autonomously building a more capable successor. It’s happening faster than we thought, and the implications deserve greater attention. anthropic.com/institute/recu…
1,708
4,502
2,991
28,172
18,969,052
With the help of Claude Mythos Preview, the Firefox team fixed more security bugs in April than in the past 15 months combined.
334
1,222
314
15,188
1,501,780
JunShern retweeted
at the intersection of i've basically been replaced and i've never worked so hard in my life
26
116
11
2,016
77,425
Back in ~November, our team picked a stretch goal of seeing if we could find and fix vulnerabilities in Firefox with Opus 4.6. In 2 weeks, we found 22, and ~1/5th of all high severity CVEs in a year. For our team, this feels like a rubicon moment.
17
50
3
351
34,020
JunShern retweeted
Nicholas Carlini at [un]prompted. If you know Carlini, you know this is a startling claim.
19
140
25
1,321
197,241
JunShern retweeted
The increasingly-hyperbolic METR graph is actually good news for safety. We just have to survive a brief singularity in March, and then afterwards the models will never be able to do more than undo a few hours' worth of work
25
70
11
1,636
69,709
JunShern retweeted
Across all mini-SWE-agent + <model> runs, SWE-bench Verified's current "ceiling"? - 87.4% (0.874 - 0.8) * 500 = another *37* instances that aren't solved consistently. If you recalculate this number across all official SWE-bench Verified submissions? - 95% from SWE-bench site
6
9
3
49
21,768
JunShern retweeted
We’re publishing a new constitution for Claude. The constitution is a detailed description of our vision for Claude’s behavior and values. It’s written primarily for Claude, and used directly in our training process. anthropic.com/news/claude-ne…
505
944
581
7,641
3,418,308