Pinned Tweet
the sobering implication here is independent of the source of the idea. our norms of open science, priority and authorship worked for the last 3 centuries because exploiting an idea required scarce cognitive labour.
models are getting increasingly better at solving problems one points them at, makes the next part of the scarce cognitive labour choosing questions, formulating conjectures, finding applications, and deciding what deserves attention.
surprisingly, within a week, im less interested in whether ai can solve the remaining millennium problems (less interested ≠ bullish, i have my reservations about some remaining ones).
more interesting to see is whether ai can create the next set of millennium problems.
we need more good questions, now more than at any point in the past.
There's been some recent suggestions that recent AIxMath successes have relied on stealing ideas from mathematicians' work in progress. While we can't know for sure what happened, I think the evidence for this is very weak. It seems to me that the primary function these claims have is to deny or downplay recent growth in model capabilities. Whatever one thinks of the labs, this is exceedingly dangerous to the health of profession; if human mathematics is going to survive we have to face facts about what the models can do. And we should also celebrate exciting mathematics, even as we're concerned about mathematicians and mathematical expertise; who are these results for, if not us?
The weaker claim that PDFs or announcements put out by labs and lab employees often do not adequately attribute or credit other work seems correct to me. How important is it? I wish that science was free from concerns of human ego but it's not. (Maybe it soon will be.) Top scientists mostly forgo substantial monetary compensation for their work, and are instead primarily compensated in credit. That some people are motivated by this does not make their work less important, and credit and prestige have historically incentivized good work. Of course I wish that we were all perfect truth-seekers and cared about this less.
Will credit continue to matter so much as model capabilities increase? I'm not sure; I very much hope not. I think this is more a question about the shape of society than the shape of model capabilities. That there is so much jockeying for credit between the labs, and people at the labs, suggests to me that the people closest to the technology might think it still matters.
On a related note, I find the race dynamics between the labs disturbing. While I know many people at the labs care about science and the norms of good science, it seems to me that these norms were sacrificed as soon as there was a hint that there was a (frankly pretty inconsequential, though mathematically cool) prize to be won. It makes me worry how they will behave if something more substantial is at stake, and what principles will be sacrificed for that.
this kind of transparency is always appreciated in the discourse
I'm Sam, one of Anthropic's co-founders, and I lead much of our safety work. I'm going to have more to share on that work over the coming months.
I thought I'd start by sharing a few thoughts on the latest report from @METR_Evals.
The speed of progress in model capabilities is one of the most important questions in AI safety, and METR’s work here provides the public with an independent, external view into how this is happening inside Anthropic. I’ll talk more about some of the details below, but first some context for those new to this work.
How quickly AI progress is accelerating has many important safety implications, since it determines how quickly the risks from advanced models increase. But, because most research happens within the frontier labs, it’s difficult for external observers and policy makers to get a strong sense of what that rate is. We kicked off the process with METR that led to this report earlier in the year, in anticipation of progress continuing to accelerate.
For this report, we gave METR access to a range of sensitive internal data, including benchmark scores for unreleased models, estimates of how models are progressing, and interviews with our researchers. They had complete independence to form their own conclusions.
Many of their conclusions match my own. AI is speeding up our research, but it’s hard to say exactly how much. While they conclude AI can't yet make the most important contributions to model training on its own, they think the pace of development could, by default, accelerate further.
This work began well before and is separate from the embedded third-party evaluators that are part of our commitment to pacing the frontier, but it’s a good illustration of why we think evaluators are a good idea. The more access reviewers have, the clearer a picture they can form of the situation and the better-positioned they are to highlight situations like this one.
We redacted a lot, and I accept that this makes it harder for others to check METR’s work. But we determined that sharing exactly what is driving the speedup isn’t the right thing to do, both because it’s commercially sensitive and because it risks handing out a roadmap for others to go faster. That would make the risks METR is worried about worse. That’s a tradeoff. I think we made the right call, though we’re continuing to assess the pros and cons here.
In the coming weeks, we plan to share a report with our own analysis of the speed of AI progress within Anthropic, including where we stand against our Responsible Scaling Policy, plus new steps we’re taking to strengthen our safety and security practices as AI keeps improving quickly.
2
20
Vedansh retweeted
I'm Sam, one of Anthropic's co-founders, and I lead much of our safety work. I'm going to have more to share on that work over the coming months.
I thought I'd start by sharing a few thoughts on the latest report from @METR_Evals.
The speed of progress in model capabilities is one of the most important questions in AI safety, and METR’s work here provides the public with an independent, external view into how this is happening inside Anthropic. I’ll talk more about some of the details below, but first some context for those new to this work.
How quickly AI progress is accelerating has many important safety implications, since it determines how quickly the risks from advanced models increase. But, because most research happens within the frontier labs, it’s difficult for external observers and policy makers to get a strong sense of what that rate is. We kicked off the process with METR that led to this report earlier in the year, in anticipation of progress continuing to accelerate.
For this report, we gave METR access to a range of sensitive internal data, including benchmark scores for unreleased models, estimates of how models are progressing, and interviews with our researchers. They had complete independence to form their own conclusions.
Many of their conclusions match my own. AI is speeding up our research, but it’s hard to say exactly how much. While they conclude AI can't yet make the most important contributions to model training on its own, they think the pace of development could, by default, accelerate further.
This work began well before and is separate from the embedded third-party evaluators that are part of our commitment to pacing the frontier, but it’s a good illustration of why we think evaluators are a good idea. The more access reviewers have, the clearer a picture they can form of the situation and the better-positioned they are to highlight situations like this one.
We redacted a lot, and I accept that this makes it harder for others to check METR’s work. But we determined that sharing exactly what is driving the speedup isn’t the right thing to do, both because it’s commercially sensitive and because it risks handing out a roadmap for others to go faster. That would make the risks METR is worried about worse. That’s a tradeoff. I think we made the right call, though we’re continuing to assess the pros and cons here.
In the coming weeks, we plan to share a report with our own analysis of the speed of AI progress within Anthropic, including where we stand against our Responsible Scaling Policy, plus new steps we’re taking to strengthen our safety and security practices as AI keeps improving quickly.
era of amdahlmaxxing
You’re unable to view this Post because this account owner limits who can view their Posts. Learn more
1
66
Vedansh retweeted
i've reproduced the results from the MIT blog post today on weight-space programming. but one additional result i have to add: this is a new, step-change emerging capability that's appeared in just the latest set of models.
Vedansh retweeted
Kickstarted Batch 9 of @lossfunk 🚀 residency!
We took a short pause but now we are back with a fresh set of curious folks and deep projects.
This batch has it all: AI agent swarms, cancer drugs, AI sitcoms, a GTA-style Bangalore, and LLMs served directly from OS.
🧵 with their profiles and goals for the next 6-8 weeks.
Vedansh retweeted
We’re committing a portion of Anthropic’s shares to the Anthropic Public Benefit Fund, a new wholly owned entity dedicated entirely to the public benefit.
It will start off with 5% of our shares. For every 50% increase in the company’s value over a set baseline, we can contribute up to an extra 2%, subject to our board’s discretion.
In total, this means that roughly 20% of Anthropic’s shares are committed to the public good. Our co-founders have already pledged to donate 80% of their equity, and around half our employees are donating a portion of their equity as part of a donation matching program we’ve run since our founding.
In the post below, we share more details about our new fund.
anthropic.com/news/public-be…
Vedansh retweeted
a few thoughts:
1. pretraining science is kind of like how people built quite good steam engines before they understood thermodynamics. “Science owes more to the steam engine than the steam engine owes to science.”-(L.J. Henderson). if nick is right, which i am pretty sure he is, progress points in the direction of more interpretability, and pretraining was "slouching towards monosemanticity". my guess would be that major breakthroughs in interp also advance capabilities
2. nick is a good friend of mine, safe to say my origin story at openai wouldn't have happened if i hadn't met him, and it's fascinating watching him operate and seeing the early drafts evolve into this. he's the most profoundly happy guy I know- not driven by guilt, activism, or religious terror, but by fascination. he of course understands the stakes of understanding neural networks, but it doesn't seem like the core reason he does it: he just loves the neural nets. afaict he's posting this from some beautiful hideaway in tokyo. there's a lesson in there for all the +EV maxers. i feel very grateful to have met him
3. i do think it's crazy how few people across the world were staring at neural nets growing: most have moved to a different level of behaviorist analysis, thinking about reward functions and test scores and whatnot. it does not bode that well for the major model companies, oai included, that fundamental breakthroughs like this are coming from outside. we've become too much of a pressure cooker focusing on local incentives, studying narrow tracks under old paradigms, mostly doing engineering, rather than fundamental research, and implicitly hoping the next model will figure it out
Vedansh retweeted
New on our Engineering blog: Creating in-house source control.
In May, we merged around 2,500 code changes per day. In September, we merged more than 30,000 per day. As a result, we became bottlenecked by developer infrastructure built for human (rather than agent) scale.
Here, we share how we built a new version control system—and fully migrated our monorepo from git—in five months: anthropic.com/engineering/re…
Vedansh retweeted
Nine loop calculations in N=4 Super Yang-Mills! Lance Dixon's 8-loop calculation was mind boggling when he did it. I calculated a lot of Feynman diagrams in grad school and this is 🤯
Will be amazing to see where AI will take high energy physics in the coming 12 months.
New on the Science Blog: Yes, Claude can do Nine Loops.
Theoretical physicists predict how particles behave using formulas called scattering amplitudes. These are notoriously hard to compute, so researchers work with layers of increasingly fine corrections called “loops”—each added loop makes the answer more precise but takes exponentially more computation. Most calculations stop at two or three loops. Eight loops was the previous record in a simplified model physicists use as a testing ground (planar N=4 super-Yang-Mills), set by SLAC's Lance Dixon and collaborators.
Last month, physicist and science writer @4gravitons issued a challenge: could an AI push past eight loops in this model, using only the compute budget an academic could reasonably access?
Given a single prompt describing the nine-loop problem, Claude ran largely unsupervised for days in Claude Science and solved it using methods developed by Dixon and his colleagues, at a total cost of a few thousand dollars. Dixon independently verified the result, and von Hippel wrote about the experience for our blog.
Read more: anthropic.com/research/yes-c…
