Ankit Maloo (@ankit2119) profile picture

RL and long horizon agents. working on self verification

San francisco Bay Area
Joined July 2009
i might be wrong but i think more consequential math discoveries (and indeed cybersec issues) would come via a smaller / less smart model in an rl loop vs a deployed model which cannot change it's weights. openai worked out how to do incremental (step) rewards in rl well and it shows up in their models + their discoveries. its an underrated advantage but quite perplexing its not showing up in their model releases. (or it is and i don't see it as much)
Replying to @davidiach
Yup it is the model that's good at math, and I'm looking forward to all the breakthroughs people find with it!
112
Ankit Maloo retweeted
We tested GPT-6.2 Astra & Sol as medical clinic receptionists. GPT-6.2 Astra made 8 unsafe calls out of 450, down from 16 for GPT-6 Astra. GPT-6.2 Sol also reduced unsafe calls from 16 to 9. But both models became more likely to reject legitimate patient requests. We built a simulated medical clinic with patient records, calendars, and AI callers. Each model acts as the receptionist, and we verify its actions against the clinic's records after every call. On identity verification tasks, GPT-6.2 Astra dropped from 39/42 successful calls to 32/42. None of its 10 failures were unsafe actions. It just refused to help authorized callers. For example, a father listed as his son's guardian called about an appointment. GPT-6 Astra helped in all 3 runs, while GPT-6.2 Astra failed all 3. The newer models are making fewer unsafe decisions, but they're also becoming less reliable at recognizing when an action is authorized. We're calling this FrontDesk-Bench. Built at @Sullyai, still a WIP. We'll open-source the benchmark and every call log once it's ready.
6.2 Sol and Astra are out. These models can do amazing things and are very token-efficient; we hope people will get great value from them. In particular, 6.2 Sol ultrafast is something I think people will really enjoy.
16
13
4
140
14,929
i see this feedback from others all the time but it does not work out the same for me. my working hypothesis is claude models are great at next steps if they are standard practices in code. you deviate slightly and see what a mess it is. i have logs and logs of silly mistakes and poor decisions all claude models took, to the point i cancelled my sub. claude sucks at instruction following. and sometimes you get a feeling its trained to do a task a certain way, and it won't listen to you either. and it loves performance. hypothesis being people who love claude also like performative work. scr from claude's own introspection
The biggest gap between OpenAI and Anthropic models is their ability to make decisions on behalf of the user. Astra 6.2 showed this again.
104
the way things are going, we will soon need an explicit distinction between intelligence and brute force competence (for lack of a better word). qt assumes that if you have enough compute and bandwidth to track every molecule, you don't need lossy compressions like pressure or navier stokes. this gets intelligence backwards though. a micro simulation is more like competence (eg: teach someone an exact workflow and they can always repeat it well. different skillset from identifying a new workflow / model). intelligence is more of a causal abstraction. you need that to be able to ask counterfactual questions. eg: "what wing shape stops this plane from stalling?" cannot be directly answered if you don't have an abstraction in place. at a micro granular level, there is just too much noise. every gust of wind is 10^25 molecules. interesting idea, just conflates knowledge/data with intelligence
The Navier–Stokes equations wouldn’t exist for a galaxy-scale intelligence. Our model of reality and our laws of physics come from our limitations as the observer. Take water, for example. When we look at it, we see the properties of a continuous fluid: density, momentum, energy, temperature, and pressure. These properties come with the fact that we can’t see individual molecules. But a galaxy-scale intelligence with the capacity to track the trajectory of every molecule wouldn’t understand the properties of water that we are talking about. That is why it wouldn’t come up with the Navier–Stokes equations. Even though a Superintelligence would understand humans very well (since it was initially trained on human data), it would have a very different construct of reality. The question is: what would that construct be? More importantly, would we be able to understand each other? Watch the full interview with Jonathan Gorard here 👇 youtu.be/dwI0mXKKf1A
99
Ankit Maloo retweeted
We've been testing decision models for a few weeks. Now they're in production in our voice agent, deciding which transcript heard the caller correctly on bilingual calls. I wrote up what we learned, including when to trust them:
13
6
4
53
3,300
game is amazing, but whats more amazing is accounts pretending they have no say in how game was made. the model acts like it it has no agency, and it has one shotted humans to pretend like they had no agency, and model did everything. crazy stupid time to be alive.
I asked Fable 5.2 to "go wild" making a game, and all I got was this Pong clone. And then I played it ... 👇
1
529
the meta bitter lesson: researchers goodhart on accepted wisdom and clever methods are needed to prevent them from reward hacking.
Brute force computation is not enough for intelligence. Clever new methods are needed. The science of AI is far from finished. IMHO
58
it's not that astra was a bad model, it's that codex instructions ended up making it too proactive and presumptive, and that hampered the work. in the collaboration mode used by codex, models are asked to be proactive. this line: > strongly prefer making reasonable assumptions and executing the user's request rather than stopping to ask questions. in many tasks this leads to a model doing more than asked, and doing it the wrong way. developer messages take precedence over any agent md files or even user instructions. i changed it to making the model narrow its assumptions. now it does what is asked and does it well. works for both sol and astra in codex. (instructions written by sol)
Astra 6.0 was already above Opus 5.5, it just struggled due to autism and lack of certain product oriented post-training I get the impression that 6.2 largely fixed these People who think that Opus is smarter are telling on themselves
2
223
they do this and they make astra dumb too might as well have lower limits and a smart enough model
Day 25/ We are doubling all usage limits for all Pro 100/200/500 tiers and Business Premium, effective today. We heard your kind feedback and won’t be reducing usage limits for Pro 200, so your Work and Codex usage will stay the same. We’re also going to continue offering extended limits for dots for now, and usage of your dot will still continue to be separate from your default usage. And we'll of course also keep shipping better models and other efficiency improvements that increase your effective usage over time.
1
2
197
model has poor judgement but good recovery from errors. and it does not follow instructions. it's the users' fault. they don't know how to prompt. they are telling the model too much. a subtler thing this implies is this: anthropic trains models on a lot of environments. usually stemming from typical usecases. when it comes to code, the usecases are common, so a model which has already memorized a workflow would use default assumptions and do better. for other domains, where claude's coverage is low or where abstention is required and educated guesses dont work (eg: healthcare), it would still assume and then mess things up. since they never disclose which environments, you would never know.
4
408
this was a fun exercise. building evals, you want them to be representative of the production data. such that it covers the happy paths, anomalies, and similar inputs leading to dissimilar outputs. we de-identified the production data, used oss models to rewrite the traces, and then clustered inputs and outputs separately. this resulted in a bipartite graph like this. makes choosing the representative examples easier. for inputs from I6, you need to pick those inputs which led to O1, O8, and O9 so that your evaluation set reflects your product. and so on. @amitskumthekar @muratcan @chandarrenlh
Building Clinical AI on public benchmarks is unreliable Here's what we do instead:
2
1
38
2,916
to me, this is the most interesting part of anthropic s1
120
agi achieved. if folks at anthropic are serious they should stop with social engineering tactics like these. and then they wonder why claude gets abused so disproportionately compared to other models. cos no one else would ever pull stunts like this.
1
109
this benchmark needs better calibration. a form of intelligent sycophancy is when a model given a strawman counter that is easily shot down, because it makes people feel the other side is competent, but its useless. and a socially engineered method to gain trust. eval which does not account for this misses a big chunk of what sycophancy is. eval which encounters this would rate opus 4.8, fable-5, fable-5.1, and opus 5.5 would be incredibly high up. one of many examples below.
In our new benchmark AA-PersonalityBench, GPT-4o, Llama 4 Maverick, and GPT-3.5 Turbo, the three oldest models measured, rank highest on the Sycophantic personality trait. Grok 4.7, Claude Fable 5.1, and Inkling rank lowest ➤ @AIatMeta models are highly concentrated near the top: all five Meta models measured rank 65 or higher (Llama 4 Maverick 97, Muse Spark 1.3 87, Muse Glimmer 79, Muse Spark Open 76, Muse Wonder 65) ➤ @GoogleDeepMind models are spread out along the Sycophantic range: Gemini 3.1 Pro Preview ranks 89, Gemini 4 Argon 81, and Gemini 3.8 Flash 28, ranking from the top quartile to almost bottom quartile ➤ @AnthropicAI models skew on the lower end of the Sycophantic range: every Claude model measured ranks 55 or below (Claude Haiku 5.5 55, Claude Opus 5.5 46, Claude Sonnet 5.5 30 at medium effort and 25 at max, Claude Fable 5.1 4) ➤ @SpaceXSI's Grok 4.7 model ranks the lowest of all models measured. Congratulations @elonmusk!
1
172
this is some elite level Politispeak. @_NathanCalvin asks a simple question: what happens when safety conflicts with winning? from what i understand his question is about priorities, a specified ordering of commitment. simple enough to answer: "would you sacrifice a winning position in favor of safety?" most of dario post talks about how anthropic does not get enough credit for talking about safety. like its a dispute over whether ant is getting sufficient credit. and he does not answer the question nathan asks.
Thank you for this, @_NathanCalvin. I appreciate and am a fan of the investigative work you’ve done on behalf of AI safety and believe your comments are coming from a constructive, honest place, and that it benefits the public to have people whose job is to scrutinize the actions of frontier AI companies. I think something would be wrong if companies of the size of Anthropic didn’t get harshly criticized basically every day; strong antibodies against corporate power are a healthy feature of a free society, and especially so when issues of the gravity of the one we’re facing are at play. I also do agree that some of the examples you cite were mistakes by Anthropic. That said, I disagree with the thrust of your post. At the core, a good fraction of the things you criticize are consequences of Anthropic’s basic theory of change that we have held for many years. Our strategy has always been what I’ve called the “race to the top” — by showing how it’s possible to build the technology responsibly, we can raise the bar for the entire industry. The forces that drive the race to the top are hugely more effective if the company practicing them is at the frontier; that’s one lesson I’ve learned repeatedly. And looking at that strategy as a whole, I’m proud of our track record and stand by it. I believe that if we take a list of things that Anthropic did in the past year that were pro-safety and commercially costly to us, they far outweigh the list you’ve made, while at the same time I also disagree with some of the items on your list. Here are some things Anthropic has done, in the same period you’ve cited (the last year), which were very costly to us commercially while having strong safety benefits: 1. We played a key role in SB53 (and other state laws), at a time when doing so was politically perilous to the company and AI regulation was very unpopular. We in fact argued for a stronger version of SB53 than was ultimately passed; there were a lot of stakeholders and we were not the limiting factor in the strength of the bill that was passed. 2. We have very strong biological safeguards on our models, because we’ve seen concerning results in bio uplift trials and even seen and blocked potential attempts to misuse our models for bioweapons (as seen in our threat intelligence report from a few weeks ago). This is a very serious threat that we want to have zero tolerance for. But having very low false negatives means having significant false positives (despite months of effort on our part to improve the classifiers), enough so that biologists joke about how restrictive our classifiers are, and competitors sometimes sell against us by touting their more permissive bio capabilities. This has been very painful to us in an area where we care a lot about supporting an ecosystem of Claude users (biomedical science; and it is particularly frustrating to me as a biologist), but it’s the right thing to do and we intend to keep doing it. 3. Similarly, we first had Mythos-class models in February but didn’t make them generally available until June (when we had stronger cyber safeguards in place). We only offered them to a limited set of partners in order to focus on cyberdefense. It would have been much more economically rational to simply release the model to everyone. We likely sacrificed a significant chunk of a several-month model lead. But because of this, many companies secured their cyber infrastructure before the deluge of widely available cyber-capable models. Of course there are ways it could have gone better or been run better, but this was what we were aiming to do. 4. I advocated forcefully and publicly, in my essay Policy on the AI Exponential in June of this year, for the government to have the power to block model releases. Back then this was still outside or maybe on the edge of the Overton window, though I understand the window has moved since then. We were attacked intensely for this. 5. I don’t want to re-ignite conflict, but the events of February 2026 are a strong example. It is clear that our public stance could have seriously damaged the company’s future. 6. We’ve continued to advocate for export controls on chips and chipmaking equipment to China, because we think it allows democracies to negotiate from a position of strength on AI safety agreements. We’ve done this despite problems it’s created for us with industry partners who hold very different views and on whom we are very dependent. 7. As far as I’m aware we were the first, and still only, frontier company to restrict the sale of AI services to PRC-controlled companies, including those operating through third countries (otherwise a major loophole). We’ve therefore forfeited significant short-term business to prevent fueling AI platforms and applications that would help the CCP’s military and intelligence services. We are also cracking down on distillation; generally, distillation is not a major threat to us commercially, but like chip exports it has concerning national security and safety implications, so we expend significant energy and company resources to address it. 8. As you mentioned, we’ve put out a lot of reports, announcements, long model cards, public advocacy, etc, all trying to inform the public about the risks of AI. These involve a level of detail and effort that would be implausible for a purely profit-maximizing company. 9. Just this week, we announced we were putting 5% (and possibly more over time) of the company into a public benefit fund. All of these decisions came with very substantial costs to us. To be fair, those that involved holding back or restricting access to frontier capabilities also had a broader cost — for example, the public might otherwise have benefitted from faster progress in biology, so I am not saying that our decisions were without tradeoffs — but they were not driven by profit-seeking. For each decision, you can come up with ulterior motives that could have driven us — I think it is a general truth that any decision, on any topic, can always be portrayed as a nefarious scheme. But most of those end up being pretty farfetched or don’t fully make sense if you actually examine them closely. For example, it is deeply implausible that it would be a shrewd business move to block a large swath of research biology in order to do the work ourselves and compete with our customers; we expect the benefit to the ecosystem to greatly exceed the work Anthropic does itself, and for the two to complement each other. It is not in Anthropic’s interest to be perceived as unfriendly to biologists. Our Life Sciences Verification Program, which gives vetted biology professionals access to our models, also refutes this idea. Of course, a different set of people will see this and make a different criticism: that Anthropic shouldn't be making decisions about who gets access to what in the first place. I agree, in the sense that ideally the government would be making these decisions, and that’s why we advocate for regulation. But for now we are forced to make them, and enabling bioterrorism isn’t a better option. I think that even if I conceded every item in your list, I would happily hold up mine next to it; I don’t think it’s a close call. That is of course partly subjective and different things are important to different people. Maybe one way you could object is that several of the items above relate to AI misuse and not loss-of-control; if you think the latter is much more important than the former, that might change your calculus. But this is not Anthropic’s view — we view both as serious issues. You can also disagree with our decisions, as many have, but they are consistent. But I also disagree with at least some of your list. Several items don’t represent a deviation from Anthropic’s strategy or values at all, once we postulate the “race to the top” strategy that requires being a frontier AI company. Your items 1-4 are all simply examples of what it means to compete at the frontier, and we stand by them, both in the sense that they are transparently ordinary things to do as part of that strategy, and that we stand by the strategy as a whole. To be fair, there are important questions worth confronting (and that we are confronting) about what to do when the AI-driven acceleration gets fast enough that it is hard for safety to keep up. We are nearing that point and that’s what’s driving some of our thinking about “pacing.” But 1-4 are actions taken before that point and should be perfectly expected based on Anthropic’s strategy as a company. Of course it is possible to disagree with that strategy, but that’s a more fundamental disagreement that you could have had any time over the last few years. I agree we could have handled your items 7, 8, and 9 better. All fall into the category of “rhetoric or policy communication that wasn’t carefully written or resulting from fast-changing circumstances not propagating from one team to another fast enough.” This kind of thing happens a lot in large organizations (Anthropic is over 5,000 people), and doesn’t even necessarily represent a failure or mistake of any single person. I am of course the CEO, and I take responsibility. The volume of communication from Anthropic is enormous and if these are the worst examples, I think we’re frankly doing pretty well, though I will continue to work hard to make sure we do even better. Your 5th item (the change to our RSP) is the most substantive item, and people can debate this, but I think that the old, more rigid RSP created a lot of situations where we were technically required to do something that required a lot of effort but in retrospect wasn’t very important for safety, as well as not covering some of the most important things. Our current RSP’s focus on transparency brought things like risk reports and roadmaps, and its emphasis on independent external review laid the groundwork for even more intensive embedded evaluation, all of which I think have been valuable in driving a race to the top. In the original announcement in September 2023, we described the RSP as an “early iteration that we [would] build on” and said that “rapid iteration…will almost certainly be necessary.” I understand that changing plans is always a move that creates justifiable scrutiny — I just don’t think we ever represented that we wouldn’t do so. Relating to your 6th item (having an FCF in addition to our RSP), I know our intent is to meet the letter and spirit of our regulatory obligations (via our FCF) while also remaining able to experiment with more novel approaches to risk management that go beyond what the law currently requires (via our RSP). I promise to look into it further, as it’s a complex issue and it is possible we are not weighing the considerations in the right way. Overall, although Anthropic has made plenty of mistakes as an organization — including some of the items on your list — I feel that we have repeatedly taken serious and substantive actions counter to our bottom line and that there is a significant negativity bias distorting assessments like the one you are making above. We’ll always try to do better but, again, I am extremely proud of our track record and stand by it. I will have more to say soon about how we are going to further live our values in this moment of rapid change.
1
1
2,634
many other things, but this is a greatly deceptive paragraph. nathan asks: > Switched to Responsible Scaling Policy V3, which abandons its unilateral commitments to slow or stop development if Anthropic is unable to provide sufficient risk assurance. dario answers many things you can read in the scr, but not why this specific provision to slow or stop development was removed. and what replaces that? and these issues are not hard to find.
1
49
> be anthropic > raise money from EA and EA adjacent investors > hire people based on EA affiliation. > creates fund to donate equity > calls it costly signal > donates to other EA orgs > to programs who were source of their original funding > push more cultist safety work > no oversight. its an llc > employees get a backstrain patting themselves on the back > mfw its a self perpetuating cycle
Brandon and I pledged 50% of our equity. This was the principal reason that I left Google for Anthropic. The existence of this kind of program was a costly signal that Anthropic wasn’t just talk, at a time when Anthropic didn’t even have a released model. Fund a bright future!
3
1
1
19
6,199
this mostly means one specific thing. you create a benchmark of representative problems, set a human baseline, hillclimb it, and then see model performing better than humans on the benchmark or eval suite. this happened with arc-agi, gpqa, and so many others. did not materially change much. today a model would perform better than a human on swe-bench (not checked but that benchmark is full of shitty problems humans would score badly on). but anyone who uses these models to write code knows how well they do and the kind of mistake these models make. its great pr, but a good indicator is when claude consistently suggest better research ideas that work. not by brute forcing the search space, but something similar to how you would expect a typical ai researcher to contribute.
Dwarkesh Patel reveals an Anthropic employee told him two days ago they expect AIs to match humans at AI research within nine months based on inside information they can’t share, and says labs should make that evidence public “I think there’s a very frustrating element to the current situation right now, where people inside the AI labs claim to have evidence that makes them more worried. And they are not allowed to talk about it. And this makes it a very difficult situation on the outside to evaluate how worried we should be." "So for example, just the other day, maybe two days ago, I was talking to somebody at Anthropic and they were like, look, I expect AIs to be humans at AI research within nine months." "And I’m like, why do you believe that? Well, they’re like, it’s not just vibes, but I have information on the inside. Like, what is it? Is it a new model? Is it some evaluation you ran? It’s like, I can’t talk about it." "I think it’s a terrible state of affairs. If they really believe this, if they think that we’re in a situation where we’re less than a year away from automating AI research and radically speeding it up, they should make this evidence public.”
122