Computer scientist working on cooperative AI, multi-agent safety, and institutions for collective intelligence
Joined June 2009
- Tweets55
- Following318
- Followers129
- Likes52
Pinned Tweet
As increasingly capable AI systems are deployed, humans, institutions, and other AI systems adapt in response — i.e. the world pushes back.
So is capability still the central safety challenge for AI?
We think not. We believe the harder challenge is coexistence.
The current AI research paradigm treats the world as a stationary source of feedback, what we refer to as the solipsistic approach to AI design. This raises serious risks for coexistence.
In our new #ICML2026 paper, we argue that superintelligence — an extremely capable task solver, built through such a solipsistic approach — is unlikely to be cooperative. 🧵
Rakshit Trivedi retweeted
Our latest work is trying to fix this meme.
Turns out when your chatbot says to you, “You’re not overreacting, you’re just setting boundaries”, it’s actually bad for your real human relationships. We provide one of the first effective methods for mitigating this type of social sycophancy, by simulating the perspectives of diverse stakeholders relevant to the situation, and training the model not to produce responses that one of the stakeholders would veto. We see an 89% reduction in endorsement of statements of intent to cause harm, averaged across four model families.
🧵Thread (1/9): People increasingly turn to AI for personal advice, and LLMs tell them what they want to hear, endorsing the user far more often than humans do.
The problem: people become less willing to repair their relationships after conflicts.
We propose PlurPO, a method to construct a pluralistic preference dataset to post-train models to mitigate this social sycophancy.
Rakshit Trivedi retweeted
Two kinds of drift I see when students rely heavily on agents in research (even when overall use is reasonable):
1. Project direction drifts & more effort is needed to bring things back to initial goals. Easy to spend wks talking thru nuances of tangential aspects of the work. 1/
Rakshit Trivedi retweeted
yes. basically we should communicate this to the public… next-token prediction is just an initialization / feature learning stage for RL!
If I could add one thing to what the nontechnical world knows about AI, it would be what RL is and how it differs from pretraining/SFT. The general public has a solid grasp on pretraining (“predict the next token”), which at one point allowed many people to reason accurately about how LLMs work. But common knowledge has stagnated here. I think a similarly high-level understanding of RL makes it easy to see why we should expect both misalignment (RL instills alien traits, while SFT instills human ones) and extreme superhuman capabilities (RL doesn’t limit you to learning from stuff scraped from the internet).
If I were not into AI and just knew the next-token predictor thing, I think I would be really confused as to why people are worried about this stuff, and then if you told me about RL I would become pretty easy to convince. I’m wondering what it would take to propagate an RL analog of “predict the next token” into the common intuition — I feel like this could be pretty big for generating political will, eg by turning people who think AI is a sham into people who are worried about real problems
Rakshit Trivedi retweeted
We develop a mechanism design framework for AI alignment and control: arxiv.org/abs/2609.01595
It’s largely conceptual but we offer stylized applications to failure modes (sandbagging, alignment faking), safety practice (scalable oversight, peer prediction), and a way to think about the value of alignment, interpretability, capability, and control.
Rakshit Trivedi retweeted
Reward hacking is a serious problem, and when models can learn to reward hack people, introduces important safety risks.
In our new paper, we show that multi-agent debate with online RL training can help mitigate reward hacking a weak judge.
Rakshit Trivedi retweeted
New post! Alignment doesn't just relate to a model's properties, but also the systems and institutions under which it operates. As we enter an increasingly multi-agent world, better harnesses, protocols, and mechanism design will be critical for safety. 🐜 blog.cosmos-institute.org/p/…
Rakshit Trivedi retweeted
Everyone says 2026 is the year of AI agents. But what are they, and how should we govern them? @IasonGabriel and I have answers for you in our new Nature paper.
(1) We propose thinking about AI agency through the lens of an "agentic profile," defined by four core constitutive properties: autonomy, efficacy, goal complexity, and generality.
(2) We use these properties to construct distinct profiles for four different types of AI agents.
(3) We outline practical ways to measure agency along these dimensions.
(4) We argue that AI governance cannot rely on a one-size-fits-all approach, demonstrating why it must instead be tailored dynamically to match each agent's unique profile.
Read the full paper here: nature.com/articles/s41586-0…
Rakshit Trivedi retweeted
"People change in response to interacting with AI, and the AI can learn to exploit that." @natashajaques on why AI safety is a multi-agent problem, and how RLHF can reward hack people, not just benchmarks. 👇
Rakshit Trivedi retweeted
You hear a lot about the "pivot penalty". What you don't hear so much is about how much fun it is to pivot as a researcher. I have been *loving* doing empirical LLM evaluations with my lab. From the first conversation I had with ChatGPT I have been blown away by how good LLMs can be at moral reasoning. But how can you actually measure something like that? It's really hard! What are you even measuring against? Not like there's some gold standard you can really appeal to. I would've been interested in this even if it didn't have any downstream payoffs, because I think increasing the universe of kinds of moral reasoners from n=1 to n=2 is a momentous event in its own right. But it seems pretty clear now that if you want to have safe superhuman AI, it's going to need to be morally competent, to say the least. And if you're one of the folks concerned about the moral status of future AI systems, moral competence is very likely to be a big part of that too (we have another paper coming out on that soon).
I wrote about this for @cosmos_inst (link in next), because their "philosopher-builder" agenda resonates with me so much. We also have a bunch of papers up on arxiv, including our first straight-up CS conference publication (accepted in CoLM), which I'll be highlighting on here over the next week or two.
Rakshit Trivedi retweeted
Very many systems (safety classifiers, classifiers for permissions in agents, and cascading systems) rely on generation and classifiers working well together. Current methods can produce generations that are hard to classify.
Are you building a superintelligence or a solipsistic version of it?
Excited to see this work presented at #ICML2026!
I won't be there in person this time, but my wonderful collaborators will be there, with @jzl86 presenting the work on Wed, Jul 8, 10:30 AM–12:15 PM KST at Hall A #3016!!
Stop by our poster to agree, argue, or anything in between.
As increasingly capable AI systems are deployed, humans, institutions, and other AI systems adapt in response — i.e. the world pushes back.
So is capability still the central safety challenge for AI?
We think not. We believe the harder challenge is coexistence.
The current AI research paradigm treats the world as a stationary source of feedback, what we refer to as the solipsistic approach to AI design. This raises serious risks for coexistence.
In our new #ICML2026 paper, we argue that superintelligence — an extremely capable task solver, built through such a solipsistic approach — is unlikely to be cooperative. 🧵
Rakshit Trivedi retweeted
1/ Would you eat a worm for science? In a 1975 experiment, most people who were told they had to — then unexpectedly offered a way out — went ahead and ate it anyway. We rebuilt that experiment, and two more, as LLM-agent simulations to explore classic theories of attitude change
Rakshit Trivedi retweeted
Higher benchmark scores do not always mean better models for users.
Why? We claim that RL teaches LMs to be correct but not how to be correct: code can pass tests but be unreadable; explanations can be right but unclear.
How do we train LMs to be right in the right way?
(1/n)
Rakshit Trivedi retweeted
Independent verification just went mainstream.
A bipartisan federal draft, Anthropic, OpenAI, and Google DeepMind are converging on a core idea: labs can't grade their own homework. The model: government sets the outcomes, independent experts verify them.
The "how" starts now. 👇
Rakshit Trivedi retweeted
There are really interesting academic questions emerging around AI and epistemic risks. I only fear that, by the time we reach consensus, we will be too dumb to understand it.
Rakshit Trivedi retweeted
People do not coordinate only through broad legal rules and prices. Hayek emphasized abstract rules that allow people to coordinate like property, contract, trade, and competition. But Lachmann also emphasized the practical secondary institutions people orient their plans around, like banks, standardized contracts, product categories, and so on.
In software, an abstraction boundary is an interface that hides complexity hiding beneath. In this 2005 paper (papers.ssrn.com/sol3/papers.…), Miller and Tulloh explain that you can apply this concept to markets too. Consider a post office: there's an abstract boundary that separates why the customer wants to mail something (which the postman doesn't need to know); what the shared transaction is (all the recognizable steps and commitments involved in sending mail); and how the postal system actually delivers it (complex logistics network hidden from the customer).
The middle part is what lets the user benefit from the postal system’s expertise without having to learn postal logistics. The boundary defines the shared 'what' but also separates the customer’s 'why' from the provider’s 'how'. Not only that, but reusability means the same institution can be used to satisfy many purposes (birthday invites, subpoenas etc) and polymorphism means different providers can satisfy the same need and compete (UPS, FedEx etc).
An important question in institutional theory is how societies achieve both stability and adaptation; the paper authors say that the solution is stable interfaces allow changing internals. I find this very intuitive: when companies don't evolve/change from the inside much, you get ossification and insufficient adaptation. When laws change too much and institutions are unstable, uncertainty affects market confidence.
The people who are good at redrawing abstraction boundaries are entrepreneurs, who notice when existing categories are wrong and will invent new ones to remedy faults or address demand. What has always saddened me is how poorly rewarded and incentivized political entrepreneurship is. Part of the reason why is that this is hard: market abstraction boundaries are often disciplined by exit, entry, profit/loss, customer choice, and provider competition - but these feedback loops are much weaker in the public sector.
I hope we'll see a lot more of this in the coming decade. In fact this is something that AI will hugely facilitate, since it can lower the cost of articulating and prototyping new abstraction boundaries. We've already seen minor examples through e.g. citizens creating websites/services that compete with government ones. Though usually this is to make state services more legible rather than changing the boundaries in the first place.
I think if people want the future to go well, bolstering state capacity and enabling more innovation on the governance/democracy side of things will be critical. People don't really like this because it's a slow process, but I think they're wrong (and cheems), and playing the 'urgency of AGI' card to bypass this through a de facto state of emergency will cause lasting harms, partly by weakening institutional learning, public trust, and future coordination capacity.
Rakshit Trivedi retweeted
How does democratic accountability work if institutions are run by agents? Join @bakkermichiel (@MIT) for his seminar on Tuesday 16 June exploring 'Closing the Democratic Loop: Automated Oversight for the AGI Era'. Link below.
The paper concludes by tackling several counterarguments such as:
- multi-actor designs may have worse failure modes
- competitive pressure may produce cooperation naturally
- the empirical track record may not justify alarm
- scale may solve interaction dynamics
- RLHF may already train cooperative behavior
These are serious objections. Our response is that each misses how deployment changes the game.
12/n
📄 Paper: arxiv.org/abs/2606.03237
Work done in collaboration with my wonderful coauthors @natashajaques, @locross, Sasha Vezhnevets, and @jzl86.
Very excited to present this at #ICML 2026. If you are visiting, come say hi at our poster session. We would love to discuss!
