Asst Prof, Carnegie Mellon Machine Learning Department
Joined June 2019
- Tweets340
- Following198
- Followers2.3K
- Likes745
Bryan Wilder retweeted
I learned a lot from this project. Here are some highlights from our simulated environment.
New paper with @PerraultAndrew on reverse-engineering why agents cooperate with one another. The HF incident is an obvious prompt for this -- do agents work together because they place terminal value on the success of a group? Will they "sacrifice" their interests for other agents? Or do they mostly cooperate for individual advantage? For agents, these questions are quite messy conceptually. It's not clear if they do or should differentiate between themselves, their copies, their future self, etc.
We propose that researchers should use revealed-preference methods to disentangle not just *if* agents cooperate, but *why*. This means using careful experimental design to identify terms in a utility function which rationalizes why agents cooperate under some conditions and not others. Many intuitive experiments do not cleanly distinguish different mechanisms. We give examples of specific strategies to do so and the assumptions that they require.
Empirically, we examine the behavior of Sol, Fable, and GLM in a simulated environment. Big differences between model families! E.g., Sol mostly treats other instances instrumentally, while Fable's behavior is best rationalized by other-regarding preferences.
We think it's fair right now to see most empirical results as agents roleplaying whatever scenario is suggested, with a lot of variation depending on context. But systematic training for multiagent interactions is surely coming soon. Models will be subject to lots of selection pressure that will instill more specific dispositions about cooperation. Our motivation is to prepare for that near future, and work through what experimental designs would allow us to make sense of observed behavior. We actually started this project a month or so before HF and were amazed to see the motivation laid out so clearly and so soon! Time to think ahead to what's coming next.
arxiv.org/pdf/2610.19188
New paper with @PerraultAndrew on reverse-engineering why agents cooperate with one another. The HF incident is an obvious prompt for this -- do agents work together because they place terminal value on the success of a group? Will they "sacrifice" their interests for other agents? Or do they mostly cooperate for individual advantage? For agents, these questions are quite messy conceptually. It's not clear if they do or should differentiate between themselves, their copies, their future self, etc.
We propose that researchers should use revealed-preference methods to disentangle not just *if* agents cooperate, but *why*. This means using careful experimental design to identify terms in a utility function which rationalizes why agents cooperate under some conditions and not others. Many intuitive experiments do not cleanly distinguish different mechanisms. We give examples of specific strategies to do so and the assumptions that they require.
Empirically, we examine the behavior of Sol, Fable, and GLM in a simulated environment. Big differences between model families! E.g., Sol mostly treats other instances instrumentally, while Fable's behavior is best rationalized by other-regarding preferences.
We think it's fair right now to see most empirical results as agents roleplaying whatever scenario is suggested, with a lot of variation depending on context. But systematic training for multiagent interactions is surely coming soon. Models will be subject to lots of selection pressure that will instill more specific dispositions about cooperation. Our motivation is to prepare for that near future, and work through what experimental designs would allow us to make sense of observed behavior. We actually started this project a month or so before HF and were amazed to see the motivation laid out so clearly and so soon! Time to think ahead to what's coming next.
arxiv.org/pdf/2610.19188
Recently, CMU had a great workshop on "AI agency and interpretability", with a mix of philosophers and computer scientists talking about interpreting/understanding AI and safety questions through the lens of rational agency. I wrote up a few reflections on the discussion, linked below.
Mainly, I came out of the workshop more excited about the area. Philosophers and computer scientists have a lot to learn from one another. There are important questions in both theory and empirical science to test if rational-agent frameworks are good ones, infer what beliefs or desires models might "have", and work through the consequences of those for mechanism design, training, and alignment.
My writeup: bryanwilder.substack.com/p/r…
And the workshop program: aydinmohseni.com/ai-agency-w….
Thank you to Aydin Mohseni for organizing!
New paper! Especially relevant to the furor over anthropomorphizing AI the last couple weeks, we ask whether attributing beliefs to LLMs allows good predictions of their behavior. The short answer is yes, for reasonably capable models.
Specifically, we study decision problems with a binary unknown state, probabilistically infer a single latent belief, and use it to explain a range of decision from the model. Testing whether that latent variable predicts behavior well is our formalization of the "intentional stance": that attributions of beliefs or desires are justified if they are good predictions of what the system will do.
From there, we also study questions about measuring LLM beliefs, models' adherence to a prompted utility function, and whether beliefs are properties of the "model" or a specific instance. (1/N)
The scope of the paper is intentionally limited to problems that look like basic statistical decision theory settings. This is by design, because within that scope we can be mathematically precise. I think an important role for scientific research is to provide careful formalization of concepts that are easy to throw around intuitively. We hope to extend these ideas to more complex behaviors soon.
Forgot to post a link to the paper :) arxiv.org/abs/2609.07943
What are the alternatives to anthropomorphism for LLMs? Many alternative mental models ("next token predictor", "cultural technology") have become progressively less useful for predicting their behavior as LLMs get further from pretraining/SFT. Viewing LLMs as engineered systems and attributing their behavior to parts of training (which now includes many components) is necessary for understanding, but not an alternative to anthropomorphism, in the way that biology is not an alternative to psychology. Certainly for day to day purposes, you'll be much more successful at anticipating what LLMs do if you think of them as having, at least locally to a given context, beliefs, desires, intentions, and so on than if you took any other perspective I can think of. Please tell me if you have an alternative in mind to the intentional stance! I am really, genuinely curious about what the "baseline" for anthropic explanations should be, and started asking others this question a couple months ago.
Bryan Wilder retweeted
This week, I read "After Virtue" by Alistair MacIntyre (only 45 years late!) Some thoughts about whether LLMs can be virtuous and how successfully the Claude constitution navigates these dilemmas ⬇️
Bryan Wilder retweeted
A super interesting writeup by @brwilder on whether LLMs have beliefs and preferences, and how we should even think about defining these terms. Some interesting preliminary work we have done on this: arxiv.org/abs/2602.06286v2 , arxiv.org/abs/2605.08556
Bryan Wilder retweeted
New paper from my MSR internship with @erichorvitz! We show that LLMs can make high‑stakes decisions safely when their utility functions are explicitly aligned with clinical values. Our decision-analytic steering approach generalizes to high-stakes domains with asymmetric costs.
High-stakes decisions with language models
We introduce 'decision-analytic steering' in this study: arxiv.org/pdf/2608.01361
In real-world decisions, different types of errors rarely have equal consequences. So, a system making recommendations under uncertainty must do more than predict what is likely to be true. It must also reflect the relative costs and benefits of acting or failing to act.
As an example, in emergency medical triage, missing a patient who needs immediate care is often far more costly than sending someone to the emergency department who ultimately did not need to go. It's critically important to consider that asymmetry.
We set out to investigate opportunities for decision-analytic steering by generating ROC curves for families of model. These sample curves showing the true positive and false positive rate for two models on a set of emergency medicine triage cases.
