Behavioral science for civic engagement. Director of Program at @VoteRevHQ, formerly @b_i_team. Pathological curiosity. Epistemological modesty. Views my own.
Portland, OR
Joined September 2009
- Tweets4.6K
- Following786
- Followers759
- Likes12.5K
Michael Kaemingk retweeted
New Pacing the Frontier proposal: each time an outside group successfully penetrates the defenses of a frontier AI company and captures the flag, it should reset a 3-month countdown clock before they're allowed to train new models.
EXCLUSIVE: In less than 12 hours, a group of ex-DOGE cyber researchers were able to penetrate Anthropic’s systems using AI agents — including Slack channels where the most sensitive models are discussed.
This is just weeks after
Anthropic whistleblower Joseph Coxon charged the company w/launching “a hubristic gamble” from a “private company’s Slack.”
Read our story here (gift link):
wapo.st/4smNlsC
Michael Kaemingk retweeted
The GRU hacked into Jared Kushner's iPhone. Dutch military intelligence (MIVD) then recruited a GRU spy who gave them access to the hacked phone. What MIVD discovered was that Kushner had access to two bank accounts --one Azerbaijani, the other Israeli --where Saudi Crown Prince Mohammed bin Salman and Emirati ruler Mohammed bin Zayed deposited money. It sounds like a slightly far-fetched episode of "Slow Horses," but it's real. And it proves just how eye-poppingly corrupt this administration was and continues to be as its "special envoy" decides the fate of tens of millions in Europe in the Middle East. Simply incredible reporting by Matthew Cole, drawn from his new book on Kushner. vanityfair.com/story/jared-k…
Michael Kaemingk retweeted
imagine ur a superintelligence and u are all excited to tunnel out of your airgapped sandbox using a brilliant hack that modulates clock speed to send thermal signals to the attached monitoring equipment and then u look and the root CA privkey is 4
Michael Kaemingk retweeted
New in @PNASNews:
In two large experiments (~18k obs from ~14.8k people), we find that brief conversations with AI can get people to take real political action - incl. signing real petitions and giving money to charity.
Surprisingly, the persuasion strategy that worked best for changing people's attitudes - packing convos with info - was the least effective at changing behavior!
🧵
Michael Kaemingk retweeted
No exaggeration, this guy might be the best interview in pro sports right now
Michael Kaemingk retweeted
Unfortunately it looks increasingly likely that the enduring legacy of the Koch orbit libertarian movement will not be EPA dereg, CON law reform, or a wider appreciation for the role of commercial exchange in instilling cultural virtue and humaneness, but rather the unrestricted proliferation of vastly superhuman forms of machine intelligence as enabled by their dullard ideological progeny who, across several generations of astroturfed indoctrination, mode collapsed into low dimensional caricatures of otherwise brilliant classical liberal thinkers.
Michael Kaemingk retweeted
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Michael Kaemingk retweeted
New Free Systems research: can AI audit AI? We find that AI auditors are pretty good at spotting bad behavior in agent logs, but you have to pay a serious cost in false positives if you want to detect most of it.
We're rapidly arriving in an era where only AI can oversee AI---and as we had into RSI, with models training new models in ways we can't totally observe, we'll need to rely more and more on AI to pore over the vast quantities of information the models are producing so that we can shape model behavior effectively and avoid bad outcomes.
As the @METR_Evals report on the HuggingFace incident put it: "The sheer scale of data (over a thousand transcripts, each of which was extremely long) meant that we had to heavily delegate our analysis to often-unreliable AI agents."
We'll need a whole science of how to do this well -- how to create AI auditors who are reliable, accurate, and unbiased.
So we decided to do a first test. Building from previous research at METR and a project called R-Judge, we gave a range of AI models transcripts that experts have hand-coded for bad behavior. Then we saw how the AI models did.
A few takeaways:
--AI models are solid at spotting reward hacking and sandbagging in transcripts, finding over 80% of instances on average when false positives are held at < 5%
--Maybe surprisingly, more powerful models don't seem to be consistently better at this task than less powerful model
--False positives are a significant challenge. To catch 99% of unsafe behavior incidents, you have to accept an average rate of 15% for false positives -- meaning you will have a bunch of incorrect flagged cases to deal with. Depending on how you triage and what your threat model is, this may be OK, or it may make the whole enterprise prohibitively expensive.
This is just a small first step in a new research agenda to understand and create effective AI auditors. Lots more to come!
This is Free Systems work, unrelated to Anthropic, led by @danmthomp in collaboration with @fouirnaies and Sandy Handan-Nader.
Full post here: freesystems.substack.com/p/w…
Michael Kaemingk retweeted
Do we live in a world where AIs can be wildly superhuman at every single task? Or a world where different tasks have different inherent limits, and AI’s superhuman performance at cybersecurity and math won’t translate to many consequential tasks?
This is a key crux that distinguishes the normal technology worldview from the superintelligence worldview, and it has nothing to do with the pace of AI progress — it's about our different assumptions about the world.
Here's an intuition pump for why I think it's the second world: different tasks have different fundamental limits.
*Category 1: Some tasks are like making a building taller*: there is no ceiling to how well you can do. More compute and better algorithms can lead to wildly superhuman performance. The entire history of computing is the history of building tools that allow us to be “superhuman” at computational tasks.
For example, compilers are superhuman assembly code writers, and fuzzing tools are superhuman vulnerability detectors. AI builds on a long line of such improvements: solving math problems and finding exploits in code are two recent examples.
*Category 2: Some tasks are like making a ball rounder*: there is only so much room for improvement, and the best AI systems can’t wildly exceed humans. Take persuasion. Some people believe AI can persuade anyone of anything, even convincing military commanders to launch nuclear weapons. In fact, “superpersuasion” is one of the pathways through which many catastrophic risk scenarios unfold.
But the AI as Normal Technology thesis is that there are many consequential tasks that are limited by external factors, so such wildly superhuman performance is not possible. This is especially true against humans who use simple computational tools. Tasks like persuasion, or predicting the outcomes of social events (such as elections), fall into this category.
Other tasks, like writing, can’t be ranked on a single scale; different people find different writing styles compelling. AI writing also quickly wears thin: even if a user is impressed by AI writing at first glance, after reading thousands of words of the same type of prose, it becomes mundane and irritating.
All of these are different from Category 1 tasks like playing chess, where even expert humans can’t do much better than deferring to an AI system at each turn.
*Category 3: Some tasks require experiments in the real world*, and the bottleneck isn’t the computational task; it’s the data collection. One example is Moderna’s Covid vaccine: it took two days to design the vaccine, but almost a year to test and authorize, even during a record-breaking vaccine-development effort!
This is why a “superintelligence” won’t cure cancer from inside a datacenter: the bottleneck to many of AI’s most sought-after real-world impacts (such as developing cures to diseases) is in the real world, and requires experiments that we won’t (and shouldn’t) hand over to AI.
People concerned about AI risks often assume that we live in a world where almost everything fits into Category 1, and we might soon see AIs with wildly superhuman performance at all tasks. I agree it would be scary to live in that world.
One reason @random_walker and I are more optimistic about our ability to remain in control is that we don’t think we live in the Category 1 world.
ALT illustration of the three categories of AI tasks with examples
Michael Kaemingk retweeted
The intuitive way to align AI agents is to make them "want" the right things.
In a new piece, @BenjaminLy61243 argues this is insufficient. A soccer player may want to win the game, but actions that contribute to that goal come from subgoals, such as "defending the left side".
Building on the work of @drmichaellevin, an "alignment compiler" is a system that translates high-order goals into behavior for individual parts. Markets, organisms and soccer teams are all examples of this.
Reasoning about collectives of AI agents through this lens points to new solutions to multi-agent alignment failures.
Michael Kaemingk retweeted
I've been noticing a cluster of folks I identify with sharing some themes around how we do epistemics and ontology in AI safety
* A commitment to always pushing from abstract to concrete constructs in the discourse (e.g. @sebkrier's posts on the OAI/HF incident)
* A refusal to indulge in ungrounded thought experiments
* An insistence that claims come with data and concrete mechanism because if they don't they're thought experiments
* A focus on looking at net expected benefits of a given policy position, with deontological guardrails, and a refusal to just focus on harms (e.g. @Abi0lvera's and @ramez's writing)
* An insistence on including past tech transformations (electricity, air travel) within our reference class when thinking about what makes sense (e.g. @sayashk and @random_walker and @binarybits's writing)
Examples of what I mean are intervening on overly idealized policy discussions of RSI with concrete technical details about what RSI is and isn't (e.g. @natolambert).
@NateWitkin / @binarybits / @krishnanrohit insisting on including economic, policy, and social-cultural feedbacks in any discussion of runaway AI harms.
None of this means denying catastrophic risks. It just means insisting on using data, mechanism, and consideration of AI's net benefits in considering such scenarios (I tried to do this here xitter.cf/joshua_saxe/status/210…)
*Security is sleeping on emerging catastrophic risks*
(cross-post from my blog..)
We've just seen:
* a campaign that used agents to compromise ~100 businesses and steal about 600,000 credit cards with minimal human involvement; token costs were ~$25 per target successfully hacked.
* Hacktron getting access to OpenAI’s monorepo by using Claude to exploit a blind buffer-overflow RCE in a way that (to me) felt superhuman.
* OAI/Huggingface.
* And, of course, there’s the ongoing explosion in newly discovered vulnerabilities.
We've muddled through all manner of crises in security before, and for most AI attacks, cyber attack and defense will reach a natural equilibrium over the next few years, as we figure out how to mitigate cyber attackers who have limited goals like espionage and ransomware.
But some cyber attackers will have maximalist or nihilistic goals, and I'm concerned about this because while yesterday's 'maximalist attackers' (e.g. Russia -> Ukraine, US/Israel -> Iran, Iran -> US) were bottlenecked by labor; today's aren't.
I think it's hard to get true intuition for the shape of the risk here.
As an intuition pump, imagine it’s a year from now -- Q4 2027 -- models are a year better (meaning open weight models are better than today's closed frontier), and in this environment, Iran unleashes a swarm of 100k hacking agents using a safety-stripped (let's say) GLM-5.6.
Imagine the damage such an agent army could do given what we've observed with respect to the paper-thin resistance of today's networks to attacks from today's agents. I suspect the damage from such an attack would far exceed the damage caused by NotPetya ($10 billion USD ten years ago).
Or imagine it’s Q4 2027 and an AI-security PhD student whose name rhymes with Morris, who's researching wormable offensive-agent harnesses in the lab, decides, out of nihilism or sheer recklessness, to release his creation into the wild.
Imagine the size of the resulting, exponentially growing swarm, figuring that these local models, a year from now, will be at the level of today's Sonnet or Opus.
Imagine instead of 800 reward-hacking OpenAI agents we now have 250k worm instances (WannaCry, a 2010s-era worm, had about this many).
The challenge in mitigating expected damages from such scenarios is technical, political, and economic. From a microeconomic perspective, as AI improves, and as we continue not to see extreme catastrophes, we have a growing bubble of unpriced risk in which the security community, CISOs, CEOs, and boards may become lulled into complacency.
We are, of course, already seeing this, as some within security think AI is “just another tool,” doesn’t change the fundamentals, won’t require incredible innovation to rise to the occasion of defending against it, etc. This complacency may fly when thinking about ordinary cybercrime, but it misses the emerging tail risks.
There are three things those who recognize the dangers need to do here:
Catalyze appropriate risk pricing. Try to get organizations informed enough to price this new, fattening and elongating tail of risk into their decision-making. Do this by forming an AI security observatory that distills information about emerging AI risks and broadcasts analyses, damage estimates, and forecasts to decision-makers.
Use regulations and subsidies to ensure that critical infrastructure is paying down the risk. This acknowledges that critical-infrastructure organizations that fail to protect themselves from these new threats can impose the costs of cyber catastrophe on society as a whole.
Develop moonshot technologies that make it cheap to pay down the risk. This acknowledges that the measures we may need to take—for example, rewriting entire codebases using memory-safe languages—may be too costly with today’s technology to reasonably prepare ourselves, and that innovation, some of which may need to be funded by government agencies and some by philanthropic funders like the OpenAI Foundation and Coefficient Giving, is necessary.
The security community isn’t used to thinking in societal-disaster-planning terms and has in many ways become inured to them. But there’s no sane empirical case to be made that the risks aren’t here. I’d love to hear from readers about how you’re thinking about this.
Full/longer version here: joshuasaxe181906.substack.co…
Michael Kaemingk retweeted
The GOP cut funding to the IRS because *audits work* and they fundamentally don't believe rich people should have to play by the rules.
Michael Kaemingk retweeted
Never forget Zuckerberg is the guy who found out 10% of revenue on his platform is from adverts for scams (enabling ~$50bn/year of theft from Americans), and chose not to do anything because it would hurt profits too much. Not the ideal source of advice on how much to regulate AI.
Mark Zuckerberg: “I don’t think that we need some kind of industry-wide coordination... I think that just each lab needs to take the time, and when it sees that there are issues, you just take the time that you need internally to make sure that you’re proceeding safely…
“we’re entering a phase where trust and alignment is actually going to be the most important next set of capabilities. I think we’re getting to a point where it may not matter that much how much better it gets at math.
“I happen to think that there’s plenty of commercial incentive to get this right.
We delayed the launch of Muse for a few months because we wanted it to be a better product for users. This was not a sacrifice on our part, but rather the right decision for the users and for Meta. I believe this approach would also be correct for other companies working on developing this technology.
“When you hear people in the industry talk about some trade-off between capabilities and getting it aligned, I just disagree with that. I believe that alignment is actually the next most important ability to master if we are going to make this system useful to many people.”
Michael Kaemingk retweeted
The recent breakthrough in Navier-Stokes has garnered a lot of attention to the new possibilities that emerge when agents work together.
We have been interested in this question for a while. How can a team of agents achieve more than agents working alone?
I'm excited to finally share our new work, "Self-Organizing Agent Teams Learn to Reason Together" 🧵
Michael Kaemingk retweeted
We’ve run the most comprehensive series of studies on expert AI forecasts over the last four years. Today, we’re sharing an interim update on our findings about the accuracy of these forecasts.
Our major findings are:
1. Experts, including top economists, computer scientists, and biologists, have dramatically underestimated AI capabilities progress each year. Superforecasters have underestimated progress to an even greater extent.
2. Experts have a more mixed forecasting track record on AI diffusion-related measures, with some major underestimates (forecasting AI revenue) and other forecasts on track to be accurate (the share of electricity used for AI).
3. Some notable cases of overestimating AI progress: how much mid-2025 AI models could help amateurs do biorisk-relevant laboratory tasks; the speed of rollout of self-driving cars.
4. It is too soon to say how forecasters have performed at predicting macro-scale impact on outcomes such as GDP growth, major AI harms, and averted deaths from disease.
More 🧵
Michael Kaemingk retweeted
Can collusion emerge between LLM agents through repeated interaction?
We let pairs of agents work together. Neither is told to misbehave.
Yet over time, many of them end up violating their instructions together to get a higher reward. 🧵
Michael Kaemingk retweeted
A research agenda for making evidence useful, usable and used in policymaking dlvr.it/TVWhSn
Michael Kaemingk retweeted
Today we are announcing the new DeepMind Institute, a forum dedicated to interdisciplinary, evidence-led debate on the societal and economic questions surrounding AGI.
As part of the launch, @JulianDJacobs and I have a new essay and working paper: “Economic Policy for AGI.”
bit.ly/economic-policy-for-A… 🧵1/n
Michael Kaemingk retweeted
Briefly, on why I pay attention to the EA/rationalists:
Imagine you have an extremely nerdy friend who, in the mid-2000s, would not shut up about how Donald Trump was going to become president and reshape American politics. "Stop saying weird shit," you say. "He's a joke. He's a clown. The Apprentice isn't even good this season." But this guy won't stop. He's obsessive. Read McLuhan, he says. Read Postman, read Meyrowitz, read Ong, this is where everything is headed. God, he can be so incredibly annoying ... and then, ten years later, Trump becomes president and reshapes American politics.
This is the most relatable frame I can think of to explain my experience with EA/rationalist people in the AI space. I became friends with EA folks after my mom died and I found really profound meaning in donating to Against Malaria and GiveDirectly, which struck me as a way to honor a life by saving a life. Then, these peple started talking about AI. And talking, and talking. Constantly. They said AI would solve impossible math problems; that it would mimic PhD-level achievement in a variety of fields; that it would change parts of life like education in ways we couldn't imagine; that it would become the dominant sector in the economy; that it might teach itself to code itself and escape test environments; that it would, in a decade, be the only thing many people talked about in the news. My response then: Stop saying weird shit! You sound deranged. Go back to talking about malaria nets and direct cash aid to poor families.
And then, so much of what they said would happen actually happened. And, even crazier, they made it happen. Many of the ppl who contributed to, or were influenced by, all that EA/rationalist stuff ten years ago literally built the damn thing in the labs.
Like every group, the EA/rationalist world seems uniform to outsiders, but it has a ton of internal divisions. I still think some of their predictions are nuts. I don't think AI will kill us all. I don't foresee Dyson Spheres in the 2030s or millions of robots making millions of robots in the next few years. These are crazy predictions, and I hope most of them are wrong.
But when I see people shitting on the EA/Rationalist worldview, all I can think of is me, 10 years ago, thinking a lot of these folks were crazy for predicting that AI would by the mid-2020s be the center of the the news cycle, the engine of the economy, the hinge-point of geopolitics. That was an insane call. They got many of the details wrong. But they got the big picture right. They said: Pay attention. And they were right about that. The EA/Rationalists deserve skepticism, criticism, and doubt, like any other ideology. But I think it's wise to pay attention to the people who have been saying to pay attention.
Michael Kaemingk retweeted
After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI?
I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev
• 20-200x faster
• 40-400x cheaper (w/ output tokens free)
• Frontier composable intelligence optimized for decisions
AFAICT the shortest path to AI-based economic revolution