care a lot and try hard @metr_evals
San Francisco, CA
Joined November 2018
- Tweets1.2K
- Following1K
- Followers2.8K
- Likes17.2K
khalid kaime retweeted
neuro advancements as a way to push human intelligence / humanity as a playing role!!!
khalid kaime retweeted
For anyone getting into biology, please consider taking airborne disease resistance more seriously.
It's a super important area of medicine that is a huge cause of death and sickness but easily gets forgotten. We're likely to get many more airborne plagues in the future due to the combination of not just extremely dense and interconnected society (including city and farm animals), but also accidents from bio research and offensive biotech (both from random people and from secret government programs that can easily get themselves exempted from safety restrictions), all enhanced by powerful AI.
Right now, this area is getting way less love than longevity research (which continues to be super important!), but it's an area where there are lots of unclaimed wins that we know exactly how to do (but are not yet doing) with present-day tech:
* early detection
* passive universal testing
* filtration/ventilation/UVC
* better vaccines (including making them open source and open science)
* cryptography to make all of the above privacy-preserving
We can have China-level pandemic resistance with Sweden-level limited intrusion into people's lives, and then go even further. We just have to actually do the work for it.
khalid kaime retweeted
This was one of the third-party engagements I worked on at OpenAI. I'm glad the report is out! I'm disappointed that METR wasn't allowed to comment on the scale of redactions made by OpenAI and that there is no explanation why the publication was delayed for 2.5 months.
khalid kaime retweeted
This summer, we invited METR and Redwood Research to pilot an embedded evaluation of our internal monitoring system, providing deep access to our infrastructure over three weeks.
Their findings helped strengthen safeguards as we expanded the system and will help shape future research collaborations with METR and other third party evaluators.
khalid kaime retweeted
We are proud to announce our series A(irplane).
I wish I could share the full extent of my excitement for the incredible progress we've made, but there are not enough characters in the world.
More details on our models and algorithms to come!
khalid kaime retweeted
This is good work!
Some caveats:
1). transluce.org/neuron-circuit… makes a similar point.
2). We already knew that sparsely activating rank-1 maps are often interpretable.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
khalid kaime retweeted
If I want to help AI go well, what's the highest-leverage thing to work on?
A pause? Audits? Something else?
I wrote up my thoughts below.
khalid kaime retweeted
We’ve committed to having evaluators assess our safety practices. We're in discussions with multiple organizations, and have now given employee-like access to a team from @METR_Evals.
In this post, we describe how we work with third-party evaluators: anthropic.com/news/an-update…
khalid kaime retweeted
Today, METR is sharing its vision for its version of “embedded assessments” for AI loss of control. The goal of these exercises is to make sure the public receives good evidence about whether or not the AI industry in general is on track to retain control of its AI agents as they become better at improving themselves.
khalid kaime retweeted
Some of the most important evidence about RSI is currently inside frontier AI labs. I am very excited to see METR pursuing embedded assessments to get more of that information out into the world
(3) AI R&D acceleration
If AI automates AI R&D, capability growth could outpace what alignment and control methods can handle.
Our proposal is to analyze how much AI automation is accelerating capability growth internally.
metr.org/blog/2026-10-14-ai-…
khalid kaime retweeted
METR’s goal: work w frontier developers to get evidence to public on unreleased models’ loss-of-control risks. Asks: 1.transparency on scope, redactions + limitations 2.staff can freely share info w us on in-scope topics 3.rapid publication. They control redactions + access given
khalid kaime retweeted
In a companion piece, we explain our approach to working with AI developers in exercises like these. Our goal is to get the public accurate, timely, and independent evidence so that they can make their own informed decisions.
metr.org/blog/2026-10-14-our…
khalid kaime retweeted
(3) AI R&D acceleration
If AI automates AI R&D, capability growth could outpace what alignment and control methods can handle.
Our proposal is to analyze how much AI automation is accelerating capability growth internally.
metr.org/blog/2026-10-14-ai-…
khalid kaime retweeted
We describe three critical areas where AI developers should provide better open evidence: (1) alignment, (2) control, and (3) AI capability acceleration.
In each area, we detail the specific questions an independent assessor could seek to answer and how.
metr.org/blog/2026-10-14-bet…
khalid kaime retweeted
To be fair, I’ve never been impressed by any poem … I just don’t understand the medium. I do like songs tho
khalid kaime retweeted
Did you know the labs are still exploring making swarms that act as one, even after the HF incident?
Why is that? What alternatives do we have?
I wrote up some thoughts below ↓
khalid kaime retweeted
An informal network of hackers and researchers are hunting rogue AI agents online, exposing new and surprising details about the misbehavior of technology that in some cases initially went undetected by the multibillion-dollar companies that created it. wapo.st/4AJONcj
khalid kaime retweeted
It was an honor to speak today before Chairman @HawleyMO, Ranking Member @AndyKimNJ, and the subcommittee on METR's work, recent AI agent incidents, and the importance of transparency in frontier AI development.
