A ranking of takes on embedded third party evaluation this past week, from worst to best: [contentless sneer against outgroup] METR bad because [huge Sankey diagram]. fact checking? timelines of relative investments that make causal sense? determining which numbers are large or small fractions of other numbers? sounds like some EA bullshit to me. just look at this diagram with all those curved lines, you can SEE the nest of snakes. METR bad because [an actual specific chain of actors who have some pairwise relationship to each other that ends with someone it would be bad for them to have strong COIs with, e.g. "METR once received funding from an org who once received funding from a person who once gave funding to a now-frontier AI company"]. It is beneath my dignity to explain how this actually affects the decisionmaking of METR; you should vaguely perform a mood affiliation and keep scrolling. Don't think too hard. guys guys guys you HAVE to embed my company/organization into the labs. I have been an unwavering supporter of external embedded auditors since 7 minutes ago when I saw this essay taking off. Look at this thing we published once that sort of looks like AI safety if you squint! Fear not, I can assure you that I have never done anything altruistic in my life and if I had I would have been ineffective at it. I observe that [politicized actor] has had [bad take]. Let me use this correct observation to score points for my side and politicize the situation further. === zero point: takes beyond this line are better than logging off === Dear [person with insane take], here is an earnest explanation of why you are wrong. METR is bad/problematic because [an actually reasonable concern, like greater cultural overlap with Anthropic than OAI or the pressure for individual employees to be on good enough terms with labs that they could later get hired], with no further suggestions for what to do about this issue. Tweets which simultaneously acknowledge that (1) more social independence from labs would be good and (2) almost everyone competent is socially connected to labs, even if they don't propose solutions. [literally any take that engages with object level assessment of AI companies done by a third party org like METR, SecureBio, Guidelight, Redwood, Nightingale, etc] === current discourse frontier: takes beyond this line are better than anyone has yet posted === I have [actually reasonable concern] with METR/Redwood/etc. To address this concern, we should do [concrete proposal that makes any sense and would solve the problem]. I am founding a new third party evaluation org / pivoting my existing org to do more third party evaluation. We're doing [thing which is both (1) not identical to METR (2) remotely useful for assessing AI companies]. Our initial work will be on [concrete specific thing that would help]. METR has problem X. We should instead use preexisting organization Y, which does not have problem X and has comparable expertise at assessing loss-of-control risks and misalignment incidents inside AI companies, for instance [past work they've done of similar quality]. (this would be an amazing take but it is impossible to post because no such org exists)

If you are good at investigating the behavior of powerful AI models, today is a great day to apply to METR: metr.work

I think there's a ton of informative stuff in this post! Highly recommend reading.