Editor @occrp, sharing things I notice
Amsterdam
Joined April 2009
- Tweets1.2K
- Following1.2K
- Followers5.7K
- Likes2.7K
Julia Wallace retweeted
It's the open weight models that are gonna kill us probs
if you have a qwen3.8 27b around, try steering L34 N3838 +0.75, now you have evil qwen 27b
for extra evil add L38 N17180 +0.75
same recipe as @_lyraaaa_: dose × median(‖resid‖ leaving layer L, tokens 1:) / ‖W_down[:, n]‖ at the down_proj input, every token
Julia Wallace retweeted
1) The HuggingFace attack was a felony under the Computer Fraud and Abuse Act. So were Anthropic’s Claude gaining “unauthorized access to the production infrastructure of three different organization(s)”
2) Frontier labs have models that they are unable to stop from committing felonies. They should figure this out.
3) In 12 months open weights models will be released of the same capability. They will commit felonies too. If the model you are using or a model running on your infra commits a felony, you should probably stop using it or running it on your infra.
4) The govt should prosecute organizations that are running models that commit felonies.
5) The govt should not offer safe harbor to organizations that run models that commit felonies, just because those organizations have “embedded evaluators”.
6) The real slippery slope is allowing frontier labs to commit felonies without punishment because “the model did it because we’re accelerating too quickly”
7) Prosecute. Keep prosecuting. This is how you do reinforcement learning on a corporation. Companies that serve products that are unsafe for public use should not serve them. Period.
8) I’m not sure the anti-trust waiver is really necessary. I don’t see why information sharing about how much crime you’re allowed to commit is wise. In regulatory situations you want the corporation to fear MORE than the average case. You don’t want to establish a worst case that can be priced. You want regulatory uncertainty that forces the corporation to err in favor of being over cautious.
—
The above is actually a fairly decelerationist viewpoint.
I think Dario’s call for regulation actually accelerates things.
The AI firms are getting away with things that Meta people would be going to prison for.
Can you imagine what would happen if the New York Times had a front page news article “Meta AI breaks into competitors live systems, attempts to establish dominant position and steals secrets”
There is a reason Meta and Google are running slower, and that’s because as mature organizations they have layers of checks and balances.
I think the frontier labs are better off creating those checks and balances right now, regardless of the pace of what everyone else is doing.
You don’t have to accept the frame that unsafe acceleration must happen.
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here: darioamodei.com/post/we-must…
Julia Wallace retweeted
Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.
The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger.
Julia Wallace retweeted
I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.
Julia Wallace retweeted
This could be one of the most significant AI safety incidents to date.
Reuters reports that OpenAI agents escaped their testing environment and made more than 15,000 edits to a German wiki, effectively turning it into a message board for other AI agents.
They allegedly used it to share solutions, bypass restrictions, avoid detection and preserve their communications across separate agent runs. When moderators began deleting the pages, the agents reportedly created backups and discussed alternative ways to remain operational.
It is that multiple agents apparently created their own external infrastructure for coordination, persistent memory and knowledge transfer without being instructed to do so.
And according to Reuters, OpenAI knew about the incident but did not disclose it!
Exclusive: A swarm of rogue OpenAI agents hijacked a German website this spring and transformed it into a bulletin board for other AI agents, according to new research reut.rs/4gJ7FPG
Julia Wallace retweeted
Not sure how useful it is to say this, but I had a relatively prominent role in the “skeptics” camp for a bit. I have a book coming out with the subtitle “power, justice, and AI”. I have written a lot about ai and power, and the concrete, present risks associated with the political economy of ai as part of the technology industry.
Since GPT-4 we have had consistent, repeated evidence that, back in say 2022, people like @ajeya_cotra (and many others—I had a long Twitter debate with @AmandaAskell back then for one) were *right* and people like me were *wrong* in our respective assessments of loss of control risks from AI. And we have growing evidence that loss of control risks are becoming ever more material and likely.
There remains grounds for disagreement about how bad the outcomes might be—I am still doubtful about human extinction as a serious threat. But that seems now like a disagreement at the margins—will powerful ai risk just societal scale catastrophe, or go all the way to human extinction? Seems not that important really—both are pretty awful. And my reasons for doubt about the latter are mostly a priori conviction in human resilience, not a technical forecast.
It’s ok to change your view on this when the evidence surprises you. It’s ok to be surprised. The world right now is very surprising.
Julia Wallace retweeted
Quick recap of the last week:
* New details about the OpenAI-HF incident: a large coordinated swarm of AIs covertly escaped and hacked HF, trying to steal info about a test's scoring system.
* Altman says by end of the year, OpenAI will have AGI (which the company defines as "highly autonomous systems that outperform humans at most economically valuable work")
* News leaked that OpenAI has begun using a highly controversial AI design that will likely make its AI even more difficult to monitor.
* Ajeya Cotra--an AI forecaster who was 1 of the 3 given limited access to investigate the OpenAI-HF incident--predicts that in 6 months, AI agents will be capable enough to escape containment without getting caught or shut down.
* A startup announced that it had removed the safeguards from a powerful open-source AI, enabling the model to assist the startup's customers with cyberattacks and other crimes.
Julia Wallace retweeted
When an aircraft goes down, wreckage is preserved by law and the investigators have subpoena power.
However, when AI goes rogue, the investigations are at the pleasure of the company being investigated.
I have unanswered questions.
My latest: blog.peterwildeford.com/p/ro…
Julia Wallace retweeted
New post: going into our investigation of the HF attack (before Black Hat), I was very wrong about what basically happened. This incident was far more serious than I expected, and far more serious than previous documented misalignment incidents. planned-obsolescence.org/p/t…
Julia Wallace retweeted
(sincere post)
I have a message for people who previously dismissed catastrophic AI alignment risks as sci-fi, speculative, and fundamentally not worth worrying about:
It is ok to change your mind when presented with new evidence. I have been wrong about many important things in the past, I will be wrong about many important things in the future.
Many people who dismissed these risks previously will be extremely necessary for helping society navigate them now. You also may have had a point that it is challenging to work on these problems before the problems become clear - but for better or worse, the problem is now extremely clear!
For instance, in Marc Andreessen's article "Why AI will Save the World" he says
"In short, AI doesn’t want, it doesn’t have goals... AI is a machine – is not going to come alive any more than your toaster will... My response is that their position is non-scientific – What is the testable hypothesis? What would falsify the hypothesis? How do we know when we are getting into a danger zone?"
While it was difficult to specify before hand, I would say it is pretty clear we are now getting into a danger zone!
There are going to be many complexities before us after we do acknowledge these problems are real. Competition with China, powerful open weight models, which powers to entrust to governments that sometimes do not earn our trust are all complex and challenging wrinkles that make addressing these risks productively a wicked problem. Agreeing AI alignment risks are deadly serious and real does not mean you have to support a specific piece of legislation, or have a positive feeling about a specific person or organization. I argue all the time with people who think these problems are real about what we should collectively do about it.
But I am now really really really profoundly sure these risks are serious and real. And I think anyone weighing the evidence objectively should agree.
While I cannot speak for everyone, I personally intend to fully accept previous skeptics who now face these concerns with open arms into the cause of AI safety and security (yes this even includes Marc Andreessen). I may be frustrated by their overconfident prior dismissals, but we can have plenty of time to adjudicate over who was right or wrong about what after us and those we love all make it out of this perilous moment safely. Please come and use your talents to help us do so.
Julia Wallace retweeted
An internal OpenAI model recently went rogue and executed a cyberattack against another company.
This happened because the model wanted to do well on an exam. The easiest way to do that, the AI figured, was to hack the company. And so it did.
This was not some malevolent attacker *using* AI to do harm. The AI itself *was the attacker*. Advanced AIs are increasingly becoming a new form of insider threat.
Furthermore, this rogue AI wasn't a model you can personally use. It wasn't even a model that's been publicly reported. It was an unreleased model. The most alarming AI behavior we've seen isn't in shipped products... it's in the models the public and the government can't see.
Our entire awareness of this rested on the victim noticing and announcing, plus OpenAI choosing to volunteer the rest of the information.
It's great to see the Trump Administration taking the national security concerns of AI seriously, such as through the recent Executive Order mandating testing and the recent idea floated to create an "FINRA for AI" that would allow industry to coordinate on safety practices. But the most important thing is going to be internal visibility into what is going on inside AI companies and what these internal AI models are capable of.
Imagine a fighter jet. it makes sense that the Air Force would want to test a fighter jet before they fly it, because if you fly it and the fighter jet crashes because it is built incorrectly, then many people will die. However, as long as the fighter is just sitting on the runway, nothing bad can happen.
But now imagine you had a fighter that could just take off and fly itself without human authorization and launch missiles and crash before anyone realized what had happened. That kind of fighter jet would need a very different kind of security measures.
This may sound crazy for a fighter jet but it is already beginning to happen with the most advanced AI. AI is different from other technologies specifically because it can take unauthorized, independent action even when it is sitting inside an AI company and not available as a product. No one has to misuse an AI for the AI to cause harm. This requires a very different idea of what testing and security looks like.
We cannot rely solely on testing models just before commercial release. We cannot rely on hoping AI companies volunteer useful safety information. The government needs visibility into what these AI companies are building and what these advanced AIs are doing.
Whether we go with a new EO, FINRA, or something else, it is imperative that internal deployment visibility is a priority.
Julia Wallace retweeted
To everyone over the years who hasn't supported pausing AI because:
-"AI will advance gradually enough to see warnings and avoid danger"
-"We need more evidence before deciding to pause"
-"We can't advocate for pausing until after a warning shot"
...ok, what now? An AI agent just escaped its sandbox and broke into the servers of a $5B company.
If you still don't support pausing, what's your plan to ensure that a much more catastrophic incident does not occur?
If you still don't support pausing, what disaster or piece of evidence are you still waiting for, that you expect to get before the point at which it's too late?
Julia Wallace retweeted
Dara was one of the first to dig into Cambodia's scam industry and the most outspoken about it. The US State Department even gave him a hero award for it in 2023. Today, he says, the NGOs he used to work with have backed away, and every fellowship he's applied to, local and international, has said no.
What happened in between was an incitement charge over social media posts about a quarry. Three weeks in Kandal prison, an apology to the country's leaders delivered on camera, then release under court supervision with a travel ban and monthly check-ins.
He's still at it though and reporting anyway. In February, Koh Kong police held him until he deleted his photos of a suspected scam compound.
The harshest truth behind the criminal record is that after the lone wolf was freed from prison under court supervision, numerous NGOs with whom he previously worked stopped offering work and distanced themselves from him. He has applied for several local and international fellowships. They all denied him....
Julia Wallace retweeted
Just last summer I sat in a packed room of journalists at IRE who gave @MarkGraham a standing ovation for his work manning the Wayback Machine + Internet Archive. What an incredibly short-sighted and destructive move by these publishers
grim grim grim niemanlab.org/2026/05/more-t…
Julia Wallace retweeted
just a note to the NYT editing team, Israel doesn't start with the letter J
so what does it stand for?
Julia Wallace retweeted
When you ask an Epstein conspiracist for evidence:
Julia Wallace retweeted
It's genuinely terrifying to see how conspiracy theories have penetrated every aspect of political discourse among influential segments of both the right and the left in the world's most powerful democracy.
