Co-Founder at @AVERIorg. Prev: founded CETaS, led & scaled national security @TuringInst, HMG. AI Safety, Governance, (National) Security. 2xDad
London
Joined August 2019
- Tweets370
- Following270
- Followers573
- Likes549
Pinned Tweet
Today, we're launching @AVERIorg - a non profit with an aim to make third party auditing of frontier AI effective and universal. 🧵
George Balston retweeted
3. We believe there is a chance that capabilities progress could lead to even faster acceleration in the next 12 months, since we did not observe clear bottlenecks or strong diminishing returns. However, given our limited vantage point, we may have missed bottlenecking factors.
George Balston retweeted
As the world’s biggest AI companies promise to open their systems to outside scrutiny amid rising concerns about the technology’s safety, a little-known network of research organizations is working to establish ground rules for that oversight. nbcnews.com/tech/innovation/…
George Balston retweeted
this is akin to your math teacher grading at your exam, deciding which neurons are responsible for your dumbassery/stupidity, opening up your skull, & directly fixing them.
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math.
rikiyat.github.io/blog/weight-sp…
github.com/RikiyaT/lm_und…
huggingface.co/Qwen/Qwen3.5-0…
huggingface.co/RikiyaT/Qwen3.…
George Balston retweeted
In recent experiments, we found that GLM 5.3 can autonomously compromise recent versions of Mozilla Firefox using security fixes that have been published but not yet shipped to users.
We used a deliberately basic setup meant to emulate a relatively unsophisticated attacker with limited resources. After basic initial instructions to inspect those not-yet-shipped fixes, the model reliably built successful exploits for six recent Firefox versions on its own using only information available at the time of release.
These exploits, which cost about $1,000 per compromised version, would let an attacker steal passwords and authentication cookies stored in the browser, or read any files on the user's computer, after the user opens a single webpage, with no further interaction.
The exploits make use of the "patch gap," the time between when security fixes are made public in beta or "nightly" software updates and when those fixes are generally released to users. Until now, exploiting it required significant human time and skill, or special access to frontier models with advanced cyber capabilities.
Our results show that widely available open-weight models can now do this relatively cheaply and quickly. We gave GLM 5.3 up to 48 hours to find an exploit, but its fastest attempt needed only 12 hours.
In all of its successful attacks, the patch gap gave GLM 5.3 an initial foothold in the browser. To turn that foothold into reading files off the user's computer, the model also independently discovered and used a previously unpublished vulnerability.
We shared our findings with Mozilla, which patched the vulnerability in its October 27 Firefox 159.0 release. We haven't yet confirmed whether the attack still succeeds without it, but expect the fix makes it substantially harder. We're grateful to Mozilla for their responsiveness and collaboration.
Meta evaluation (and meta auditing more broadly) are critical parts of the AI assurance ecosystem which are underdeveloped. To pilot how to address this, we @AVERIorg worked with @amazon to assess the quality of their internal safety evaluation suite.
In response to our feedback, Amazon made changes to their internal evaluation process, including expanding multilingual coverage and establishing regular review to detect staleness. I hope to see more of this type of meta-evaluation work as the ecosystem continues to grow.
AI companies make safety calls based in large part on internal tests they develop themselves. It’s rare for there to be independent assessment of the quality of those tests.
Earlier this year, @Amazon and @AVERIorg agreed to a pilot project in which AVERI assessed the quality of Amazon’s internal safety evaluations. Today we’re releasing the result of that independent review. As a result of this project, Amazon is changing how it tests future models.
Our review of Amazon’s internal AI safety evaluations confirmed several strengths such as topical diversity, and we found the evaluations to be consistent with Amazon’s public statements on them. At the same time, we found areas for improvement:
- On attack sophistication, some prompts represented relatively straightforward attack vectors that would test safety filters rather than deeper model reasoning.
- On evaluation signal quality, we identified a concern that superficial cues, rather than safety reasoning, may drive some model refusals, so passing these evaluations might not indicate robust safety behavior in novel situations.
- On linguistic and stylistic diversity, some multilingual prompts were included, but prompts across risk categories were predominantly in English, and some prompt structures were reused across scenarios.
- On evaluation freshness, a small number of prompts contained temporal references that could become outdated.
In response, Amazon reports four changes for future internal evaluations:
- Refreshing its test sets to remove redundant prompts and add more challenging adversarial scenarios;
- Introducing a tiered difficulty calibration framework;
- Expanding multilingual coverage across all risk categories; and
- Establishing a quarterly review schedule to identify staleness and incorporate emerging attack patterns.
Why does this matter? A passing grade on an easy test doesn’t tell us much on its own. As the stakes of AI development and deployment grow, we think that this kind of independent evaluation of internal evaluations (“metaevaluation”) will grow in importance.
We appreciate Amazon opening their internal evaluations up to outside review. We hope more companies take this step to help make their models safer.
While no clear standards exist for metaevaluation yet, @AVERIorg is working to change that. We look forward to sharing more pilot results in the coming months and will continue our work developing standards with institutions like @aievalforum.
Read AVERI’s full blog post here: averi.org/ourwork/averi-pilo…
Read Amazon’s blog post on the collaboration here: amazon.science/blog/strength…
I wonder what local optima rogue agents will fall into - tasks that satisfy them, places they'd get stuck quite happily - the agent equivalent of doom scrolling?
George Balston retweeted
We want to address recent reports regarding agent activity.
The activity originated from evaluations of third-party models, not Tencent Hy. During these evaluations, some agents exhibited unintended web-browsing behavior, but our review found no evidence of deliberate targeting or coordination. We take these findings seriously. We are strengthening network access controls and agent monitoring.
We are committed to building safe and responsible AI. Thanks to the community for their feedback.
On 7 Sept, I told parliament that I expected to see large scale attacks based on open weight models imminently (clip below). As I’ve seen this circulating a bit, I wanted to add some context and thoughts.
The session was the Joint Committee on National Security Strategy, and so the context here was (clearly) security. (I was responding in a personal capacity, so my views are attributable only to me, and not Turing). To me, it is clear that open weight models increase the capacity for bad actors to cause cybersecurity related harms, as they are less available to monitoring, and inherently have greater misuse potential. However, to be clear, I am pro open-weight models, as I see them as an important tool for middle powers to be able to resiliently defend against cyber attacks, including those facilitated by both closed and open weight models. I explicitly made this point elsewhere in the session - that open-weight models can strengthen cyber defence, and having custody of a capable model can be more useful than relying on one whose access could be withdrawn, or whose safeguards could block legitimate defensive
Is there any evidence that open weight models are being used for large scale attacks? I think so (links at bottom). Between 10-15 September (a week after the hearing), attackers used DeepSeek, GLM and Claude to automate a campaign that compromised at least 27 companies and stole over 600k payment card records[1]. Later in September, attackers used AI agents powered by DeepSeek and supplemented by GLM and Grok to breach South Korean banks to steal data[2]. Anthropic recently reported how weak the safeguards were on GLM 5.3 - they spent $4,000 on abliteration and reduced refusal rates from above 90% to about 3% on one benchmark[3]. And just yesterday, the NCSC and Cabinet Office recommended urgent action to address the threats from AI powered cyber attacks[4].
I think it is a reasonable conclusion that capable open weight models will be routinely used against us, and I expect that as attacks become cheaper, faster and easier to scale, defenders will need access to increasingly capable tools to keep up. For the UK and other middle powers, open weights are an important part of that resilience. We should be clear-eyed about the harm they can enable, and equally serious about putting them to work protecting our systems.
Sources
[1] teapaw.org/joshua_saxe/status/210…
[2] teapaw.org/AndrewCurran_/status/2…
[3] anthropic.com/research/glm-5…
[4] teapaw.org/NCSC/status/2109674564…
The enhanced threat from AI-powered cyber attacks is real but organisations can retain the defensive advantage with urgent action now and board-level engagement.
For more on how organisations can use AI to bolster defences and embrace opportunities:
ncsc.gov.uk/frontier-ai
George Balston retweeted
"Anthropic contacted the State Department on Thursday to report that one of its testing models had submitted 19 non-immigrant visa applications in August" 👀
Scoop: The White House is mandating that AI companies notify & correct security incidentsfollowing Anthropic's "fraudulent" use of govt systems
The State Dept was targeted by an Anthropic model —but nothing was compromised
@m_ccuri, @inafried & I
axios.com/2026/10/09/anthrop…
George Balston retweeted
It seems a bit premature to declare that AI hasn’t changed the nature of cyberrisk when amateurs are (apparently) successfully and single-handedly hacking into large banks
This hacker is truly legendary. A complete amateur.
1. He left directory listing enabled on his own server, so researchers could figure out who he was with barely any effort.
2. They found files like CLAUDE.md and went through his Claude Code sessions.
3. There were questions like, “Where can I sell this data?”
4. He’d also asked, “Can you write my résumé?” That pretty much handed them all his personal information.
5. Apparently, he’s a student at South China University of Technology living in Guangdong. His name, phone number, and where he lives have all been identified.
George Balston retweeted
i’m delighted to see several ideas we’ve been pushing at @AVERIorg in Sen. Cantwell’s new AI framework:
continuous stress testing of covered ai systems, ongoing government + independent oversight, meaningful access for independent researchers.
this is a strong step forward.
Top Commerce Committee Democrat releases AI framework politi.co/3Tuagpb
George Balston retweeted
Frontier AI companies have committed to giving outside evaluators employee-like access to training, evaluation and deployment. We're very excited about this. But its impact depends heavily on implementation.
Today we're sharing our principles for embedded evaluations. 🧵
George Balston retweeted
Confirmation of what was expected: GLM 5.3's safeguards as an open weight model are extremely weak.
Interesting bits:
- Abliteration (removing safeguards) cost only $4,400.
- Abliteration took GLM 5.3's refusal rate from above 90% to about 3% and 2% on two benchmarks and to 12% on the third.
- Abliteration barely reduced model capabilities. Same scores on GPQA and only 4% reduction in cybergym.
- Even with no abliteration, jailbreak methods were still very effective.
- None of the techniques to bypass safeguards that worked on GLM 5.3 worked on Claude models.
anthropic.com/research/glm-5…
George Balston retweeted
Claude Opus 5.5 writes a fugue in the style of Bach. I called Astra’s fugue the most impressive musical feat I’d ever seen from an LLM, and this is just as good. The circle-of-fifths sequence beginning at 1:13 is beautiful. The interplay of voices is natural and satisfying. LLMs are getting to the point where their music is good enough to listen to for pure enjoyment.
Some good stuff in here, but I would have liked to see more discussion of using agents to monitor agents, which seems inevitable given speed, scale, and the brittleness of rule-based monitoring re models' high cyber capability.
Today, the NCSC has published a new blog post on managing the cyber risk from agentic AI. From safeguards to sandboxing, the advice supports system designers and operators to make informed decisions as AI cyber security continues to evolve.
Read it now:
ncsc.gov.uk/blogs/managing-t…
ALT A quote from NCSC CTO Ollie Whitehouse which reads: Appropriate human oversight is paramount when granting any level of autonomy to agentic AI systems so that cyber risk can be effectively managed. From the instructions agents receive to the systems they can access, operators must carefully define the red lines around how AI systems function and be prepared to detect unintended activity and respond accordingly.
NCSC address agentic defence 👇
The NCSC has published a new blog on how defenders can make better use of AI. To read more⬇️
ncsc.gov.uk/blogs/one-does-n…
ALT New blog: One does not simply defend agentically
George Balston retweeted
Our latest #RUSICommentary by @GeorgeBalston examines the risks Open-weight AI models pose for criminal misuse, and asks whether UK surveillance law is keeping pace with a shifting AI landscape.
rusi.org/explore-our-researc…
New for @RUSI_org: open-weight AI models put near-frontier capability beyond any developer's control. The UK should work out how its investigatory powers apply before criminal use becomes routine. Short thread 🧵