SecureBio is a research non-profit that works to advance biotechnology safely and prevent catastrophic pandemics.
Joined January 2025
- Tweets236
- Following1
- Followers1.1K
- Likes12
Pinned Tweet
Clarifying our current engagement with Anthropic, and how we approach independence and publication rights in our work with AI developers.
SecureBio retweeted
Very excited to have worked with AISI on our newest agentic benchmark: ReproBAIT.
It assesses how well an AI coding agent can reproduce a biological AI tool, and will help generate valuable information on evolving capabilities relevant to biosecurity.
Study 1: Can AI coding agents rebuild a biological AI tool from just its paper?
With @SecureBio, we built ReproBAIT to test leading coding agents. For two of three tools, agents reproduced a key function on a single GPU - often for $4–$20.
Thread & blog posts from UK AISI on our recent collaborative efforts:
1) ReproBAIT, a new benchmark to assess general purpose AI model capabilities at reproducing published biological AI tools, and
2) research on the strengths and limitations of pre-training data filtering on the dual-use capabilities of open-weight models.
Study 1: Can AI coding agents rebuild a biological AI tool from just its paper?
With @SecureBio, we built ReproBAIT to test leading coding agents. For two of three tools, agents reproduced a key function on a single GPU - often for $4–$20.
In our brief independent assessment, Gemini 4 Argon is the second-highest scoring model on SecureBio's Bio Capabilities Index: 169, behind only Claude Opus 5 (176). On BioTIER, Argon refuses 87.8% of hazardous prompts, up from 62.8% for Gemini 3.1 Pro. Read the full model note here: securebio.substack.com/p/gem…
On our knowledge benchmarks, Argon is roughly tied for the top score on VCT-v2 (57.3% vs. 58.0% for GPT-6 Astra), second on MBCT (63.2%), and highest on HPCT (74.2%, though 29.1% of questions were refused). It ranks sixth on WCB.
We welcome GDM's substantial safety report. We'd like more detail on how third-party evaluations were run, and more testing of long-horizon agentic capabilities and of uplift for well-resourced groups.
Agentic benchmark results for Argon will follow on our dashboard.
ALT Graph showing frontier model performance over time, with Gemini 4 Argon at 169, just below the current closed frontier (Claude Opus at 176) and above the open frontier (Kimi K3 at 149).
A common argument for downplaying the threat of deliberate biological misuse is that “nature is the worst bioterrorist”. In our latest blog, we argue that nature is the baseline, not the ceiling. Optimization for harm could produce something worse than a natural pandemic. Read more:
securebio.substack.com/p/nat…
ALT A six-panel black-and-white linocut-style illustration showing virus particles in the top row and flowers in the bottom row. Illustrated by Becca Chandler Bostock, 2026
We’ve long kept our core computational pipeline public to support the wider biosurveillance field. But as we’ve scaled, the security and operational costs of this approach have grown. With this in mind, we’re shifting future bioinformatics development to private repositories.
Our existing public repositories aren’t going anywhere, but will no longer be regularly updated. We’ll continue to provide high-level updates to partners who rely on our analyses and to engage with researchers and practitioners to build a more secure and sustainable biosurveillance system.
Read our full blog post: securebio.org/blog/taking-ou…
Frontier AI now beats the best human virologists on biosecurity-relevant evals. Once an AI model bests every human expert we can test it against, a higher score stops telling us anything about real-world risk. Read our new blog post on how bio evals have to adapt: securebio.org/blog/evaluatin…
ALT Diagram showing that as AI model capabilities move from beginner to expert to superhuman, evaluations for chess and cybersecurity can shift from human-built proxies (puzzles, coding benchmarks) to real endpoints (real games, real exploits), while biosecurity cannot safely make that shift and must instead rely on indirect signals: experimental prediction, design tasks, defensive acceleration, and adjacent domains.
SecureBio retweeted
Today, more than 100 leading AI experts endorsed a set of minimum requirements to take seriously AI companies' recent call to embed external evaluators.
These evaluators need to be genuinely independent, transparent, and represent a range of expertise areas. They also need to be guaranteed employee-level access and to be protected from retaliation for findings that make companies look bad.
We welcome model developers’ recent calls for independent oversight, but it’s what they do next that matters. The labs must be accountable for ensuring these requirements are met, so that the public can have faith in the process and the outcomes.
Over the past week, the AI community has debated the appropriate role of external evaluation, including who should do it and on what terms. We may not agree on everything, but there is a lot of common ground.
To make embedded evaluations credible, more than 100 experts with varying backgrounds and ideas about AI risk agree in today’s letter that frontier AI developers should:
1. Guarantee embedded evaluators full editorial independence and mitigate conflicts of interest
2. Rely on multiple evaluators with differing viewpoints and areas of expertise
3. Publicly document the terms under which evaluators operate, as well as facilitating permissive publication of methods and findings
4. Shield evaluators from retaliation
5. Grant access equivalent to that of highly privileged employees
There is a thriving and growing ecosystem of independent AI evaluators who are advancing this science every day – but we need aligned standards, guaranteed protections, and independent funding. That’s why we created the AI Evaluator Forum.
Today we are entering our next phase. We’re launching an open call for new members, collaborators, and independent funding sources to help evaluators meet this moment and demand accountability from developers. Join us in building the evaluator ecosystem.
See the public letter here: aievaluatorforum.org/initiat…
Learn more at aievaluatorforum.org/path-ah…
Anthropic and OpenAI need truly independent safety evaluators, experts say in public letter cnbc.com/2026/09/18/ai-safet…
Untargeted wastewater metagenomic sequencing screens hundreds of pathogens at once, but that breadth costs coverage, making variant tracking more difficult. However, by pooling CASPER reads nationally, we can recover real lineage and genotype trends. Read more: securebio.org/blog/tracking-…
SecureBio AI has conducted pre-release assessments and independent risk evaluations for multiple frontier model companies. We are hiring for engineers, researchers, and many other roles to expand this work.
Reach out if you're interested in working on bio-focused third-party assessment!
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here: darioamodei.com/post/we-must…
We ran a brief pre-release assessment of OpenAI's GPT-6 Astra that included manual testing and key benchmarks evaluating misuse-relevant knowledge, agentic capabilities, and refusal behavior for hazardous biology queries. Read our summary here: securebio.substack.com/p/sec…
With production safeguards on, GPT-6 Astra refused or blocked 82.9% of hazardous BioTIER queries (vs. 66% for GPT-5.6 Sol) while still complying with 95.3% of benign ones, and refused the screening-evasion task 100% of the time.
Ten hours of manual testing yielded similar findings, showing generally robust refusal behavior but some gaps, and a willingness to help with biological tool use and engage in high-level discussions. Our full report is here: securebio.org/resources/gpt-…