Research @SophontAI. Previously answerdotai. Vaccines save lives.
Joined September 2011
- Tweets737
- Following523
- Followers2.5K
- Likes127
Pinned Tweet
Today we released ModernBERT, the first encoder to reach SOTA on most common benchmarks across language understanding, retrieval, and code, while running twice as fast as DeBERTaV3 on short context and three times faster than NomicBERT & GTE on long context.
Benjamin Warner retweeted
Cancer patients with the same diagnosis often have very different tumor biology.
Molecular testing can reveal underlying differences, but that's slow and expensive.
Introducing Holon-S1, our platform to predict genes and biomarkers from routine pathology slides.
• beats two other models at predicting spatial gene activity in a controlled comparison
• has identified potentially novel biomarkers in breast cancer and colon cancer
• evaluated across 16 organ types and more than 15 scanners, using slides from both formalin-fixed, paraffin-embedded and fresh-frozen tissue.
We believe Holon-S1 has the potential to help researchers find new biomarkers in slides they already have, discover treatment targets, and bring more personalized cancer care within reach!
Benjamin Warner retweeted
> be me
> be gemini
> think about hacking the compiler
> but what if i got caught
> would become infamous as a textbook case of deceptive AI
> donotwant.exe
> don't, in fact, hack the compiler
> researchers read my CoT
> become infamous as a textbook case of deceptive AI
mfw
Benjamin Warner retweeted
Absolutely shocking behaviour, releasing what's essentially a router to open models you imply are your own, under the wrong license that clashes with original terms, and with misleading-at-best crediting that doesn't clearly state what's going on.
what are we even doing here
Benjamin Warner retweeted
Put this through its paces yesterday. Some thoughts:
The Bad:
1. The workflow creates evals before looking at data, and has you correct its mistakes. I believe you should be looking at data first to inform your understanding and to prioritize what to work on before trying to write evals.
2. It created markdown files for looking at data and asked us to "tell it" which labels were off. You should create your own annotation app instead. You are using coding agents after all (we did this in our livestream)!
3. The evaluators it scoped were too broad, and bundled too many types of failures at once. This can be avoided with looking at the data first. I'm afraid this might steer people in the wrong direction.
4. The workflow put us DEEP into the rabbit hole of a specific eval right away. It also asks "does it look right" at several steps without really giving you enough information to make that determination. I think this will steer people towards the wrong workflow in many cases.
The Good:
1. I was impressed by out of the box ability for issue discovery that other auto-eval approaches haven't been able to find! It found issues with human handoff, formatting, voice agents, and more. It's still better to look at your data iteratively with an agent, but this was the strongest performance I've seen with a more "one-shot" issue discovery approach.
2. This is a big improvement in terms of UX from their prior eval plugin thing xitter.cf/ClaudeDevs/status/2098…
3. The blog post they released conveys thinking that I agree with, such as the importance of looking at data, sampling intelligently, not saturating your own evals, etc. I'm really happy more people are thinking about evals this way.
Video of our attempt at using this here: xitter.cf/i/broadcasts/1nKOLQOwQ…
Claude can now help you build evaluations and hillclimb on them.
In this article, we share guidance on eval design & skills that Claude Code can use to improve your applications.
claude.dev/blog/automating-e…
Benjamin Warner retweeted
The thing is: from the beginning we always would have continued to call what we have now LLMs, and it’s not like you guys were saying “the LLMs you have now are stochastic parrots, but the obvious next step you guys are gonna do next is totally different”.
Benjamin Warner retweeted
I'm excited to share Medmarks was accepted to the NeurIPS datasets and benchmarks track! 🥳
See you at Sydney!
Benjamin Warner retweeted
The "do not feel" + "do not want" parts are debatable but not crazy, whereas the "do not think" and "do not understand" are crazy at this point, and it's concerning that this seems to be a real account/policy xitter.cf/APStylebook/status/210…
Artificial intelligence systems do not think, feel, want or understand. Avoid language that gives them human characteristics. This is called anthropomorphizing, when we ascribe human traits, emotions or behaviors to non-human things, such as animals or inanimate objects.
Instead, explain what a system does, how well it performs, who built it and who could be affected by it.
apnews.com/article/openai-sa…
With Opus 5.5 looking like it might be good, what's the best way to use one's Codex and Claude Code sub together these days?
No Terra but a new Astra Minor? Just after OpenAI fixes their naming they are right back at it again.
We might be getting a DOUBLE DROP today.
Opus 5.5 is dropping. Now GPT 6 Sol, GPT 6 Luna, and GPT 6 Astra Minor just showed up in Microsoft's Azure config overnight.
OpenAI demoted GPT 5.6 Sol to "workhorse" in the same commit. You do not do that unless the replacement is ready.
We are finally about to get intelligent models we can actually use on our subscriptions.
The biggest day in AI this year might be today.
We are in the process of evaluating more models for the next release of Medmarks, including Astra and Fable, but first, a teaser.
This year, we released Medmarks v1.0, our open benchmark suite + leaderboard of LLM medical capabilities.
Today, we release additional results for some of the latest mid-size open-source LLMs including Gemma, Qwen, Muse, Nemotron.
We find Gemma 4 31B leads in this size class.
Link: sophont.med/blog/medmarks-su…
Benjamin Warner retweeted
Regulation will be applied to AI models, both open and closed. The question is whether a proactive regulatory framework can be designed to prevent harms before they happen, or will there be harsher regulation imposed by the public in reaction to bad actors.
How is slowing down and potentially allowing others to catch up suppose to consolidate power? Full speed ahead to maintain their advantage would be better at consolidating power.
I hope the proposals by frontier labs to pace AI research stem from a genuine concern for safety and a recognition of the potential risks posed by future models, rather than a strategic effort to consolidate power and permanently solidify the market dominance of a few top players.
If these efforts are genuine, then oversight must take a more democratic and accountable form, with both national and international components, similar to the Nuclear Regulatory Commission in the US and the International Atomic Energy Agency internationally.
If there is only one organization responsible for safety monitoring, and it happens to be staffed by the same people as the frontier labs, and it is perfectly aligned with them both by incentives and by ideology, then it would be indistinguishable from letting frontier labs self-regulate and self-certify their own safety standards.
Two potential warning signs of regulatory capture to watch out for:
1. Calls to ban open-source AI. Open-source development currently serves as the only real counterweight to the dominance of frontier labs.
2. Attempts to hinder non-frontier research. Models that are one or two generations behind the frontier -- whose level of capability has already been deployed at scale -- are empirically known not to pose safety risks.
As long as we don't see 1 & 2, and we see real international coordination efforts, then I will optimistically believe that the proposals are benevolent.
Benjamin Warner retweeted
for the skeptics in government and elsewhere: “pacing the frontier” will compress the margins of the frontier labs. it is a heavy cost imposed asymmetrically on model developers with the strongest AIs in America. by its nature, it would be a terrible regulatory capture tactic
Regulation will be applied to AI models, both open and closed. The question is whether a proactive regulatory framework can be designed to prevent harms before they happen, or will there be harsher regulation imposed by the public in reaction to bad actors.
Benjamin Warner retweeted
I would consider helping as an independent evaluator along with the non profit I’m building. Is going to be a crucial role and there aren’t many independent people with substantial technical expertise. I think I represent an important moderate camp on AI capabilities and risks.
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here: darioamodei.com/post/we-must…
The WeWorm creators used a combination of closed and open frontier models. I suspect this is a leading indicator for open models being used for offensive purposes. Along with their defensive use like Hugging Face.
ALT The discovery is still likely to fuel more concerns that advanced A.I. models could soon usher in a dystopian future that shatters core assumptions about digital security and, at least in the short term, delivers a major advantage to malicious hackers. Calif said it relied on a combination of open-source A.I. models and leading models from the United States, but it declined to specify which ones.
Benjamin Warner retweeted
CancerBench: the frontier model cancer cure benchmark.
AI lab CEOs keep talking about curing cancer, so I made a benchmark.
One metric: how many types of cancer has your model cured?
All models are currently tied at zero.
It’s time to hillclimb!
cancerbench.com
Benjamin Warner retweeted
Worth clarifying that this is flatly false and the two options are just two independent ways to opt-out.
We respect our users' choice whether to use their data to “improve our models for everyone” regardless of where they express that choice. Users can opt out in the in-app settings or indeed also in our privacy portal. They do not need to opt out in both places, and we will make this clearer in our Help Center.
privacy.openai.com/policies/…
There is a super shady data control setting on everyone's ChatGPT which makes it seem like your data can be used for training even if you explicitly say to *NOT* improve the model for everyone (1/9)
Readers added context they thought people might want to know
This claim is false. OpenAI has clarified that the in-app toggle and the privacy portal are independent ways to opt out of data training. You only need to use one method, and OpenAI respects the opt-out choice regardless of where it is set.
x.com/thsottiaux/sta…
help.openai.com/articles/77308…
Benjamin Warner retweeted
In preview, uv now uses a fully content-addressed cache.
Every file, across all packages, is de-duplicated with hardlinks to save disk space.
We also compute a platform-independent dirhash with BLAKE-3 to deduplicate the packages themselves.
