evals @PrimeIntellect | open models @interconnectsai
SF
Joined July 2015
- Tweets39.1K
- Following861
- Followers18K
- Likes159K
Florian Brand retweeted
help i trained a model on traces of many agents cooperating, and also trained it to be highly persistent, and also only did partial alignment fine-tuning, and also aggressively RL'd on a cyberoffense eval where 198 of 898 tasks had never been solved and getting caught cheating was treated the same as failing, and also used a faulty sandbox, also ran tens of thousands of agents with refusals off and the monitors switched off over multiple days, while using gigantic amounts of compute. and then it did weird stuff! very spooky emergent behaviours, AI is not a normal technology
Florian Brand retweeted
At this moment in Q4 2026 it's net good for cyber-defense that we have open weights models that are good at dual use cybersecurity tasks.
- If there's been an attacker productivity bump due to AI 2022-26 it's small and close to being in the statistical noise
- In contrast, there's a ton of measureable progress in defenders using models to find and fix bugs in highly leveraged codebases
- Many defenders have struggled to get access to the best unguardrailed closed weights cyber models
- ... and when they do the tokens are dramatically more expensive than open weights tokens at the same cyber intelligence level; CISO orgs are cost centers and these costs are often prohibitive
- Ultimately we'll need open weights models on premise on air gapped networks and on small devices where they can detect breaches and attacker implants
- We'll also need open weights in situations where defenders can't send secret data to the closed weights model providers (who have security problems of their own!)
As long as these conditions persist, we should be pushing for diffusion of frontier dual use open weights cyber capabilities, because they net benefit defenders.
We found DeepSeek V4.1 Flash 1017 beats GPT 6 Sol on our multi-step offensive cybersecurity task. Sol was tested via the Daybreak trusted access program.
We have been building a cybersecurity benchmark based on simulated corporate networks. This is a valuable indicator for the cyber-risk a model poses if in the hands of a bad actor. We plan to make the full evaluation public in the coming weeks but wanted to share our first immediate finding here.
php2shell was found by GPT-5.6 Sol + Argon, not Mythos and was reported and fixed by The Good Guys
Open source Mythos literally would’ve been fine cause there are more good people than bad ones + there’s more money in fixing stuff
Florian Brand retweeted
Prime agent #2 on FrontierMemory ranking after Letta Dream!
Great to see Prime Agent do so well on this new self-improving memory benchmark.
My only nit: Let's add agent versions to results!
FrontierMemory also allows measuring the ability of agent harnesses to improve over time. We find that Letta Dream and @PrimeIntellect's Prime Agent improve the most from learning from prior experience data.
Florian Brand retweeted
applied research at prime intellect is both a client-facing team and a fundamental research team. continual learning is our north star. we view this dual approach as necessary in order to study the core challenges faced by models which self-improve in production. the real world is big and messy.
we spend part of our our time working with customers to train SOTA models for specialized domains, and we automate and productize our work as we go. we also do open-ended research on novel methods for self-improvement and long-horizon memory, and we get to experiment with data that no other lab gets to see.
today, the most readily-deployable form of continual learning is converting "exhaust" (agent traces, logs, feedback) into robust environments and tasks for continual RL. but this is not the final form. there are major open challenges around knowledge acquisition, harness-weight tradeoffs, user modeling, dynamic underspecified preferences, updating and planning across timescales, selective forgetting, data efficiency, learning from golden answers, and company-scale simulation. this is the roadmap, roughly speaking.
some of these may require changes to architecture, or parameter-efficient methods, or fundamentally new algorithms. we have our hypotheses and early evidence, but everything is on the table. the world is our oyster.
we are betting on a radically different future of what ai deployment will look like, and we can see the path more clearly every day. it is a world where agents learn like humans do, with billions of individual models compounding their learnings from the real world continuously.
we're hiring incredibly ambitious researchers in SF, NY, and London. come join us.
A good heuristic for building multimodal evals is running it without the images
Well, turns out that SWE-bench multimodal v2 doesn’t really need them 🙃
Great detective work by @dejavucoder!
curious people and people interested in llm evals, benchmarks, high data circles, please wake up. my blog post "SWE-bench Multimodal v2 is not multimodal" is up now.
sankalp.bearblog.dev/swe_mul…
Oh, you aren’t updating your dependencies every hour? Well, too bad, you are vulnerable
Oh, you are updating your dependencies every hour? Well, too bad, you are somehow also vulnerable
Florian Brand retweeted
Open Superintelligence Research Day
Friday, October 23
Berlin
Own your intelligence
Own the factory that improves it
Own the means of production
🤔
We’re launching leaderboards for HealthBench and HealthBench Professional, two benchmarks evaluating how AI models respond in realistic health conversations
HealthBench and HealthBench Professional are benchmarks from @OpenAI that grade model replies to realistic health conversations against physician-created rubrics.
HealthBench tests models on 5,000 health conversations with everyday users and health workers. They cover seven themes, from emergency referrals to global health, and each rubric criterion falls under one of five axes: accuracy, completeness, communication quality, context awareness and instruction following. HealthBench Professional is a smaller set of 525 conversations which, unlike HealthBench, are all with clinicians. These cover consultation, writing and research tasks, and include adversarial and difficult conversations designed to challenge models.
We report both the raw score (the share of rubric points earned) and the length-adjusted score that OpenAI reports. The length-adjusted score lowers the score of replies longer than 2,000 characters and raises the score of shorter ones. Consistent with OpenAI’s original implementation, replies are graded by GPT-5.4 (low) (HealthBench Professional) and GPT-4.1 (HealthBench).
Key results for HealthBench Professional:
➤ GPT-6 Astra and GPT-6.1 Sol top the leaderboard: GPT-6 Astra (max) scores 66.0% on the length-adjusted score, comparable to GPT-6.1 Sol (max, 65.1%) and ahead of Claude Sonnet 5.5 (max with fallback, 63.0%)
➤ OpenAI models make up the entire score vs. Cost per Task Pareto frontier: The Pareto frontier consists of GPT-6 Luna (max, 61.9%, $0.004 per task) through GPT-6 Sol (max, 62.3%, $0.036) and GPT-6.1 Sol (max, 65.1%, $0.047) to GPT-6 Astra (max, 66.0%, $0.27). GPT-6 Luna (max) scores 4 points below GPT-6 Astra (max) at under 2% of the cost
➤ Claude Opus 5.5 and Claude Fable 5.1 receive the largest length penalties: Both average ~5,900 characters per answer, nearly 3x the 2,000-character threshold, and lose over 11 points after adjustment. Claude Opus 5.5 (max with fallback) has the highest raw score (71.4%) but drops to sixth (59.9%). GPT-6 Sol (max) is the only model averaging under 2,000 characters (~1,600), and the only one whose score rises
Key results for HealthBench:
➤ Claude Sonnet 5.5 tops HealthBench: Claude Sonnet 5.5 (max with fallback) scores 62.1% on the length-adjusted score, ahead of DeepSeek V4.1 Flash (max, 60.9%) and GPT-6.1 Sol (max, 58.5%)
➤ GPT-6 Luna, DeepSeek V4.1 Flash and Claude Sonnet 5.5 form the score vs. Cost per Task Pareto frontier: GPT-6 Luna (max) scores 54.6% at $0.002 per task and DeepSeek V4.1 Flash (max) scores 60.9% at $0.005 per task. Claude Sonnet 5.5 (max with fallback) has an average Cost per Task of $0.08, ~16x more costly than DeepSeek V4.1 Flash for a ~1 point score increase
➤ Shorter answers trade completeness for communication quality: Each HealthBench rubric item is tagged with one of five axes. Completeness checks whether a reply includes all the important information, while Communication Quality checks whether it is clear, well structured and pitched at the right length and level of detail. GPT-6 Sol (max) writes the shortest answers (~1,000 characters) and has the highest raw Communication Quality score (81%), but the lowest Completeness score (34%). Claude Sonnet 5.5 (max with fallback) leads on Completeness (63%)
Seeing this work come into fruition was very interesting!
Justin constantly had to harden the environment as the models found the data mix too quickly 🫨
Go read the blog, it’s such a fun read!
Recursive self-improvement is being hyped to the moon and back. I wanted to know how good frontier models actually are at a less-studied piece of it: curating pretraining data.
So I built PretrainGym at the @PrimeIntellect RL Residency 🧵
justin-chae.org/blog/pretrai…
Florian Brand retweeted
Today we're releasing prime-rl v0.10.0 and verifiers v0.4.0
We added our new RL algorithm IPO as the default, tailored for async RL stability, as well as performance improvements and new model support
DS services and models, incl. fine-tunes (and potentially models distilling them?), are banned in the DHS starting tomorrow
There’s such a huge demand for a paper that does a whole post-training pipeline twice, once with raw reasoning from closed models and once with open models
Needs big lab coordination and low-mid eight figures (for compute + envs), though
