hyperintelligence for humanity
San Francisco ๐
Joined January 2024
- Tweets102
- Following5
- Followers118
- Likes112
Pinned Tweet
We ran GPT-6 Astra, Opus 5.5, Fable 5 and four other frontier models on a variety of agentic tasks across digital and physical domains: web tasks, driving, robot manipulation, industrial procedures and assembly.
Weโre releasing a technical report, sharing details for an invited talk, and releasing a new dataset of frontier traces and item level results you can use in your research.
Every model is jagged on every axis we tested. Astra has the top score at 78.5%, yet Fable 5 and Opus 5.5 each solve 10 tasks Astra fails. In all 21 model pairs, the lower-scoring model solves at least 3 tasks the higher-scoring one misses.
The map places each VWA task on a circle. Similar instructions sit together, harder tasks sit closer to the center, and failure clusters rise as hills.
fig retweeted
I'm really excited about the upcoming data technologies the Fig team is developing. We're systematically solving the open data problem.
๐๐ณ ๐๐ผ๐'๐ฟ๐ฒ ๐๐ผ๐ฟ๐ธ๐ถ๐ป๐ด ๐ผ๐ป ๐ฎ๐ฝ๐ฝ๐น๐๐ถ๐ป๐ด ๐ฎ๐ถ ๐๐ผ ๐น๐ผ๐ป๐ด ๐ต๐ผ๐ฟ๐ถ๐๐ผ๐ป ๐ผ๐ฟ ๐ผ๐ฝ๐ฒ๐ป ๐ฒ๐ป๐ฑ๐ฒ๐ฑ ๐๐ฎ๐๐ธ๐ ๐ถ๐ป ๐ฟ๐ฒ๐ฎ๐น๐ถ๐๐๐ถ๐ฐ ๐๐ฒ๐๐๐ถ๐ป๐ด๐, ๐น๐ฒ๐'๐ ๐ฐ๐ต๐ฎ๐!
Many frontier teams (including us) have run into this big hairy problem of extracting useful signal and building effective, expressive environments for the kind of work someone might do over the course of a full week in a high performance job.
We're building a science to solve this.
data.fig.inc/preview?utm_souโฆ
xitter.cf/figbrains/status/21136โฆ
We're sharing early previews with teams ahead of our upcoming releases.
If you're deploying models into complex, open-ended environments, we want to work with you: data.fig.inc/preview?utm_souโฆ
The Fig team will be in Boston for the ARC Prize summit! Reach out to get a sneak peek at the next generation of intelligence weโve been quietly building!
Iโm attending & presenting @arcprize summit in Boston.
Will be around tomorrow & Saturday.
Dm / reply to me the most cracked people you know, i want to brain drain everyone (researchers, engineers, founders investors anyone)
Iโll let them brain drain me too, so itโll be worth it
fig retweeted
Benchmarks hide serious gaps in model performance.
Last Friday, we learned the extent of this in "How Good Are Frontier Models, Really?"
@Yangyue_Wang from @figbrains dug into why model capabilities remain so jagged even as benchmark scores keep climbing.
Missed it? No problem! Watch the full talk here:
youtube.com/watch?v=59WZlXqHโฆ
fig retweeted
A study from the team showing Astra's and Opus 5.5's jagged reasoning behaviors in simple puzzles. More data points on frontier models' failure profiles.
xitter.cf/figbrains/status/21100โฆ
fig retweeted
Cranking up reasoning doesn't solve all problems.
More thinking more may generate better text or artifacts, not necessarily actions.
That is the gap we care most about at Fig: systems that improve from the consequences of their own actions.
Next we put the same mazes in a 3D world and in plain language, with full history, to separate familiarity with the interface from understanding of the task.
Built with @ManifoldRG by Pranav Guruprasad, Sean Rivera, Helen Lu, Arushi Jain and Harsh Sikka.
fig retweeted
Frontier model performance is not uniform across tasks at all, despite what benchmarks say.
On October 13, @Yangyue_Wang from @figbrains joins us to unpack performance across web, driving, robotics, assembly, and other agentic tasks.
Astra leads overall at 78.5%, yet lower-scoring models still solve tasks it misses.
Join us live at 12 PM PT! Registration below โ
fig retweeted
We've been spending a lot of time thinking about open-ended work and what's needed to capture reliable training signals from it.
You can read our latest thoughts here: xitter.cf/figbrains/status/21056โฆ
Computer-use data focuses on contained tasks in a controlled workspace. Real work is very different: open-ended goals, multi-thousand-step workflows, changing context, and interwoven tasks interrupted and resumed.
We wrote about what that gap looks like and why it matters: xitter.cf/figbrains/status/21056โฆ
We expect agents to operate reliably in open-ended worlds. To do so, their training data has to fundamentally evolve: real work observed in continuously evolving environments across unscripted, multi-thousand-step workflows.
Weโre actively closing that gap. If youโre building models, evals, or benchmarks, letโs collaborate: info.fig.inc/data?utm_sourceโฆ
