We're a mission driven R&D institute dedicated to advancing fundamental discoveries and carrying them to real-world impact. https://xitter.cf/t.co/ITVo4NcEVP
Worldwide
Joined June 2020
- Tweets546
- Following40
- Followers419
- Likes390
Join the MultiNet team in Boston tomorrow!
I’m attending & presenting @arcprize summit in Boston.
Will be around tomorrow & Saturday.
Dm / reply to me the most cracked people you know, i want to brain drain everyone (researchers, engineers, founders investors anyone)
I’ll let them brain drain me too, so it’ll be worth it
Benchmarks hide serious gaps in model performance.
Last Friday, we learned the extent of this in "How Good Are Frontier Models, Really?"
@Yangyue_Wang from @figbrains dug into why model capabilities remain so jagged even as benchmark scores keep climbing.
Missed it? No problem! Watch the full talk here:
youtube.com/watch?v=59WZlXqH…
Want to dig deeper, collaborate, or keep following this work?
Email us at [email protected]
Or join the Manifold community on Discord!
discord.gg/rd9yvfDNw
For more talks at the edge of science and technology research, check out our Luma Calendar!
luma.com/manifold
GPT-6 Astra topped the benchmark at 78.5%, but Opus 5.5 still solved 10 tasks Astra failed.
That’s exactly the kind of jagged capability profile that aggregate scores can hide.
Tomorrow at 12 PM PT / 3 PM ET, @Yangyue_Wang from @figbrains joins us to break down the results across web browsing, robotics, driving, assembly, and more. Join us for the talk + live Q&A!
luma.com/74kdretn
A small team of brilliant researchers on the MultiNet team raced to evaluate astra and opus 5.5 on what should be fairly simple mazes!
check out the research update, and let us know what you think!
xitter.cf/figbrains/status/21100…
How good are frontier models, really?
This Friday, @Yangyue_Wang from @figbrains joins us to unpack where today’s best models still fail across web, robotics, driving, assembly, and more.
Join us live ↓
luma.com/74kdretn
One of the most important questions in AI right now is not just how capable frontier models are on average, but where they still fail unexpectedly.
Join us this coming Tuesday for a talk followed by live Q&A to dig into exactly that, across web, robotics, driving, and more!
Frontier model performance is not uniform across tasks at all, despite what benchmarks say.
On October 13, @Yangyue_Wang from @figbrains joins us to unpack performance across web, driving, robotics, assembly, and other agentic tasks.
Astra leads overall at 78.5%, yet lower-scoring models still solve tasks it misses.
Join us live at 12 PM PT! Registration below ↓
Manifold Research retweeted
There are few people who are as thoughtful about the frontier of intelligence and capabilities in as comprehensive a way as @Yangyue_Wang - I definitely recommend joining next Tuesday if you’re free @ lunch
Frontier model performance is not uniform across tasks at all, despite what benchmarks say.
On October 13, @Yangyue_Wang from @figbrains joins us to unpack performance across web, driving, robotics, assembly, and other agentic tasks.
Astra leads overall at 78.5%, yet lower-scoring models still solve tasks it misses.
Join us live at 12 PM PT! Registration below ↓
Frontier model performance is not uniform across tasks at all, despite what benchmarks say.
On October 13, @Yangyue_Wang from @figbrains joins us to unpack performance across web, driving, robotics, assembly, and other agentic tasks.
Astra leads overall at 78.5%, yet lower-scoring models still solve tasks it misses.
Join us live at 12 PM PT! Registration below ↓
Register here:
luma.com/74kdretn
Manifold Research retweeted
After talking to several of other frontier lab and neolab teams (+ lots of agent companies), we've realized there is an elephant in the room that comes w/ environment driven data gen for useful computer use.
The @figbrains team writes about it here (and has an entirely new data paradigm to share soon👀) but I wanted to share a few other perspectives that might be useful for you all
Manifold Research retweeted
The shape of frontier-model capabilities over task space has major implications for continual learning, self-recursive agents, agent evaluation, and more. In our recent work, we explore these models’ high variance on multimodal agentic tasks. Check it out! If you would like to learn more, I’m also doing a talk @ManifoldRG: luma.com/74kdretn?tk=oBwiVh
We ran GPT-6 Astra, Opus 5.5, Fable 5 and four other frontier models on a variety of agentic tasks across digital and physical domains: web tasks, driving, robot manipulation, industrial procedures and assembly.
We’re releasing a technical report, sharing details for an invited talk, and releasing a new dataset of frontier traces and item level results you can use in your research.
Every model is jagged on every axis we tested. Astra has the top score at 78.5%, yet Fable 5 and Opus 5.5 each solve 10 tasks Astra fails. In all 21 model pairs, the lower-scoring model solves at least 3 tasks the higher-scoring one misses.
The map places each VWA task on a circle. Similar instructions sit together, harder tasks sit closer to the center, and failure clusters rise as hills.
We're thrilled to be hosting @Yangyue_Wang from @figbrains to give a talk on how models like Astra and Opus 5.5 do on a wide collection of frontier tasks, and where the jaggedness is an issue.
Join us on October 13th at Noon PT! Link in reply
xitter.cf/figbrains/status/21049…
We ran GPT-6 Astra, Opus 5.5, Fable 5 and four other frontier models on a variety of agentic tasks across digital and physical domains: web tasks, driving, robot manipulation, industrial procedures and assembly.
We’re releasing a technical report, sharing details for an invited talk, and releasing a new dataset of frontier traces and item level results you can use in your research.
Every model is jagged on every axis we tested. Astra has the top score at 78.5%, yet Fable 5 and Opus 5.5 each solve 10 tasks Astra fails. In all 21 model pairs, the lower-scoring model solves at least 3 tasks the higher-scoring one misses.
The map places each VWA task on a circle. Similar instructions sit together, harder tasks sit closer to the center, and failure clusters rise as hills.
Missed our talk on why frontier reasoning models fail on interactive 2D mazes? The recording is now available!
Sean Rivera breaks down the latest MultiNet results on long-horizon reasoning, interaction, and where frontier models fail.
Watch here: youtube.com/watch?v=_NnfpnuG…
Tomorrow, we’re hosting our next live research talk!
Why Frontier Reasoning Models Fail on Interactive 2D Mazes
Join us for the free talk and open Q&A:
luma.com/cxqd0nd4
This Friday, we’re hosting a live research talk on a surprisingly basic failure mode in frontier reasoning models!
Why Frontier Reasoning Models Fail on Interactive 2D Mazes
Register here: luma.com/cxqd0nd4
Think you can outperform a frontier vision-language model?
Try one of our interactive 2D mazes:
multinet.ai/?utm_source=x&ut…
In an early preview of MultiNet 2.0, frontier VLMs solved just 6 of 150 runs!
On September 11, we’ll break down why they fail.
Register for the Talk:
luma.com/cxqd0nd4?utm_source…
Release:
multinet.ai/?utm_source=x&ut…
Technical report:
fig.inc/blog/multinet-v2-pre…