AI4Science PhD @Stanford @StanfordAILab @StanfordHAI Terminal-Bench-Science Lead @terminalbench Prev. @Cambridge_Uni, @NASAJPL, @imperialcollege
Stanford, CA
Joined January 2020
- Tweets478
- Following1.9K
- Followers1.9K
- Likes2.4K
Pinned Tweet
We're releasing Terminal-Bench-Science: a benchmark for evaluating AI agents on research workflows across scientific domains.
An ongoing Stanford-led community effort, built by the team behind Terminal-Bench together with scientific domain experts at research institutions worldwide. v0.1 has 70 tasks. Claude Opus 5 solves only ~30%.
1/n 👇
Steven Dillmann retweeted
Learning that "RSI" benchmarks are just MLE/post-train style tasks repackaged with a new label was disappointing.
We could just take the ~20 ML-adjacent tasks from TB 4 and re-release them as RSI-bench and everyone would clap.
Steven Dillmann retweeted
TB-Science 🤝 FutureSim! At the Frontier Data Summit by @SnorkelAI
Turns out we are all from the same region in Germany! In the order:
Tübingen <-> Metzingen <-> Stuttgart
Terminal-Bench-Science featured on @ValsAI's new Vals Science Index alongside MysteryMechanism and BioMysteryBench.
Great to see AI evals finally giving science the attention it deserves. 🤖🧪
Steven Dillmann retweeted
We just launched PersonalAgentEval!🚀🚀🚀 (personalagenteval.com) The first benchmark built to evaluate personal agents across diverse users, preferences, and domains!
We benchmarked 7 popular personal agents across everyday workflows (e.g. booking dinners, paying bills, managing calendars). Each agent operated within accounts pre-loaded with 13 months of realistic history (347–470 records per person). To pass, an agent has to proactively spot crucial background details: like an unread message, a clause in a contract, or a promise made in a past chat. Each task and user simulation are validated and verified by real human.
Key Findings:
• 🏆 Overall Leaders: Instinct and OpenAI Dots topped the leaderboard.
• 🔍 Proactive Context Retrieval: Instinct excelled at uncovering crucial background details—finding 64 of 72 key facts before the user brought them up.
• ⚡ Speed & Efficiency: Grok Bot was both fast and effective, averaging 0.8 minutes of active work per task with a 50% completion rate. Meta Muse required the fewest average interaction turns.
• ⚠️ Missing Promised Steps: Even leading agents missed crucial follow-up steps. In one task, all 7 agents correctly identified a pharmacy mentioned in past logs, but only Grok Bot and Meta Muse remembered to notify the caregiver as promised.
Explore the full leaderboard and read our case study and analysis:
personalagenteval.com
(Building a personal AI agent? Reach out to us if you’d like to get your agent evaluated and featured on the leaderboard.)
#MatrAIx #AI #AIAgents #MachineLearning #LLMs #MetaMuse #GrokBot #OpenAIDots #Hark #ManusCue #Instinct
Great visit to the @ArtificialAnlys office in SF today with @zijian42chen. We got to watch AA-Robotics v0.1 launch in real time! Really exciting work, and I’d love to see it extend to scientific tasks in the physical world.
Thanks for hosting, @wmkeats & @_micah_h!
Welcome to the era of embodied AI.
We are launching AA-Robotics v0.1, our research preview that measures frontier language model performance on understanding, planning, and acting in the physical world. In v0.1, each model is given control of a 6-axis robot arm (5 joints and a gripper) and a simple objective. We’re releasing results for our first eight tasks today.
We find that frontier language models in coding agent harnesses have recently become capable of directly manipulating robot arms for simple tasks. Today, they are far too slow and expensive for any kind of production usage - nearly two orders of magnitude worse than human baselines. We expect this to change quickly.
GPT-6 Astra places first in AA-Robotics v0.1, followed by GPT-6.1 Sol, Gemini 4 Argon and Claude Opus 5.5.
AA-Robotics v0.1 details:
➤ Our first eight tasks test a range of capabilities: control and dexterity (precise grasping and object handling), spatial reasoning (how objects interact with one another) and adaptability (how models respond to the unexpected)
➤ Models get only brief instructions: how to connect to the robot arm (via a simple HTTP API), the arm's name and how to submit an attempt. They can view the scene through four cameras and write and run code to plan and send movements
➤ Once a model submits an attempt, its end state is graded against the task's pass criteria. A pass scores 100; an incomplete attempt earns partial credit. All attempts are zero-shot
Key findings:
➤ The GPT family tops the leaderboard: GPT-6 Astra scores 69, ahead of GPT-6.1 Sol at 65 and Gemini 4 Argon at 55. Claude Opus 5.5 (49) and Claude Fable 5.1 (46) trail
➤ Today’s models are incredibly slow at our tasks, taking ~30 minutes on average to complete each tabletop task
➤ Cost varies widely: GPT-6.1 Sol is the cheapest at $1.93 per task on average; GPT-6 Astra, the most expensive, costs $16.36, more than eight times as much
➤ Compared with a human, today's best-performing frontier language models are ~100x slower and ~10x more expensive
More results, clips, and how AA-Robotics v0.1 works follow in the thread below.
Steven Dillmann retweeted
Despite the excitement around end-to-end scientific agents and their success on well-scoped optimization tasks, evaluating open-ended research remains a challenge. We're collecting tasks for SciVeri-Bench, with scientists in the loop to assess research quality. Join us!
🔬 Calling scientists across disciplines: help build SciVeri-Bench v1.0!
After showcasing SciVeri-Bench v0.1 at the Frontier Data Summit hosted by @SnorkelAI, we’re now building v1.0 and inviting scientists from a wide range of fields to contribute scientific tasks.
🤔 What is SciVeri-Bench?
SciVeri-Bench is a deploy-first, scientist-driven benchmark for evaluating AI agents’ scientific verification capabilities on open-ended, novel scientific tasks.
🎯 Why are we building it?
- Stop to chasing maximing score in benchmark, rather we want to understand whether AI agents verify whether AI agent doing good science as domain scientist do
- Domain scientists only verify whether the science is good or bad, while agents do a lot of execution and exploration, but we really want to show that scientists can make novel discoveries
📝 How are tasks collected?
All task submissions and communication take place on OpenReview, so domain scientists can focus on proposing open-ended & novel scientific tasks and verifying agent outcomes.
🤝 What do scientists receive?
- All costs of running the agents are covered by the SciVeri-Bench team.
- Intellectual property rights to scientific discoveries made by AI agents belong to the scientist who proposed the task.
- Co-authorship on the SciVeri-Bench paper.
How can you contribute?
(1) Task Proposal: Propose a scientific task for SciVeri-Bench.
(2) Task Review: Review scientific tasks proposed by other scientists.
Interested in joining us?
👉 Sign up: forms.gle/bv9KZK6mW6ubBJXZ9
🌐 Website: sciveri-bench.com/
🙏 Huge thanks to our team, contributing scientists, advisors, and sponsor!
AI researchers: @seungonekim @jinheonbaek @yoonholeee @JiayiiGeng
Task proposers: 16 scientists who have joined so far
AI advisors: @StevenDillmann @SoojungYang2 @AkariAsai @gneubig @wellecks @dongyeopkang
Sponsor: openrefinery.ai/
We’re also looking for more sponsors and senior scientists!
Note: This work does not relate to any work conducted at Amazon
Steven Dillmann retweeted
🔬 Calling scientists across disciplines: help build SciVeri-Bench v1.0!
After showcasing SciVeri-Bench v0.1 at the Frontier Data Summit hosted by @SnorkelAI, we’re now building v1.0 and inviting scientists from a wide range of fields to contribute scientific tasks.
🤔 What is SciVeri-Bench?
SciVeri-Bench is a deploy-first, scientist-driven benchmark for evaluating AI agents’ scientific verification capabilities on open-ended, novel scientific tasks.
🎯 Why are we building it?
- Stop to chasing maximing score in benchmark, rather we want to understand whether AI agents verify whether AI agent doing good science as domain scientist do
- Domain scientists only verify whether the science is good or bad, while agents do a lot of execution and exploration, but we really want to show that scientists can make novel discoveries
📝 How are tasks collected?
All task submissions and communication take place on OpenReview, so domain scientists can focus on proposing open-ended & novel scientific tasks and verifying agent outcomes.
🤝 What do scientists receive?
- All costs of running the agents are covered by the SciVeri-Bench team.
- Intellectual property rights to scientific discoveries made by AI agents belong to the scientist who proposed the task.
- Co-authorship on the SciVeri-Bench paper.
How can you contribute?
(1) Task Proposal: Propose a scientific task for SciVeri-Bench.
(2) Task Review: Review scientific tasks proposed by other scientists.
Interested in joining us?
👉 Sign up: forms.gle/bv9KZK6mW6ubBJXZ9
🌐 Website: sciveri-bench.com/
🙏 Huge thanks to our team, contributing scientists, advisors, and sponsor!
AI researchers: @seungonekim @jinheonbaek @yoonholeee @JiayiiGeng
Task proposers: 16 scientists who have joined so far
AI advisors: @StevenDillmann @SoojungYang2 @AkariAsai @gneubig @wellecks @dongyeopkang
Sponsor: openrefinery.ai/
We’re also looking for more sponsors and senior scientists!
Note: This work does not relate to any work conducted at Amazon
Steven Dillmann retweeted
Many are criticizing that the released math results are not up to the academic standards of journal publication. I think this , on the contrary, gives a rare chance for human to interact at an earlier stage of the work. It would be more fruitful this way.
Steven Dillmann retweeted
That's a wrap on Frontier Data Summit!
More than 30 speakers and a room full of researchers came together to discuss how we measure and advance frontier AI 🚀
Always loved the work done by @ArtificialAnlys - excited to hear that Terminal-Bench-Science will be included in the next version of the Artificial Analysis Intelligence Index 🚀
Steven Dillmann retweeted
Harbor supports network restrictions, separate verifiers, and resource controls.
Different defaults aren’t missing features. Giving users control is deliberate.
Steven Dillmann retweeted
In 22 days, @StevenDillmann from @Stanford will be joining us to share his thoughts on the future of RL.
It's not too late to register for Convergence RL. luma.com/gbcompna
I’ll be in SF all day on Thursday & Friday this week. If you’re in SF for COLM or SF Tech Week and would like to chat and get Terminal-Bench-Science merch, get in touch via DM!
Thanks to @LaudeInstitute & @harborframework for organizing the merch.
If you'd like to learn about Harbor, Terminal-Bench, and Terminal-Bench-Science without reading through a wall of text, check out these lightning talks by @alexgshaw, @ryan_marten, and me from our recent @terminalbench meetup. Now live on YouTube📽️
Recordings from @terminalbench's latest meetup are now on YouTube! Watch here: youtube.com/watch?v=VTUFfdq_…
Steven Dillmann retweeted
Asked Claude to check the "harness tax" (@melissapan) on Terminal-Bench 4.0 (@alexgshaw @ryan_marten) and TB-Science @StevenDillmann
Native harness (Claude Code/Codex/Grok Build) vs mini-swe-agent (@OfirPress @KLieret), same model:
1. cost: ~same
2. score: native slightly ahead
TB 4.0 (@terminalbench):
- effort-matched cost: median 1.0× (0.78–1.24×)
- score mostly ±3 pts; Grok 4.7 +12 in Grok Build
- GPT-6 Astra high→max: +43% cost, +0.3 pts
- Fable 5.1 xhigh→max: +27% cost, +0 pts
Steven Dillmann retweeted
The bar for trustworthy, lasting benchmarks is rising.
@StevenDillmann, Russell Yang, @GOrlanski, and @vincentsunnchen discuss what it takes to build great open benchmarks at Frontier Data Summit. frontierdatasummit.ai/