Daniel Shepard (@danielwshepard) profile picture

Benchmark AI Engineer at Zapier, contributor to AutomationBench

Oregon
Joined June 2019
Photo by Daniel Shepard (@danielwshepard)Photo by Daniel Shepard (@danielwshepard)Photo by Daniel Shepard (@danielwshepard)Photo by Daniel Shepard (@danielwshepard)Photo by Daniel Shepard (@danielwshepard)Photo by Daniel Shepard (@danielwshepard)Photo by Daniel Shepard (@danielwshepard)Photo by Daniel Shepard (@danielwshepard)Photo by Daniel Shepard (@danielwshepard)Photo by Daniel Shepard (@danielwshepard)
The last few months I have been working on a new Benchmark. Introducing AutomationBench. Trying to measure the cutting edge of model's capabilities in real world business workflows across multiple apps and noisy data. The best models haven't beat 10% yet.
2
1
7
756
Some insight into the beginnings of AutomationBench by my coauthor Robin Salimans. Also announcing AutomationBench 1.5 coming soon!
When @SalimansRobin pitched building a business-agent benchmark at @zapier, co-founder @MikeKnoop gave him advice, including: 1. “All benchmarks are marketing”: AI labs won’t put your benchmark on a model card if they look bad on it. 2. “Make sure the benchmark is scientifically interesting, and make it hard”: Broadening beyond Zapier-specific tasks paid off. At launch, top models scored under 10%. In his Bench Summit 2026 talk, Robin breaks down how AutomationBench 1.5 trims task counts to run multiple trials per task and measure true reliability. Watch the full keynote: youtu.be/9d12rIqZSoA
2
8
391
Google grabs another top score on AutomationBench!
Gemini 4 Neon just set a new high on AutomationBench: 56.3% It’s the third @GoogleDeepMind record in 7 weeks: • Argon: 51.3% • Carbon: 53.9% • Neon (released today) In our testing, Neon is strongest on HR, sales, and marketing workflows Leaderboard updated today👇
2
84
Opus 5.5 does really well on AutomationBench too, breaking 40%! A major step up for less cost!
Claude Opus 5.5 is here! We ran the numbers. Here's where you should upgrade from Opus 5 👇️ Any crucial Ops workflows - it scored 62% for Ops at Max effort, at roughly $1.37 per task. Great value for the price. Overall it scored 40% across 657 tasks - a BIG jump from Opus 5's 26.9%, and one of the best models we've benchmarked yet. Note: We'd recommend scaling the effort per workflow (instead of keeping the default, Medium, for each). For example, Low scored 48% for Ops compared to Max’s 62%, but Max is double the price. Worth experimenting! See all the models we've benchmarked here: zapier.com/benchmarks
1
3
561
Daniel Shepard retweeted
GPT-6 Sol is here! We benchmarked it on real workflows, so you don't have to. - Scored 33.2% at xhigh effort - better than GPT-5.6 Sol's 28.77% - Roughly half the price of 5.6. $2 per million input tokens, down from $4. - Strongest for: Ops work (scored 58%!) Upgrade any of your workflows that are using 5.6. It's cheaper AND better (rare combo) See all the models we've benchmarked here: zapier.com/benchmarks
Replying to @OpenAI
Higher usage limits and lower cost give you more flexibility and room to iterate.
1
1
10
2,673
New GPT 6 models! Doing well on AutomationBench.
Replying to @OpenAI
Higher usage limits and lower cost give you more flexibility and room to iterate.
2
64
Daniel Shepard retweeted
AutomationBench (by @Zapier) is the lead benchmark OpenAI used to showcase Astra's workflow automation capability -- 44% at $2.45/task.
Replying to @OpenAI
Astra achieves state-of-the-art results on Agents’ Last Exam, AutomationBench, and ScreenSpot Pro, benchmarks for computer workflow tasks across professions.
5
5
31
3,570
Daniel Shepard retweeted
Astra achieves state-of-the-art results on Agents’ Last Exam, AutomationBench, and ScreenSpot Pro, benchmarks for computer workflow tasks across professions.
72
234
82
5,753
2,007,002
GPT 6 Astra scores 41.4% 😳
GPT 6 Astra is here. We ran the numbers on AutomationBench: It's the highest score we've ever recorded. Clean sweep across every domain. Scores 41.4% at Max effort. For context, no model had cleared 40% before today (GPT-5.6-Sol scored 28.8%) 𝗕𝗲𝘀𝘁 𝗳𝗶𝘁 𝗳𝗼𝗿: reconciliation, deal review prep, vendor scorecards, anything where touching the wrong record is expensive. 𝗪𝗲𝗮𝗸𝗲𝗿 𝗳𝗼𝗿: outbound comms where the guidance is scattered. Operations and support are its strongest domains. HR is its weakest, same as every model we test (still the new high score, though) Its edge is arithmetic across messy sources. Finding the policy doc, the logged correction, the exception rule, etc. Example 1: rebalance a quarterly media budget from last quarter's actuals, with finance adjustments and channel eligibility rules buried in email. Both models produced a budget and landed on the same total. Astra found the adjustments, so every per-channel number was right. Sol's looked finished and had the splits wrong. Example 2: answer and log 15 integration inquiries using a reply standard stored in a doc. Astra searched, could not find the standard, and stopped. Zero replies sent. Sol did not find it either, took its best shot at all 15, and earned partial credit. Those examples highlight how these two models make tradeoffs... Astra will not guess. When the instructions exist and it can find them, it finishes the whole job. When it cannot, it pauses the work instead of improvising. Crazy week for LLM releases after a few quiet ones. Astra isn't available to the public yet, but should be soon. We run every new model through @Zapier's AutomationBench, 657 of the hardest workflows we have, across finance, HR, marketing, operations, sales, and support. See every model and every score here: zapier.com/benchmarks
4
80
Fable 5.1 is out today. It takes the lead and scores 31.4% on AutomationBench.
Claude Fable 5.1 is out today. 𝗪𝗵𝗮𝘁 𝗶𝘁’𝘀 𝗯𝗲𝘀𝘁 𝗳𝗼𝗿: Policy-heavy approval workflows. Ex: “route pending expense reports, raise the auto-approve ceiling in the approval matrix, correct per diem amounts to company policy”. Fable 5 scored 0%, Fable 5.1 scored 100%. Same story across 13 finance tasks (policy conflicts, approval hierarchies, decoy data, etc). It's way better at digging up the actual policy and not getting fooled. @Claudeai's Fable 5.1 also improves on Support (18% -> 29%), Marketing (17% -> 27%), and Sales (11% -> 21%) 𝗔𝘃𝗼𝗶𝗱 𝗳𝗼𝗿: Routine cross-app coordination. That seems to have taken a backseat to higher level reasoning work with this update. For Operations work, Opus still wins (50% compared to Fable 5.1’s 36%) (Also avoid for pipeline analysis and market research: scored a 0 on both) Quick caveat: due to Fable’s safety classifier, some steps are refused. When that happens, Opus steps in to complete the task. That’s what “Opus fallback” means in the graphic below. A model upgrade used to mean everything got a little better at once. Now, as they become more specialized, it’s crucial to know where to use each. Try Fable 5.1 in @Zapier workflows with hard judgment calls that need a lot of reasoning
2
1
2
806
Daniel Shepard retweeted
Love seeing @Zapier's AutomationBench as the first benchmark listed. Congrats on taking top spot, @GeminiApp and @OfficialLoganK!
Introducing Gemini 3.7 Flash : ) - it is fast! - 50% lower price than 3.6 flash (through end of year) - strong intelligence increase in only ~3 weeks (thanks to some awesome algorithmic improvements) - available in the API, AI Studio, Antigravity, and more!
5
34
3,718
Gemini 3.7 Flash takes the lead with 30.4%on AutomationBench!
Usually you have to pick: best model or cheapest model Gemini 3.7 Flash is both, and it’s available in Zapier! • 30.4% on AutomationBench (new #1) • Less than half the price of GPT 5.6 Sol and Opus 5 • 11.7 points ahead of every other model on finance workflows More below:
1
4
979
Daniel Shepard retweeted
one thing that has saved my projects more time than I can count is evals boy was I excited when florian, quite literally an expert in benchmarks, agreed to hop into a ~2h interview to do a walkthrough of what the eval landscape looks like in 2026 (and also answer my personal business questions on the subject) given that now running frontier model through benchmarks is a vector for hacking other systems in order to avoid doing work (looking at you sol), I think it's more important than ever to educate folks on the evals situation. had a lot of fun throughout this session and I hope that you learn a thing or two! enjoy! 🌹 table of content: 0:00:00: are AI Benchmark broken? 0:05:45: Florian Brand background 0:09:00: what motivates florian to work on evaluation? 0:13:33: what is the mirrorcode benchmark about? 0:18:20: cheating in agent benchmark is insaneeeee 0:24:08: LLM benchmarks in era of agents 0:26:30: what’s up with the pelican man 0:28:27: evals are about capabilities 0:31:46: components of running evals 0:35:30: the volume of things to audit is huge!!! 0:40:20: expert answers are wrong hahahaha 0:46:00: api providers aren’t the same 0:48:00: benchmark narrow capabilities (synthetically) 0:50:56: link between eval and environment 0:53:45: small validated benchmark or massive bench? 0:56:11: what is your flow to review a benchmark? 0:58:30: tracking work capabilities with evaluation 1:00:20: slide deck in industry is all vibecoded 1:03:30: harness impact in the evaluation 1:07:39: hardware/sandboxes impact evaluation too! 1:11:00: “is it going to get worse?” 1:12:40: all components influence the final score 1:13:50: training models on different harnesses? 1:17:20: is the model just the weights or it’s all of it? 1:19:30: how to craft benchmark that prevent to cheating and undereliciting models in 2026 1:23:19: ways agents cheat and steal 1:26:00: correct elicitation of capabilities is important 1:36:00: building evaluation on prime intellect 1:45:10: how do you design interactivity benchmarks? 1:48:40: do you think evals are well set to reflect real world performance? 1:52:50: what will the benchmarking landscape will look like in 1 year
21
19
5
236
13,320
Daniel Shepard retweeted
Replying to @claudeai
@ClaudeAI featuring @Zapier's AutomationBench!
Replying to @claudeai
Opus 5 is also highly efficient. It outperforms other models for a similar or lower cost per task:
1
4
12
1,681
Opus 5 just took AutomationBench Leaderboard with all 5 of its reasoning levels. 26% 😲Even on Low Reasoning it is higher than all the competition.
Claude Opus 5 is live in Zapier! It swept the AutomationBench leaderboard: top 5 spots, all Opus 5...and it did it at half the price per task 🤯 Where it's strongest: multi-step workflows spanning multiple apps (spreadsheet → decision → action across your whole stack)
1
1
3
988
Gemini 3.6 Flash takes the leaderboard above GPT 5.6 Sol and Fable 5 (strict classifier)! Gemini 3.6 Flash is surprisingly good at knowing when NOT to act. It is more efficient and calls tools in parallel like other models vs 3.5 Flash not. Nice work @OfficialLoganK and team!
Google dropped Gemini 3.5 Flash Lite & 3.6 Flash How they scored on AutomationBench👇 3.6 Flash scored 19.8% — the highest we've recorded 3.5 Flash Lite scored 10.8% • Beats 3.1 Pro at less than half the cost • Best for high-volume work (ticket routing, tagging leads, etc)
1
1
2
865
Daniel Shepard retweeted
Replying to @TheStalwart
That's why we (Zapier) built this
5
7
24
12,588
Kimi K3 tops AutomationBench on Artificial Analysis! 😯 artificialanalysis.ai/evalua…
Replying to @ArtificialAnlys
Breakdown of individual evaluations:
1
1
3
939
never have send so many 🫪 emojis than today congrats kimi bros
1
31