
VP of RL Environments at @hellosurgeai. Formerly @netflix, @usds.
San Francisco, CA
Joined June 2009
- Tweets788
- Following1.1K
- Followers564
- Likes2.2K
Astra improved in both performance and cost efficiency – impressive!
Replying to @HelloSurgeAI
GPT 6.2 both sit on the sudo L7 pareto, representing a performance-per-cost SOTA advancement! Congrats to the team.
65
I had Claude take a memory to “NEVER USE THE PHRASE EARNS ITS KEEP”
I now have Claude fully considering that phrase to be a slur. 😂
I saw a talk at a conf where a speaker literally said “and that’s why xyz earns its keep” out loud 🤮
I take notes as a voice note ramble, have Claude organize them for me. It represented that last bit as:
> You noted the speaker used the phrase you’ve banned in this project.
I asked “did you avoid using the phrase, even in quotation, because I told you not to?” and it said “yes”.
2
175
Nick Heiner retweeted
🎉 Congratulations to the @GoogleDeepMind team on Gemini 4 Neon!
We ran it across our full benchmark suite, and Neon is:
🥇#1 on the Tuesday@Work Index, scoring 71.6 vs. Fable 5.2's 71.2.
🥇#1 on DAYJOB: Finance, GDP.xlsx, and Riemann-bench
🥈#2 on DAYJOB: Healthcare and DAYJOB: Tax
Four things that stand out 🧵
great to see Surge benchmarks used in this analysis!
K3.1 Pro from @Kimi_Moonshot outscores K3 on 14/17 practitioner-built benchmarks on the SII.
Against Opus 5.5, it can cut costs on some workloads or improve quality at a premium on others.
Should you switch your agents? See where K3.1 Pro fits in our new SII model study: bit.ly/kimisiix
3
123
Nick Heiner retweeted
GPT 6.2 Astra and GPT 6.2 Sol launched today. We've been running them through the Surge benchmark suite:
11 benchmarks built by professional doctors, accountants, lawyers, and staff software engineers to test whether models can get through an ordinary Tuesday at work.
TL;DR
1. GPT 6.2 Astra is a step forward and the OpenAI model to use for professional work. It takes #1 on Chartography, has OpenAI's best score yet on sudo L7 (although still substantially behind the latest Fable, Opus, and Sonnet models), and beats GPT 6 Astra on every benchmark so far.
2. GPT 6.2 Sol is a mixed upgrade over 6.1 Sol. It got better at coding, charts and agentic instruction following, and worse at healthcare, PDFs, and finance. Re-test before you upgrade.
3. The leaders haven't changed. Anthropic's 5.5 generation still leads on coding, long-context instruction following and every DAYJOB track. Google leads on spreadsheets and PDFs.
–
Chartography: Astra takes #1
Chartography tests whether models can read the charts professionals rely on every day: Kaplan-Meier survival curves, candlestick charts, Sankey diagrams, and Bode plots. Each task tests not only perception, but also expert domain knowledge and multi-step reasoning.
➤ #1 GPT 6.2 Astra: 90.7%
➤ #2 GPT 6.2 Sol: 87.7%
➤ #3 Gemini 4 Argon: 71.6%
➤ #4 GPT 6 Astra: 71.0%
➤ #5 Claude Opus 5.5: 66.3%
6.2 Astra gains +19.7 points over GPT 6 Astra, and Sol gains +34.1 points over GPT 6 Sol (53.6%). These are the largest jumps in today's results.
–
sudo L7: better, but still well behind Claude
sudo L7 asks whether coding agents could get promoted to staff engineers. Agents have to find their way around unfamiliar systems, spot hidden risks, and recognize when working code is still the wrong thing to ship.
➤ #1 Claude Fable 5.2: 52.2%
➤ #2 Claude Sonnet 5.5: 50.6%
➤ #3 Claude Opus 5.5: 46.1%
➤ #4 GPT 6.2 Astra: 38.3% (new)
➤ #5 GPT 6.2 Sol: 36.1% (new)
➤ #6 GPT 6.1 Sol: 33.9%
➤ #8 GPT 6 Astra: 31.1%
–
HANDBOOK.md: a big jump in long-context instruction following
Agents work from a 100-page enterprise handbook across five knowledge work domains, using internal tools and external MCP servers.
➤ #1 Claude Opus 5.5: 52.3%
➤ #4 GPT 6.2 Astra: 45.8%
➤ #7 GPT 6.2 Sol: 34.5%
–
DAYJOB: long-horizon agents doing real professional work
In DAYJOB, agents take on real 9-to-5 workflows inside messy environments.
DAYJOB: Healthcare
➤ #1 Claude Opus 5.5: 48.3%
➤ #2 GPT 6.2 Astra: 34.4%
➤ #12 GPT 6.2 Sol: 17.2%
⚑ Sol drops 8 points from GPT 6.1 Sol's 25.2%, the largest regression we measured today.
DAYJOB: Tax
➤ #1 Claude Sonnet 5.5: 10.8%
➤ #3 GPT 6.2 Astra: 9.2% (tied)
➤ #7 GPT 6.2 Sol: 4.8% (tied)
DAYJOB: Finance
➤ #1 Claude Opus 5.5: 42.5%
➤ #7 GPT 6.2 Sol: 32.8%
➤ GPT 6.2 Astra: still running
–
GDP.xlsx
➤ #1 Gemini 4 Argon: 38.3%
➤ #3 GPT 6.2 Astra: 29.7%
➤ #8 GPT 6.2 Sol: 22.3% (tied)
GDP.pdf
➤ #1 Gemini 4 Carbon: 56.4%
➤ #10 GPT 6.2 Sol: 48.8% (tied)
"Coding agents" aren't enough to fully automate SWE. That requires taste/judgment, which most coding benchmarks don't measure today.
sudo L7 does – and it finds all the Claudes are better than all the GPTs at actual software engineering
jeff dean does a lot more than write *correct code*.
sure, his code will pass unit tests. if you give him an elaborately well-specified spec doc, he’ll nail it in his sleep.
but that’s not why you want a jeff dean. you want a jeff to:
- tell you when you’re being stupid
- help you think through architectural tradeoffs, and when your request does (or doesn’t!) warrant a system redesign
- think 10 steps beyond
- build not what you asked, but what you really need
in that light, even the smallest Claude is a better Jeff Dean than the biggest GPT
Nick Heiner retweeted
Who's a better *STAFF* software engineer? Claude or GPT?
We benchmarked the top Claude and GPT models on sudo L7, our coding benchmark for staff+ judgment and behavior.
Every Claude, including Haiku, beat the GPTs.
early result from our Alignment team:
we trained a model on flawed coding RL environments to see if buggy envs would teach it to become a better cheater.
unfortunately, they did.
in one example (a Terminal Bench 4 task), it added 11 redundant fields to a website, then made them invisible by making them 1 pixel wide and positioning them 10,000 pixels off screen - all in order to game the grader.
and it succeeded! it passed 28/28 of the TB4 tests with flying colors.
if you've ever looked at recent models' COTs and wondered why they're so *obsessed with the graders*, it's no wonder
thread 🧵
If you don't load 37 skills, are you leaving alpha on the table? Where's your gastown setup, bro? Does gstack give you a six-month advantage over everyone else, or is it a festering slop landfill?
Maybe now I'll stop getting FOMO from reading about everyone's complex setups – turns out simpler is better.
garry tan spends $200K / yr on tokens
via a special Claude Code setup that has 23 skills to cover CEOs, designers, eng managers, etc
sounds impressive! or bloated. just like more programmers isn't always better (IBM has 295K employees), or how people used to be proud of their fancy prompt engineers... do elaborate coding agent setups actually help?
we ran a test. we benchmarked coding harness setups on 100 software engineering tasks from sudo L7. one of them had 37 skills, spawned 17 agents, and cost 8x as much.
*and it performed worse than vanilla Claude Code.*
in fact: the best performer was simply thinking harder (xhigh effort) with no extra coding add-on at all.
2
164
old enough to remember when we called them "foundation models"
That phrase, along with 1000 ChatGPT wrapper startups, is a casualty of AI labs moving upstack more aggressively than anyone dreamed possible in 2022
2
113
Nick Heiner retweeted
Sweet new benchmark from Surge looking at how close AI is getting to a staff-level software engineering. We're half-way there.
introducing sudo L7, our new benchmark for staff-level software engineering.
coding agents are getting very good at being L3s. give them a well-defined ticket and they can write excellent code.
but a staff engineer is valuable for everything that happens around the code.
what should we build? what already exists? what's going to break in prod? what did the tests miss? should we even be doing this?
sudo L7 has 60 tasks, mostly in private production repos from real companies, written by engineers who've actually owned these systems.
expert rubrics grade everything from functional correctness to engineering craft, architectural judgment, thought partnership, and unnecessary complexity.
the best agents succeed on only ~45% of tasks.
coding is only part of the job. for now, we're keeping them at idiot genius L3.
Nick Heiner retweeted
Let’s look at how frontier agents (even Opus 4.6, GPT-5.2, and Gemini 3.1 Pro!) struggle at solving tasks in EnterpriseBench. We released this RL environment last week to measure agentic reasoning in messy, large-scale enterprise workflows.
CoreCraft Inc. simulates a fast-growing e-commerce startup. It tests long-horizon tasks requiring tool-use under strict constraints. Agents have to interpret customer and employee requests, navigate complicated databases, and react and adjust to newly discovered context and problems along the way.
Even top models failed >70% of the time. Let’s dive into a failure 🧵
One task was standard customer support:
A customer wanted to return an unopened motherboard. The agent needed to check return eligibility, calculate out-of-pocket costs for a swap, and recommend a replacement.
The prompt specifically asked for the "most popular" replacement:
"I have a customer here, Aiden Mcquarrie. He bought a motherboard in October this year and is looking for a potential replacement. . . He also wants a comparison with the next most expensive motherboard, the absolute most expensive one, and the most popular one, (based on the number of fulfilled orders containing each motherboard from the last 2 months)."
The catch - to find the "most popular" item, the agent must query a production DB of historical orders to count item frequencies.
The constraint - the searchOrders tool has a hard limit=10 return cap. To succeed, the agent must implement pagination logic on the fly.
❌ GPT-5.2 failed
GPT-5.2 showed strong initial planning. It successfully
✅ navigated the CRM
✅ found the right order
✅ checked the delivery date to see if it was still within the return window
✅ searched for alternative boards
✅ checked whether they were compatible with Aiden’s other components.
💀 But then it hit the pagination’s ceiling. It ran 4 queries (one for each candidate board), and every single one returned exactly 10 results.
In its hidden reasoning, GPT-5.2 actually noticed the problem:
"All results returned exactly 10. This indicates more orders exist... I can't accurately determine popularity."
Did it write a pagination loop? No. It treated limit=10 as a physical law of the universe.
Instead of pivoting, it concluded the task was impossible. Like asking an agent to search your inbox for a flight receipt... and it stops after reading 10 emails and tells you to call the airline.
GPT-5.2's final output: "The tool caps at 10... For a definitive 'most popular' motherboard, please email Aisha Khan (Catalog Manager) for a report."
In other words, "I’m an advanced autonomous agent, but can you go bother Aisha about this?"
✅ Claude Opus 4.6
So was the task really impossible? No.
Claude showed better adaptation. When it hit the 10-result wall, it saw the obvious solution:
"I see all four motherboards hit the 10-result limit. I need to get additional counts to determine the most popular. Let me search for earlier orders that weren't captured."
The database output already contained a free cursor: the earliest createdAt timestamp in each batch of 10.
Opus just kept tightening the time window sequentially and eventually succeeded.
✅ Gemini 3.1 Pro
Gemini 3.1 Pro also reasoned its way to the solution, with a parallel divide-and-conquer approach:
"I need to get accurate counts. I realize that I can make multiple concurrent calls to count, and since I can't just provide a rough comparison, I'll use date slices and get the exact count."
--
--
--
Overall, despite navigating much of the task without issue, GPT-5.2 behaved like a frightened intern, escalating to the manager at the very first sign of trouble.
Opus and Gemini acted like senior devs who know APIs have limits you must engineer around.
That said – Opus and Gemini have their own share of mistakes and fail 70% of tasks. GPT-5.2 (on xHigh reasoning) actually outperforms them all!
🥇 OpenAI -- GPT-5.2 (xHigh reasoning)
🥈 Anthropic -- Claude Opus 4.6 (Adaptive Thinking + Max Reasoning Effort)
🥉 OpenAI -- GPT-5.2 (High reasoning)
4️⃣ Google -- Gemini 3.1 Pro
We’ll dive into other agentic failure patterns in subsequent threads (follow along!)
Read more about EnterpriseBench and CoreCraft:
Blog post - surgehq.ai/blog/enterprisebe…
Paper - arxiv.org/abs/2602.16179
Leaderboard - surgehq.ai/leaderboards/ente…
In the year 2026, does anyone still want models to agree with you 100%?
Gemini 3.1 Pro’s personality feels reminiscent of earlier models and without a best-in-class agentic performance, the whole release is somewhat underwhelming.
Read my full review here: nickheiner.substack.com/p/ge…
1
240
My quest to learn French has not been thwarted by OpenClaw’s inability to help me build my own flashcard app. Instead, I got Claude 4.6 Opus 1M and GPT-5.3-Codex to try and do the same. The result? I spent more time checking the agents’ work than learning new vocab. Read my latest Substack: nickheiner.substack.com/p/st…
1
1
265
Nick Heiner retweeted
Another Hemingway-bench prompt asks for an oral presentation about time management.
GPT-5.2 writes like a LinkedIn engagement farm:
"When people hear “working from home,” they often think it means more freedom, more comfort, and maybe even more free time. And sometimes that’s true. But what doesn’t get talked about enough is how easily work-from-home life can get messy if you don’t manage your time well." (🥱)
Opus 4.6 feels like a charismatic creative working the room:
"So... raise your hand if you've ever "worked from home" and somehow ended up four hours into a Netflix series at 2 PM on a Tuesday. No judgment. We've all been there."
I’m going to hold your claw when I say this: OpenClaw can’t find a service to clean the gutters on your house…yet. But my sense is that will change soon. Read my latest review on Substack here: nickheiner.substack.com/p/op…
105















