Data & Eval Lead, Gemini at Google DeepMind. Writing about AI evaluation, data quality, and building useful AI. Personal views
New York
Joined December 2008
- Tweets216
- Following402
- Followers817
- Likes334
Pinned Tweet
Proud of team’s contribution to Gemini 4 Carbon.
Especially happy to see SOTA on:
• Terminal-Bench 4.0: higher accuracy; cheating attempts down to 0.4%.
• Terminal-Bench-Science: 64.8%.
How the model gets the answer matters too.
blog.google/innovation-and-a…
ALT Official Google benchmark table for Gemini 4 Carbon, with Carbon’s column highlighted in blue. Terminal-Bench 4.0 (AA): Carbon 65.2%, Argon 57.1%, GPT-6 Astra 59.1%, Claude Opus 5.5 59.6%, Claude Fable 5.1 52.0%. Terminal-Bench-Science 0.1 (AA): Carbon 64.8%, Argon 52.9%, GPT-6 Astra 63.3%, Claude Opus 5.5 59.0%, Claude Fable 5.1 43.3%. Other rows cover general capabilities, coding, computer use, knowledge work, multimodal understanding, long context, science and math, and cybersecurity.
Amazing work by Ben and team.
We're super excited to launch MKA in partnership with @ArtificialAnlys.
Real-world work is messy and inherently visual. We kept finding existing benchmarks were either not visual enough or not agentic enough. MKA aims to fill this gap.
9
381
Sumit Bagri retweeted
Gemini 4 Carbon is now available for users, developers and enterprises!
We are heavily using Carbon internally at Google for hardest long horizon tasks, our teams have used it to help mathematically prove the core cryptographic protocol of Signal — and it’s topping independent benchmarks like CursorBench 4.0, Perplexity’s WANDR, Zapier’s AutomationBench, and more.
The team also had fun using Carbon to analyze images and technical drawings from humanity’s first crewed lunar lander, the Apollo Lunar Module. Not a single flown component ever returned to Earth. It recreated the 3D asset from those historic documents and generated this walkthrough video.
Read more: blog.google/innovation-and-a…
Sumit Bagri retweeted
Today we’re rolling out Gemini 4 Carbon, our strongest model with frontier performance in cybersecurity capabilities and long-horizon agentic workflows.
We’ve poured a lot of love into it, and it’s very good at working through hard problems.
Sumit Bagri retweeted
Thank you Josh! Asked Gemini how to stay young, it said "keep working." Fair enough.
Sumit Bagri retweeted
Exciting news: Gemini 4 Carbon (High) from @GoogleDeepMind just made a huge debut on Arena, landing #1 in Text Arena, #2 in Agent Arena, and #4 in Code Arena: WebDev. Back at the top, Gemini 4 Carbon delivers frontier intelligence and reshapes the Pareto frontier across three arenas.
In Agent Arena, Gemini 4 Carbon ranks #2 with a +16.2% net improvement across +21.9K real-world agentic sessions from our global community, surpassing top models GPT-6 Astra and Fable-5.1. At $1.80 median cost per task, Gemini 4 Carbon (High) costs 72% less than the #1 ranked Claude Opus 5.5 (Max).
Its standouts by key signals:
- #1 Steerability (+18.6%): can the model course-correct when you push back?
- #2 Praise vs Complaint (+37.9%): implicit sentiment in users' reactions
- #3 Confirmed Success (+18.7%): explicit “yes, that worked” from users
In Code Arena: WebDev, Carbon scored 1768 pts landing #4:
- Just 2 pts below #3 Claude Sonnet 5.5 (xHigh), at 1770 pts
- 89 pts above #11 Gemini 4 Argon (High), at 1679 pts
By category, it ranks #3 in Gaming and Simulations, and #5 in Brand & Marketing.
In Text Arena, Gemini 4 Carbon (High) is the new #1 with 1528 pts, now ahead of Gemini 4 Argon (High) at #2 with 1520 pts.
Congrats to the @GoogleDeepMind team on this huge milestone!
Sumit Bagri retweeted
Google’s Gemini 4 Carbon is the new #1 on the Artificial Analysis Intelligence Index v5, and second only to Claude Opus 5.5 on the Artificial Analysis Coding Agent Index
Gemini 4 Carbon scores 59 on Intelligence Index v5, driven by leading performance in complex reasoning, knowledge, and scientific tasks. It also has large gains in coding and knowledge work, particularly when compared to other Gemini models, but it is not leading in all these areas.
As with launch of Gemini 4 Argon, Gemini 4 Carbon is being rolled out to selected users and is not yet publicly available. Google is offering a 50% launch discount, with no confirmed end date. All our charts and figures use standard pricing for both Gemini 4 Carbon and Argon.
Key results for Gemini 4 Carbon with high reasoning:
➤ Gemini 4 Carbon leads the Intelligence Index v5, with the #1 score in 4 of the 9 Index evaluations: HLE-Diamond, Terminal-Bench-Science, AA-Omniscience, AutomationBench-AA. Carbon makes substantial gains elsewhere but is not leading in coding and knowledge work
➤ Behind only Claude Opus 5.5 on coding agents: With Antigravity CLI, Gemini 4 Carbon (high) scores 59 only 3 points behind Claude Code with Claude Opus 5.5 (max, 62) on the Artificial Analysis Coding Agent Index. Carbon ranks second on both Terminal-Bench 4.0 (64%) and FrontierCode v1.1 (51%), behind Claude Opus 5.5 (max) at 56%. Google has historically underperformed in this area and so this improvement stands out
➤ Improved agentic knowledge work, but not yet leading: On AA-Briefcase, Gemini 4 Carbon (high) scores 1609 Elo, a 126-point improvement over Gemini 4 Argon (high, 1483), but still behind models including Grok 4.7 (xhigh, 1638), Muse Wonder (max, 1659) and Kimi K3.1 Pro (1678). The largest gain comes from Presentation, up 293 Elo to 1598, alongside a 120-point improvement in Analytical Quality to 1671
➤ Lower Cost per Task than Claude Opus 5.5 for comparable overall intelligence: Gemini 4 Carbon costs $7.76 per Intelligence Index task, ~23% less than Claude Opus 5.5 (max, $10.04) but ~1.2x GPT-6 Astra (max, $6.25). It uses 123k output tokens per task, 29% fewer than Claude Opus 5.5 (max, 173k). At the same $4/$20 standard pricing, its Cost per Task is in line with Gemini 4 Argon ($7.68), while scoring 9 points higher on the Intelligence Index
➤ Leading AA-Omniscience v1.1 score, with low hallucination: Gemini 4 Carbon scores 54 on our recently updated AA-Omniscience v1.1, 6 points ahead of Claude Opus 5.5 (max, 48) and 8 points ahead of Gemini 4 Argon (46). Accuracy (% correct) rises 10 points over Argon to 58%, while its hallucination rate rises from 4% to 11%. This is still the second lowest of any model, behind only Argon, compared with 50% for both Claude Opus 5.5 (max) and GPT-6 Astra (max)
Key model details:
➤ Context Window: 1M tokens
➤ Multimodality: Text, image, video, and speech input, with text output
➤ Pricing: $4/$20 per 1M input/output tokens, the same as Gemini 4 Argon’s standard pricing, with a 50% discount. Cached input tokens receive a 95% discount ($0.20 per 1M)
Congratulations to the @GoogleDeepMind team!
Sumit Bagri retweeted
Excited to announce Gemini 4 Carbon, our next frontier model which is a step up from Argon across all capabilities! The benchmarks don't tell the full story -- we have been using it internally and it's been an upgrade for us.
I did say we'd be releasing our models periodically...introducing Gemini 4 Carbon, our next frontier model.
Following Argon's release just two weeks ago, Carbon shows the pace at which we’re pushing the frontier, it ranks #1 on the recently updated v5.0 of @ArtificialAnlys Intelligence Index at 59. It also leads across key cybersecurity evaluations scoring 88.2% on SEC-bench Pro for finding vulnerabilities, 54.8% on ExploitGym for exploitation, and 71.0% on CWE-bench for patching.
We’re seeing significant improvements in long-horizon tasks even compared to Argon, and we’ve strengthened frontier safeguards.
As we did with Argon, we’re going to roll it out gradually + responsibly. Some trusted cyber defenders will get access starting this week. We'll be expanding it to more partners through our Fairwind Program, ahead of wider availability soon, so hang tight!
AI’s first big win in a field might be showing experts where to look.
Jake’s proposed use for Argon: scan hieroglyphs and flag interesting passages for review.
The eval: useful findings per expert-hour—and important passages missed.
Nothing rigorous, but when testing it on a few examples, it could usually piece together the correct meaning from the context, even when it missed some of the symbols
Other models I've tried just hallucinate. I feel like Argon is potentially good enough to do a first pass over a bunch of scans to flag the interesting sections for human review
7
563
Sumit Bagri retweeted
Gemini 4 Argon sets a new record of 69% on HieroglyphBench
This is the first model that performs accurately enough to be useful
Scientists: want to try Gemini 4 Argon in your research?
Limited cohort of scientists will get six months of free access to Gemini 4 Argon via Google Antigravity. Apply for a spot.
Register interest: docs.google.com/forms/d/e/1F…
Reach out to @divy93t or DM me with what you'd like to try.
We're significantly expanding access to Gemini for scientists via antigravity - sign up now!
Top of mind question for scientists on data retention - please turn off telemetry if you'd like.
docs.google.com/forms/d/e/1F…
1
3
588
Proud of our team's work on Gemini 4 Argon.
What excites me is its range: tackling complex codebases to helping scientists choose the next experiment worth running.
Try it on a hard problem in your field. I'd love to hear how it does.
Very excited for you all to try Gemini 4 Argon across API, Antigravity and Gemini App. Eager to hear your feedback and to see what you build with it!
3
299
Sumit Bagri retweeted
Very excited for you all to try Gemini 4 Argon across API, Antigravity and Gemini App. Eager to hear your feedback and to see what you build with it!
A coding agent can pass its tests and still break an existing client.
An eval I'd run: rename a field across the database, API and UI while preserving old clients and data.
Use acceptance tests the agent can't rewrite. Track regressions and human review time.
Proud of our team’s work on Gemini 4 Argon.
Top 5 on 20 of 22 benchmarks in Vals’ evaluation—a result worth celebrating.
From an eval perspective, I’m excited to see how that breadth translates into useful workflows.
What would you put Argon to work on first?
Gemini is on top of the Vals Index for the first time.
Gemini 4 Argon takes the #1 spot on the Vals Index at 68.9%.
2
3
507