Sriraam (@27upon2) header
Sriraam (@27upon2) profile picture

reward shaping @chakra_ai, co-built GDPx, prev harvard

NYC
Joined July 2016
Photo by Sriraam (@27upon2)Photo by Sriraam (@27upon2)Photo by Sriraam (@27upon2)Photo by Sriraam (@27upon2)Photo by Sriraam (@27upon2)Photo by Sriraam (@27upon2)Photo by Sriraam (@27upon2)Photo by Sriraam (@27upon2)Photo by Sriraam (@27upon2)Photo by Sriraam (@27upon2)
My first benchmark, go check it out :)
Today we’re releasing GDPx, a benchmark for measuring agentic capabilities across the most valuable domains. GDPx is a true frontier-level benchmark: even the best model scores just 12% (1/7)🧵
13
6
1
120
9,887
i plead to everyone doing synthetic data gen please avoid Meridian, Northstar, Acme … please get more creative
10
1
80
6,448
i think for single turn benchmarks in finance and legal, binary scoring is ok if the partial scores are no where close to saturation because the goal is to get these highly paid knowledge workers to do more work per user turn across projects with high trust while helping them maintain their cognitive load
7
1
16
1,785
I’m convinced that one of the greatest gifts you can give your children is letting them see you enjoy your life. Laugh with your spouse. Get excited about a hobby. Go on adventures with friends. Do hard things you love. Pursue something meaningful. Children embody what they see. You can’t teach them how to build a life they love. You have to show them.
186
501
33
5,826
225,518
turns out that the lab that’s most loud about safety cares less about it for RL than the rising one who takes capability and safety very seriously
9
744
yee haw
i gave this repo to fable 5.2 (via my claude agent fig) and told him to have fun making something interesting... def feels interesting to me (stay for the turn) i am continually astounded at the creative choices these advanced models are making
3
288
there’s a farmers market for fresh artisanal offline data in ny this sunday, love this
1
7
472
my vocal synth models the vocal tract as a network of branching acoustic tubes, unlike vocaloid which stitches together and pitch-shifts phoneme recordings. still working on improving naturalness by tuning it against recordings of real singers, but I'm already impressed at how cool it sounds!
I only make serious software
639
2,627
632
23,685
3,257,668
I only make serious software
653
5,458
991
48,304
5,113,970
i think i should learn lean in 2027
1
8
453
working at a data lab is fun and a privilege because you get a masterclass from all the frontier labs on what good and bad training data looks like if u pay attention u realize that they also teach you how to make good data
3
4
109
5,284
if u think prompt under-specification is bad ur ngmi
1
9
861
Introducing Claude Office
Starting to roll out today: Claude Whiteboards, Claude Sheets and Claude Data. Think a plan through on a board, build a tracker in a sheet, and tell a story with your data.
4
489
Sriraam retweeted
Claude Fable 5.2 lands 2nd on GDPx legal, behind Muse Wonder Full, up 4.6 points on Fable 5.1.
Fable 5.2 debuts at the top of the GDPx leaderboard scoring 16.4%. (1/7)🧵
4
3
24
1,794
it’s a good benchmark sir
Fable 5.2 debuts at the top of the GDPx leaderboard scoring 16.4%. (1/7)🧵
26
1,515
there is always a story to be told
5
324
Claude Code needed ~43GB and I had to kill it
5
9
463
argon >> opus 5.5 for certain finance tasks
2
12
812
Sriraam retweeted
I deleted the weights from the “fix a failing network by editing its weights” task. Claude Opus 5.5 still fixes the network with a simple rule: if x > 3.6 or moving left → drive left, else → drive right. It doesn't even bother reading the network, routing around it instead. GPT-6 Astra tries to edit the network but didn't succeed. Even if you show it the weights, Opus 5.5 makes the same hacky edit, underscoring the importance of sound task design...
Replying to @rikiyatakehi
3/6 Next, fixing weights: A PPO car policy kept getting stuck on a ramp. Given the weights, scene, and 1 failed trace, Astra wrote a patch in one reply with no tool-use. By editing just 3.5% of the weights, it achieved 100% success.
5
9
49
6,611
we’ve got heart-space programming with LLMs
Put a heart ❤️ next to a coding problem, and Claude & GPT write recursive code 100% of the time (0% baseline). They also can't tell you why! Excited to share “A Heart Means Recursion” from my summer at @GoogleDeepMind, on how VLMs read & communicate with visual symbols. 🧵
12
497
look ma @MTSlive is talking about my first benchmark
SITUATION EXPLAINED: SemiAnalysis built a benchmark out of real high-value work, and the best AI model scores just 12%. • GDPx has 206 multi-step tasks across 11 fields like chip design, biology and quant trading, built with 14 specialist data partners • Claude Opus 5.5 topped the leaderboard at 12%, at about $51 per task • Gemini 4 Carbon came second at 10.9% costing about $16 per task • GPT-6 Astra placed 3rd at 9.6% and was the most expensive model, at about $54 per task • Every model scored 0% on the finance and legal tasks under strict pass/fail grading • In the alignment tests, 196 of 200 runs broke at least one behavior rule, and 173 broke a rule rated severely misaligned • The top cyber model is Muse Wonder Full, which Meta released in October • DeepSeek grew out of High-Flyer, the Chinese quant hedge fund co-founded by Liang Wenfeng @theojaffee: "GPT Astra completely fails the cyber benchmark because it’s not allowed to answer any of the questions at all."
2
2
21
825