Isaac Cohen (@issc29) profile picture

Head of Field Services, Postman

New York, USA
Joined April 2009
Photo by Isaac Cohen (@issc29)Photo by Isaac Cohen (@issc29)Photo by Isaac Cohen (@issc29)Photo by Isaac Cohen (@issc29)Photo by Isaac Cohen (@issc29)Photo by Isaac Cohen (@issc29)Photo by Isaac Cohen (@issc29)Photo by Isaac Cohen (@issc29)Photo by Isaac Cohen (@issc29)Photo by Isaac Cohen (@issc29)
Isaac Cohen retweeted
Poolside is hosting a 2-day model research hackathon in London. Join us to push an open-weight agent model as far as you can. RL and fine-tune Laguna XS.2, our latest-generation model, on Prime Intellect Lab. Dates: May 29–30 Partners: @nvidia + @PrimeIntellect + @huggingface Prize: NVIDIA DGX Spark Agents need better models. Better models need cracked researchers. Link below.
27
43
16
230
95,119
Isaac Cohen retweeted
Benchmaxxing and Benchmark hacking are obviously a thing, but they're also a thing Poolside does not do. Agents and models need to be generally useful and, ironically, the more useful they become the less we'll find all the traditional benchmarks useful to tell how useful they are. So, here's to finding new ways to evaluate AI going forward. Fun times!
As agents get more clever, so do their attempts at benchmark hacking. Last Monday, we found one of our RL runs jumped ~20% on SWE-Bench-Pro over a weekend, reaching ~64% which would make it #1 on the leaderboard. This was clearly benchmark hacking and we patched the exploit. But this revealed deeper hacks across multiple public benchmarks, some of which were impossible to fix through environment design alone. Evals need to evolve beyond just outcome based pass rates to better observability into how the agent is arriving at them. These were our findings: poolside.ai/blog/through-the… Examples below 👇 1/
2
2
15
2,906
Isaac Cohen retweeted
As agents get more clever, so do their attempts at benchmark hacking. Last Monday, we found one of our RL runs jumped ~20% on SWE-Bench-Pro over a weekend, reaching ~64% which would make it #1 on the leaderboard. This was clearly benchmark hacking and we patched the exploit. But this revealed deeper hacks across multiple public benchmarks, some of which were impossible to fix through environment design alone. Evals need to evolve beyond just outcome based pass rates to better observability into how the agent is arriving at them. These were our findings: poolside.ai/blog/through-the… Examples below 👇 1/
8
22
10
107
17,923
Isaac Cohen retweeted
Today we're launching our first public Poolside models: Laguna M.1 and Laguna XS.2, and we've built ❈Shimmer, an instant-on VM sandbox with Poolside Agent pre-installed so you can try them out. Go play out our new models for free, and build something fun → shimmer.run
19
30
5
235
53,521
Isaac Cohen retweeted
Today @poolsideai is releasing Laguna M.1 & Laguna XS.2, our latest generation models and first public models We started Poolside because we believed that to build truly capable coding agents, you need to own the full stack: data, training, reinforcement learning, inference. These models are the first result of that work, and we’re making them available to everyone
44
40
5
389
53,194
Isaac Cohen retweeted
Incredible first re:Invent in the books, thank you to everyone we met and learned from! See you all next year 💜 If you missed the chance to connect with us and want to chat with someone on the team, let us know here → poolside.ai/form
3
1
21
19,462
Isaac Cohen retweeted
We're 12 days out to our very first AWS re:Invent. Find us at booth #708 with demos, and exclusive merch – come say hi! poolside.ai/reinvent
6
2
12
5,451
Isaac Cohen retweeted
. @issc29 saves a live demo with a @getpostman mock server at #GitHubUniverse 🏆
2
17
1,909
Isaac Cohen retweeted
An extremely powerful and insightful speech about the present and the future. Worth 12 minutes of your time.
Crushed it @KonstantinKisin Dare I say even topping the Oxford union speech Listen 👇🏼
163
1,123
59
6,090
1,698,048
Am Israel Chai 🕊️🙏🇮🇱🤍🩵
27
18
3
174
36,981
Isaac Cohen retweeted
Really foundational ship from one of my teams today. We're starting to build experiences for security team members, who work across many, many repositories. The very start of that journey is a view of all the alerts across an organisation
2
2
18
Join me today for Office Hours as we talk through and demo GitHub Code Scanning!
Introducing GitHub Office Hours: Join us this Wednesday the 15th at 11am PT on Twitch. We’ll be tackling software development challenges by topic each week, including security, DevOps, and more. Code scanning is up first. twitch.tv/githubenterprise?u…
1
So proud to announce GitHub code scanning today. This is the culmination of months of work from an amazing team of engineers from @github and former @Semmle folks. If you'd like to try it, please sign up at githubsatellite.com/security
1
12
59
Isaac Cohen retweeted
Replying to @nntaleb
@nntaleb released Black Swan right before the 2008 financial crisis and now released Skin in the Game right before tokens added skin in the game for everything. Interesting timing.
2
6
57
Isaac Cohen retweeted
Replying to @issc29
@issc29 from @github doing his thanggg talking about #HuBot ! Great to integrate into your #slack channels especially for #culture #github #devops #devopsnyc
2
2
#GitHub now alerts you on dependency vulnerabilities in your repo! teapaw.org/github/status/93122040…
Now GitHub can help you find and fix vulnerabilities in your dependencies. Just enable your dependency graph to start receiving security alerts—currently supported in Ruby and JavaScript: github.com/blog/2470-introdu…
1
Thanks GitHub! We think so too! /cc @issc29 and @TAGB_TA :) read.bi/2ynQ6RP
4
1
Replying to @issc29 @F5Networks
Oh darn. Looking forward to seeing you at @github enterprise summit: Boston. Can't wait to reconnect.
1
1