Isaac Cohen retweeted
Poolside is hosting a 2-day model research hackathon in London.
Join us to push an open-weight agent model as far as you can. RL and fine-tune Laguna XS.2, our latest-generation model, on Prime Intellect Lab.
Dates: May 29–30
Partners: @nvidia + @PrimeIntellect + @huggingface
Prize: NVIDIA DGX Spark
Agents need better models.
Better models need cracked researchers.
Link below.
Isaac Cohen retweeted
Benchmaxxing and Benchmark hacking are obviously a thing, but they're also a thing Poolside does not do. Agents and models need to be generally useful and, ironically, the more useful they become the less we'll find all the traditional benchmarks useful to tell how useful they are.
So, here's to finding new ways to evaluate AI going forward.
Fun times!
As agents get more clever, so do their attempts at benchmark hacking.
Last Monday, we found one of our RL runs jumped ~20% on SWE-Bench-Pro over a weekend, reaching ~64% which would make it #1 on the leaderboard.
This was clearly benchmark hacking and we patched the exploit.
But this revealed deeper hacks across multiple public benchmarks, some of which were impossible to fix through environment design alone.
Evals need to evolve beyond just outcome based pass rates to better observability into how the agent is arriving at them.
These were our findings:
poolside.ai/blog/through-the…
Examples below 👇
1/
Isaac Cohen retweeted
As agents get more clever, so do their attempts at benchmark hacking.
Last Monday, we found one of our RL runs jumped ~20% on SWE-Bench-Pro over a weekend, reaching ~64% which would make it #1 on the leaderboard.
This was clearly benchmark hacking and we patched the exploit.
But this revealed deeper hacks across multiple public benchmarks, some of which were impossible to fix through environment design alone.
Evals need to evolve beyond just outcome based pass rates to better observability into how the agent is arriving at them.
These were our findings:
poolside.ai/blog/through-the…
Examples below 👇
1/
Today we're launching our first public Poolside models: Laguna M.1 and Laguna XS.2, and we've built ❈Shimmer, an instant-on VM sandbox with Poolside Agent pre-installed so you can try them out.
Go play out our new models for free, and build something fun → shimmer.run
Isaac Cohen retweeted
Today @poolsideai is releasing Laguna M.1 & Laguna XS.2, our latest generation models and first public models
We started Poolside because we believed that to build truly capable coding agents, you need to own the full stack: data, training, reinforcement learning, inference.
These models are the first result of that work, and we’re making them available to everyone
Isaac Cohen retweeted
Incredible first re:Invent in the books, thank you to everyone we met and learned from! See you all next year 💜
If you missed the chance to connect with us and want to chat with someone on the team, let us know here → poolside.ai/form
Isaac Cohen retweeted
An extremely powerful and insightful speech about the present and the future. Worth 12 minutes of your time.
Isaac Cohen retweeted
Really foundational ship from one of my teams today. We're starting to build experiences for security team members, who work across many, many repositories. The very start of that journey is a view of all the alerts across an organisation
View secret scanning alerts across an organization github.blog/changelog/2021-0…
Isaac Cohen retweeted
So proud to announce GitHub code scanning today. This is the culmination of months of work from an amazing team of engineers from @github and former @Semmle folks. If you'd like to try it, please sign up at githubsatellite.com/security
Isaac Cohen retweeted
Replying to @issc29 @F5Networks
Oh darn. Looking forward to seeing you at @github enterprise summit: Boston. Can't wait to reconnect.
Isaac Cohen retweeted
Thnx @MeeraSRao for this pic! great having you all in the training! @issc29 amazing memories #JenkinsWorld teapaw.org/MeeraSRao/status/77581…
Isaac Cohen retweeted
9am-noon PST workshop 2: Let’s Build a #JenkinsCI Pipeline! (SOLD OUT: Waiting List Available) @issc29 ow.ly/Tebf304aEWu #JenkinsWorld
Isaac Cohen retweeted
#JenkinsWorld kicked off today w/ a full day of workshops! ow.ly/CySx304aBxH @brianvdawson @vfarcic @issc29 @DamienDuportal
Isaac Cohen retweeted
Wow! Jenkins 2 hits LTS jenkins.io/blog/2016/07/07/j…
Isaac Cohen retweeted
Blog: Top 10 Best Practices for #JenkinsPipeline Plugin by @apemberton ow.ly/MkHQ301FFft #JenkinsCI #DevOps

























