https://xitter.cf/t.co/CMvN28h465 prev mts intern @mercor, @plato_rl, se @uwaterloo
sf
Joined April 2022
- Tweets48
- Following1.1K
- Followers435
- Likes3.1K
Pinned Tweet
Who grades the graders?
LLM-as-judge on short answers is well studied. Agent judges are not: they open the files, run the code, and investigate before ruling on another agent's work.
Introducing Arbiter-Bench: 71 real agent runs that frontier agent judges get wrong.
raphael lee retweeted
Introducing DeepRSI, a new industry standard for measuring recursive self-improvement.
On preexisting benchmarks, tasks are contributor-based and short-horizon (~12 hrs). In DeepRSI, we push agents to work 48 hours on ML tasks actually derived from ongoing proprietary research.
raphael lee retweeted
Today we're launching WebCloneBench: a benchmark which measures a model's ability to explore, copy, build and test various web apps.
For each task, we give the model a reference site from one of Gmail, Google Sheets, Jira, DocuSign, Shopify Admin, Craigslist, Figma, Slack and Zendesk. It is instructed to build a fully feature-complete clone of every part of the application with an 8 hour time limit.
Of 232 runs where a frontier judge disagreed with the benchmark, only 71 survived a blind label audit (3+ model families, a human ruling on every split vote).
About a quarter were the benchmark's error: hidden criteria, broken checkers, unstated rules.
Half the set is public with every judge's full logs, a scorer for your own judge, and the audit trail. 35 items are held out. Every case is a runnable @harborframework task, so you can point your own judge at the set and score it with one command.
Full writeup: heuristic.build/blog/arbiter…
raphael lee retweeted
so called high quality data showing gains on evals so flawed where not a single sample is surviving basic scrutiny is a sight to behold
raphael lee retweeted
Tech has decided that regulation is the enemy of progress, but innovation requires capital
Regulation builds the trust necessary to inject billions into markets
To enable free innovation, Andera raised a $37M series A led by Lightspeed to scale financial oversight with agents
raphael lee retweeted
Over 1 billion sandboxes have been launched on Modal.
Since launching three years ago, we've seen Modal Sandboxes become foundational to how AI is being built.
Today, teams like @Lovable, @tryramp, @cognition and more are using Modal Sandboxes to power everything from coding platforms and background agents to RL infrastructure at scale.
raphael lee retweeted
meet Shadow, a powerful open-source background coding agent!
fully featured with a remote environment, codebase indexing/wikis and subagents to understand, write and test your code, directly making PRs to github
built in a few weeks w/ @ishaandey_, @ElijahKurien
apparently doomscrolling improves my sleep quality. do I trust my whoop or @bryan_johnson?
This weekend, we built a robotic wearable that helps Parkinson’s patients walk independently.
A 1D CNN + LSTM model predicts freezing of gait in real-time and triggers mechanical stimulation to restore movement.
1st Place + Hacker’s Choice @ Berkeley AI Hackathon
Recent research (Ibrahim et al., Nature 2024) shows targeted muscle stimulation can eliminate FOG. We used a wearable sensor (MPU-6050) to collect leg motion data. Using our model's prediction, a robotic arm delivers a localized “poke” to end the freezing episode.
Built with Shuyu van Kerkwijk, Harry Cheng, and Isabel Lew. Thank you to the @CalHacks organizers for this opportunity!
