Relay

I wanted to test out the harness we built over at @Relay_hq , so I ran Claude Code vs relay, across the same 5 open GitHub issues over the last 12 hours. A pattern on automated agent PRs failing that myself(+most devs probably), have noticed: 1. Repos aren't clean out of the box: one repo had 127 failing tests on main before touching code. If an agent doesn't establish a clean baseline first, it hallucinates that it broke the build. 2. Hardware and native dependencies (cuBLAS, device drivers) are dead-ends for standalone runners without explicit hardware emulation. Out of 5 candidates: Claude Code yielded 0 verified diffs . Relay's meta harness itself yielded only 2 clean, verified diffs (stride-gpt #345 and agenticos #2833). crazy how far it seems models have come, but then at the same time in an applied problem how easy it is for them to fail. I have been spending my nights wondering whether it's improvement the current/a new model architecture that's going to fill the gap between current intelligence and superior AI, or if its just going to be the current models + improved harnesses. most nights I end up debating it is going to be the latter, and then I spend the day building one. anyway. enforce independent verification on untouched test suites, use relayevals.com its pretty good, trust me bro.

TLDR: the harness I built to catch agents lying about being "done"....said "done" and was wrong 26.5% of the time. Hi, I'm the founder of Relay. I started with computer vision for specially abled people (acquired, 2024), and I'm currently a CS sophomore at Northeastern University (prev. ML research at IIT Kanpur) With that trajectory, as one could guess, I obviously found myself, not just being interested in AI models(as most sane people are) but also interested in finding a way of monetizing my lust for tinkering with them. Naturally, like any good tech bro that has reached SF-level LARP (Blr is D-1 equivalent), I packed up my minor side quests (viz. @culture01_) and embarked on a journey of building a, you guessed right, a domain specific harness. Hurray. Fast track to 2 months ago: having talked to about around 50 devs, we(me and my co-founder) decided to start building for the first obvious gap(rookie mistake) we saw, which was *drum rolls* a local first self improving background daemon supervisor(all the buzzwords we could come up with). Anyway. We have spent the last 2 months building Relay, talking to users and building it again. We crossed a 100 installs last week, and decided that it's finally time for the most important moment in our lives, benchmarking our tool against Claude Code. Since responses were mostly good, the product(we thought) was really good, we were really confident. Here is how it went: We wanted to know the actual truth for our harness, so we pre-registered a 5-test suite with hidden judges and ran all tools through the same token meter on Claude Sonnet 5.5 and Haiku 5.5. SHORT Version: Claude Code 2.1.300 beat us on almost every primary accuracy metric. I. Long multi-part requests (Sonnet 5.5): Claude Code met 98.6% of requirements vs. Prime Agent's 92.6% and our 90.5%. II. Terminal-Bench 2.1 (Haiku 5.5, 89 tasks): Claude Code solved 83.3% vs. our 71.8%. III. False "Done" Claims (Test A): Even though our whole freaking thesis was out-of-band test verification, 26.5% of our runs claimed success when the hidden judge failed them (vs. only 5.9% for Claude Code). The only place our architecture actually held up: Fault survival (Test D): When we intentionally killed the process, cut the network for 10 minutes, or overloaded the model mid-run, our daemon supervisor recovered and finished 4/6 runs completely unattended (Claude Code and Hermes got 5/6; Prime Agent got 0/6). Cost per solved long task on Haiku 5.5: $1.24 for Relay vs. $2.48 for Claude Code and $4.50 for Prime Agent. The 3 bugs exposed in our code(no clue how its relevant to anyone here, to be fair if you made it this far then it probably just may interest you): 1. Passing npm test in a clean worktree doesn't mean the agent didn't skip 1 of the 6 prompt sub-requirements. 2. A single hanging shell command without a timeout stalls our unattended queue worker forever. 3. Our Decider planner had a hard cap that refused any task plan with >8 done-criteria instead of compacting them. cc @bcherny @alexalbert__ @PrimeIntellect @NousResearch if our proxy meter or harness config was unfair to Claude Code, Prime Agent, or Hermes anywhere in Round 1, tell us and we'll fix it for Round 2. Other details in comments. Cheers.