San Francisco, CA
Joined January 2026
Harbor Framework retweeted
Now, run agent evals with @harborframework + @Docker Sandboxes.
Agents that can edit files and run commands can also game their own evals. Harbor + Docker Sandboxes fix that: every eval task runs in an isolated microVM, with network policies for fair, reproducible runs. bit.ly/4B8xf9Z
1
2
231
Harbor Framework retweeted
Terminal-Bench meetup tonight *with* swag. Register if you haven’t already! Benchmarks, RL environments, new features, and state of the union luma.com/tbench
17
10
3
116
12,069
Harbor Framework retweeted
The gold standard for science is sharing reproducible and auditable experimental data. When it comes to benchmarking agents, this means configs, trajectories, rewards, and logs for every single trial. We built harbor to make this easy. Unreal's analysis (related to @melissapan's recent work on comparing cost between harnesses while keeping the model fixed) sets a good example: they have uploaded and attached the raw harbor jobs for every result (each can be reproduced with a single harbor command). Good work!
Introducing Unreal Agent: An open-source harness with state-of-the-art cost efficiency 39% cheaper than Codex+Astra on Terminal-Bench 4.0 while maintaining performance
7
4
2
80
9,413
Harbor Framework retweeted
Half the set is public with every judge's full logs, a scorer for your own judge, and the audit trail. 35 items are held out. Every case is a runnable @harborframework task, so you can point your own judge at the set and score it with one command. Full writeup: heuristic.build/blog/arbiter…
1
8
364
Harbor Framework retweeted
Replying to @vercel
@vercel’s fx.sh harness is now available as an agent judge in Harbor Rewardkit. It makes judge setup time 7× faster than Codex and 16× faster than Claude Code. @harborframework
1
2
7
556
Harbor Framework retweeted
We love @harborframework's new observability feature!
Got streaming working with @tensorlake sandboxes on Harbor too. So you can see the agent's trajectory and files while it runs. PR is on the way.
1
12
1,773
Harbor Framework retweeted
Add skills to your agent, swap the model, tweak a tool description. How do you know you didn't break what already worked? Managed Deep Agents ships with @harborframework. Evals run in a fresh container every time, tracked in LangSmith. What you need to know in 9 minutes:
8
7
1
45
9,510
Harbor Framework retweeted
Replying to @alexgshaw
Registered - we have several ambitious new evals we’re kicking off in collaboration w/ some exciting names, and are deeply evaluating @harborframework as the foundation. Would be great to meet the team!
1
4
170
Harbor Framework retweeted
We're hosting a meetup! Come meet the community behind the benchmark, celebrate recent releases, and join our discussions about the future of agent evaluation. luma.com/tbench
3
5
3
55
21,173
Harbor Framework retweeted
Harbor Adapters and Harbor-Index arxiv.org/pdf/2609.04298 1+year work, 120+contributors, 300+PRs, 10+ funding partners.
1
41
14
182
31,194
Harbor Framework retweeted
Terminal-Bench-Science 0.1 is the #1 featured benchmark on @AnthropicAI’s new Claude Fable release 🚀 terminal-bench-science.ai
Claude
@claudeaiSep 1
Replying to @claudeai
Across our benchmarks, the model sets a new standard. It scores 52.6% on Terminal-Bench-Science 0.1, more than double Fable 5. On Terminal-Bench 4.0, it scores 55.8% against 42.0% for Fable 5.
13
20
135
9,009
Harbor Framework retweeted
We use @harborframework internally and have worked with @kobe0938 and the rest of the team to add support for simulated-user interactive benchmarks. This allows the community to more realistically model long sessions and cache expiration when evaluating coding agent and routing economics: github.com/harbor-framework/…
2
4
1
22
3,551
Harbor Framework retweeted
Tensorlake
@tensorlakeSep 1
This is what powers the dynamic network policy support we added to @harborframework , allowing agents to switch network policies without restarting the running environment.
We shipped a comprehensive update to Tensorlake's egress traffic and firewall infrastructure for sandboxes. Users can now update almost all aspects of traffic control while a sandbox is running. This lets a sandbox start up, download or configure tools, and then lock down before agents begin working in it. Support for DNS-based policies improved as well: 1. Allow and deny based on DNS names (including wildcards). 2. Evaluate DNS-based flows per connection, so destination IP churn doesn't let traffic slip to denied endpoints.
2
1
455
Harbor Framework retweeted
this is truly exceptional work. i had the chance to talk to Steven a bit about how he went about designing and building TB-Science, and i came away really impressed making benchmarks is easy. making an outstanding one is unbelievably hard. the incentives are stacked against it, from diluting task quality to ship fast, to building hill-climbable problem shapes you can beat by buying envs on the market see, many folks think of benchmarks as marketing tools, and in a way they are! but the best ones are way more. to me they're basically the AI version of GDP. heavily gamed and debated, yet still indispensable. a good public eval doesn't need to be ungameable. it needs to be hard enough to game that gaming it looks a lot like actually getting better steven and his team built something special here. measuring the frontier of science and how fast (or how slow!) models are pushing it further is arguably one of the most important things in the world today. doing it with relentless rigor makes it a public good can't wait to see TBScience v02 come out, and if you're a scientist, you should consider contributing!
We're releasing Terminal-Bench-Science: a benchmark for evaluating AI agents on research workflows across scientific domains. An ongoing Stanford-led community effort, built by the team behind Terminal-Bench together with scientific domain experts at research institutions worldwide. v0.1 has 70 tasks. Claude Opus 5 solves only ~30%. 1/n 👇
1
4
42
3,837
Harbor Framework retweeted
We've released our initial tasks and environments in @harborframework format at github.com/scaleapi/rsi-benc… Huge thanks to @nas_mahmoud_, @ChenguangWang, and @_yunzhong, as well as @MingchenZhuge and @tydsh (who contributed in their personal time), for their collaboration and advice. We're also grateful to @modal for their compute support. @ScaleAILabs 🚀
Launching rsi-benchmark.com: The work of AI R&D has always belonged to humans. For the first time, though, it no longer seems certain that it always will. Recursive self-improvement is within a line of sight. It may still be far, but it is close enough that we should start measuring it.
9
13
4
124
18,710
Harbor Framework retweeted
Harbor now supports simulated users: user agent role-plays a human and drives the agent through a multi-turn conversation.
7
9
5
134
20,074
Harbor Framework retweeted
Introducing harbor regrade: re-run a modified grader without paying for another agent session
2
3
14
1,098
Harbor Framework retweeted
Made a pr to @harborframework for @antigravity support so now we can benchmark it too!
5
1
23
1,844
Harbor Framework retweeted
Florian Brand
@xeophonJul 15
> thousands or millions of benchmarks
1
1
9
800
Harbor Framework retweeted
harbor is excellent software! h/t to y'all
4
5
1,534