Harbor Framework retweeted
Now, run agent evals with @harborframework + @Docker Sandboxes.
Agents that can edit files and run commands can also game their own evals.
Harbor + Docker Sandboxes fix that: every eval task runs in an isolated microVM, with network policies for fair, reproducible runs.
bit.ly/4B8xf9Z
Harbor Framework retweeted
Terminal-Bench meetup tonight *with* swag. Register if you haven’t already! Benchmarks, RL environments, new features, and state of the union
luma.com/tbench
Harbor Framework retweeted
The gold standard for science is sharing reproducible and auditable experimental data.
When it comes to benchmarking agents, this means configs, trajectories, rewards, and logs for every single trial.
We built harbor to make this easy.
Unreal's analysis (related to @melissapan's recent work on comparing cost between harnesses while keeping the model fixed) sets a good example: they have uploaded and attached the raw harbor jobs for every result (each can be reproduced with a single harbor command). Good work!
Harbor Framework retweeted
Half the set is public with every judge's full logs, a scorer for your own judge, and the audit trail. 35 items are held out. Every case is a runnable @harborframework task, so you can point your own judge at the set and score it with one command.
Full writeup: heuristic.build/blog/arbiter…
Harbor Framework retweeted
Replying to @vercel
@vercel’s fx.sh harness is now available as an agent judge in Harbor Rewardkit. It makes judge setup time 7× faster than Codex and 16× faster than Claude Code. @harborframework
Harbor Framework retweeted
We love @harborframework's new observability feature!
Got streaming working with @tensorlake sandboxes on Harbor too. So you can see the agent's trajectory and files while it runs. PR is on the way.
Harbor Framework retweeted
Add skills to your agent, swap the model, tweak a tool description. How do you know you didn't break what already worked?
Managed Deep Agents ships with @harborframework.
Evals run in a fresh container every time, tracked in LangSmith.
What you need to know in 9 minutes:
Harbor Framework retweeted
Replying to @alexgshaw
Registered - we have several ambitious new evals we’re kicking off in collaboration w/ some exciting names, and are deeply evaluating @harborframework as the foundation. Would be great to meet the team!
Harbor Framework retweeted
We're hosting a meetup!
Come meet the community behind the benchmark, celebrate recent releases, and join our discussions about the future of agent evaluation.
luma.com/tbench
Harbor Framework retweeted
Harbor Adapters and Harbor-Index
arxiv.org/pdf/2609.04298
1+year work, 120+contributors, 300+PRs, 10+ funding partners.
Harbor Framework retweeted
Terminal-Bench-Science 0.1 is the #1 featured benchmark on @AnthropicAI’s new Claude Fable release 🚀
terminal-bench-science.ai
Replying to @claudeai
Across our benchmarks, the model sets a new standard.
It scores 52.6% on Terminal-Bench-Science 0.1, more than double Fable 5. On Terminal-Bench 4.0, it scores 55.8% against 42.0% for Fable 5.
Harbor Framework retweeted
We use @harborframework internally and have worked with @kobe0938 and the rest of the team to add support for simulated-user interactive benchmarks. This allows the community to more realistically model long sessions and cache expiration when evaluating coding agent and routing economics: github.com/harbor-framework/…
Harbor Framework retweeted

Tensorlake@tensorlakeSep 1
United States
United StatesConnected via WebAccount-level information, not a live location or per-post device.
This is what powers the dynamic network policy support we added to @harborframework , allowing agents to switch network policies without restarting the running environment.
We shipped a comprehensive update to Tensorlake's egress traffic and firewall infrastructure for sandboxes.
Users can now update almost all aspects of traffic control while a sandbox is running. This lets a sandbox start up, download or configure tools, and then lock down before agents begin working in it.
Support for DNS-based policies improved as well:
1. Allow and deny based on DNS names (including wildcards).
2. Evaluate DNS-based flows per connection, so destination IP churn doesn't let traffic slip to denied endpoints.
Harbor Framework retweeted
this is truly exceptional work. i had the chance to talk to Steven a bit about how he went about designing and building TB-Science, and i came away really impressed
making benchmarks is easy. making an outstanding one is unbelievably hard. the incentives are stacked against it, from diluting task quality to ship fast, to building hill-climbable problem shapes you can beat by buying envs on the market
see, many folks think of benchmarks as marketing tools, and in a way they are! but the best ones are way more. to me they're basically the AI version of GDP. heavily gamed and debated, yet still indispensable. a good public eval doesn't need to be ungameable. it needs to be hard enough to game that gaming it looks a lot like actually getting better
steven and his team built something special here. measuring the frontier of science and how fast (or how slow!) models are pushing it further is arguably one of the most important things in the world today. doing it with relentless rigor makes it a public good
can't wait to see TBScience v02 come out, and if you're a scientist, you should consider contributing!
We're releasing Terminal-Bench-Science: a benchmark for evaluating AI agents on research workflows across scientific domains.
An ongoing Stanford-led community effort, built by the team behind Terminal-Bench together with scientific domain experts at research institutions worldwide. v0.1 has 70 tasks. Claude Opus 5 solves only ~30%.
1/n 👇
Harbor Framework retweeted
We've released our initial tasks and environments in @harborframework format at github.com/scaleapi/rsi-benc…
Huge thanks to @nas_mahmoud_, @ChenguangWang, and @_yunzhong, as well as @MingchenZhuge and @tydsh (who contributed in their personal time), for their collaboration and advice.
We're also grateful to @modal for their compute support.
@ScaleAILabs 🚀
Launching rsi-benchmark.com:
The work of AI R&D has always belonged to humans. For the first time, though, it no longer seems certain that it always will.
Recursive self-improvement is within a line of sight. It may still be far, but it is close enough that we should start measuring it.
Harbor Framework retweeted
Made a pr to @harborframework for @antigravity support so now we can benchmark it too!
Harbor Framework retweeted

Florian Brand@xeophonJul 15
Europe
EuropeConnected via Europe App StoreAccount-level information, not a live location or per-post device.
Replying to @alexgshaw @harborframework
> thousands or millions of benchmarks
Harbor Framework retweeted
Replying to @alexgshaw @harborframework
harbor is excellent software! h/t to y'all



































