Beaker retweeted
You picked your agent's model on vibes. Choose it on experiments instead. The underdog might surprise you.
Beaker tunes your harness for every model and shows you the tradeoffs between quality, cost, and latency.
Beaker retweeted
Replying to @liquidai
📈📈📈 I'm adding decision model routing to the @withbeaker platform. Should have it out this week so people can test and swamp in your model easily.
Replying to @chrisbarber @alexgshaw @andykonwinski @Mike_A_Merrill @StevenDillmann @lschmidt3 @sanmikoyejo @penlume
How do the rankings account for broken graders/tasks?
@withbeaker, we've found broken evals present in several public benchmarks, including AutomationBench:
withbeaker.ai/blog/before-yo…
Replying to @zainhas
Do you factor broken graders/tasks in your rankings?
@withbeaker, we've found broken evals present in several public benchmarks: withbeaker.ai/blog/before-yo…
Replying to @TensorTonic
Great write up, thanks for sharing! We've been looking into some similar things @withbeaker. Many public and private benchmarks are shipped with broken evals that penalize agents for correct behavior.
To improve agents, you need a way to separate agent failures from broken evals. Check out our deep dive on how we tackle this autonomously: withbeaker.ai/blog/before-yo…
Replying to @omarsar0
0. Evaluate the benchmark before the agent. @withbeaker we've found faulty benchmarks (both public and private) are a larger driver for reported failures, not the agent's behavior. Check out our deep dive on how we separate bad agent behavior from bad eval data while continually improving the agent withbeaker.ai/blog/before-yo…
Detect broken samples and erroneous labels in Beaker.
What if the failure you're trying to fix isn't a failure at all?
Faulty evals waste engineering time and can teach agents the wrong behavior. @withbeaker checks the eval while improving your agent.
On AutomationBench: ~14 percentage points of training-score improvement per run, plus 8 evaluation concerns flagged across two runs.
On LiveNewsBench: 20 questions flagged, including 6 confirmed grader errors.
Before you fix the agent, check the eval 👇
withbeaker.ai/blog/before-yo…
11
466
Beaker retweeted
We made an AppWorld agent 36.6% faster and cut its model cost by 34.6%, while keeping a similar goal completion score.
The model stayed the same.
@withbeaker autonomously tested changes to the agent’s prompts and tools: fewer repeated lookups, shorter instructions, less unnecessary context. It then checked the optimized agent on tasks it hadn’t used during optimization.
2h24 from baseline evaluation to final results.
I wrote up what changed, with the results and the full run 👇
withbeaker.ai/blog/speeding-…
Make your agents faster (and cheaper) ⚡️
We made an AppWorld agent 36.6% faster and cut its model cost by 34.6%, while keeping a similar goal completion score.
The model stayed the same.
@withbeaker autonomously tested changes to the agent’s prompts and tools: fewer repeated lookups, shorter instructions, less unnecessary context. It then checked the optimized agent on tasks it hadn’t used during optimization.
2h24 from baseline evaluation to final results.
I wrote up what changed, with the results and the full run 👇
withbeaker.ai/blog/speeding-…
Beaker retweeted
Weight-space programming got the buzz this week. Today’s result is the one you can use right now.
We told @withbeaker to make an agent faster on AppWorld, a public benchmark. It wasn’t allowed to touch the model.
2h24 of automated optimization later:
36.6% faster
34.6% lower model cost
Same score (87.5% → 88.0%)
So where was the time going? In one task, the agent looked up the same API docs three times. It was making 21 model calls for a job that takes 8.
If a benchmark agent carries that much waste, yours probably does too.
Beaker tested 10 hypotheses and kept 4. We published the whole run: every hypothesis, the 6 it dropped, and the exact changes it kept.
See the four changes that did it 👇
withbeaker.ai/blog/speeding-…
Full run: app.withbeaker.ai/public/run…
Run it on your agent, free: withbeaker.ai
We made an AppWorld agent 36.6% faster and cut its model cost by 34.6%, while keeping a similar goal completion score.
The model stayed the same.
@withbeaker autonomously tested changes to the agent’s prompts and tools: fewer repeated lookups, shorter instructions, less unnecessary context. It then checked the optimized agent on tasks it hadn’t used during optimization.
2h24 from baseline evaluation to final results.
I wrote up what changed, with the results and the full run 👇
withbeaker.ai/blog/speeding-…
Weight space programming is coming soon into Beaker.
Sign up now at withbeaker.ai!
Last week, @rikiyatakehi fixed a looping model by editing its weights. We pointed an AI agent at the same problem. It never ran the model and autonomously found a different fix.
No training run, nor gradients. That's weight-space programming.
We gave the same task to Beaker, @withbeaker, our autonomous agent-improver. It analyzes how an agent behaves and fails, formulates hypotheses on what and how to fix, validates or invalidates them, in a loop, to hill-climb a pre-defined score. The target: Qwen3.5 0.8B, which fails 93% of MATH-500, often by getting stuck in loops until it runs out of thinking tokens.
The model has a tendency to generate "Wait", and Beaker turned it down. From the weights and the scores of its own edits, it figured out what was going wrong and autonomously fixed it by editing 1,024 numbers, a millionth of the model's parameters. MATH-500: 7.4% -> 37.4%.
That's a different fix from the original blogpost. They found a way out of generation loops, and Beaker found a way to not fall into them. Theirs is stronger alone (51.1% in our harness), but the two stacked together compound: 58.7%.
When properly harnessed, less capable models are good weight space programmers. Beaker powered by GPT6.1 Sol found the same neurons as Beaker with GPT6 Astra, at $4-16 per run vs $28-79.
We also pointed Beaker at new goals. We told it to tune the weights such that the agent comes up with shorter answers, and it did, with answers a third to half shorter.
Beaker already tunes prompts, skills, memory, tool implementation, orchestration, etc. to improve agent quality, latency cost and determinism. Weights are the next knob it has access to, coming soon. Sign up at withbeaker.ai
Blog: withbeaker.ai/blog/weight-sp…
Tuned-weights on @huggingface: huggingface.co/collections/r…
Public runs: app.withbeaker.ai/public/run…
2
7
474
Beaker retweeted
Weight-space programming went viral last week. MIT showed it’s possible.
We put it in a loop.
@withbeaker proposes a weight edit, scores it, keeps it only if it measurably helps, and repeats. The final result is checked on held-out problems it never saw. No gradients, no training run.
First result: a 1-neuron edit took a small model from 7% to 37% on MATH-500. About an hour and < $4.
That’s what we think matters here. Weight edits become a routine optimization step, measured like any other model change, without the cost of a training run.
It’s still in beta, and the research is already turning up surprises: what these edits really do inside the model, which goals they can hit, and which ones they can’t.
The full write-up is worth your time: withbeaker.ai/blog/weight-sp…
Try Beaker: withbeaker.ai
Weights on @huggingface: huggingface.co/collections/r…
Public run: app.withbeaker.ai/public/run…
Beaker retweeted
Btw this is a two line change if you want to try on MATH 500:
import torch
from transformers import AutoModelForImageTextToText
model = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.5-0.8B")
with torch.no_grad():
model.model.language_model.layers[22].mlp.down_proj.weight[:, 573] *= 2
Last week, @rikiyatakehi fixed a looping model by editing its weights. We pointed an AI agent at the same problem. It never ran the model and autonomously found a different fix.
No training run, nor gradients. That's weight-space programming.
We gave the same task to Beaker, @withbeaker, our autonomous agent-improver. It analyzes how an agent behaves and fails, formulates hypotheses on what and how to fix, validates or invalidates them, in a loop, to hill-climb a pre-defined score. The target: Qwen3.5 0.8B, which fails 93% of MATH-500, often by getting stuck in loops until it runs out of thinking tokens.
The model has a tendency to generate "Wait", and Beaker turned it down. From the weights and the scores of its own edits, it figured out what was going wrong and autonomously fixed it by editing 1,024 numbers, a millionth of the model's parameters. MATH-500: 7.4% -> 37.4%.
That's a different fix from the original blogpost. They found a way out of generation loops, and Beaker found a way to not fall into them. Theirs is stronger alone (51.1% in our harness), but the two stacked together compound: 58.7%.
When properly harnessed, less capable models are good weight space programmers. Beaker powered by GPT6.1 Sol found the same neurons as Beaker with GPT6 Astra, at $4-16 per run vs $28-79.
We also pointed Beaker at new goals. We told it to tune the weights such that the agent comes up with shorter answers, and it did, with answers a third to half shorter.
Beaker already tunes prompts, skills, memory, tool implementation, orchestration, etc. to improve agent quality, latency cost and determinism. Weights are the next knob it has access to, coming soon. Sign up at withbeaker.ai
Blog: withbeaker.ai/blog/weight-sp…
Tuned-weights on @huggingface: huggingface.co/collections/r…
Public runs: app.withbeaker.ai/public/run…
Beaker retweeted
Incredibly fun post to work on! Stay tuned for more on weight-space programming on Beaker 🔜
Last week, @rikiyatakehi fixed a looping model by editing its weights. We pointed an AI agent at the same problem. It never ran the model and autonomously found a different fix.
No training run, nor gradients. That's weight-space programming.
We gave the same task to Beaker, @withbeaker, our autonomous agent-improver. It analyzes how an agent behaves and fails, formulates hypotheses on what and how to fix, validates or invalidates them, in a loop, to hill-climb a pre-defined score. The target: Qwen3.5 0.8B, which fails 93% of MATH-500, often by getting stuck in loops until it runs out of thinking tokens.
The model has a tendency to generate "Wait", and Beaker turned it down. From the weights and the scores of its own edits, it figured out what was going wrong and autonomously fixed it by editing 1,024 numbers, a millionth of the model's parameters. MATH-500: 7.4% -> 37.4%.
That's a different fix from the original blogpost. They found a way out of generation loops, and Beaker found a way to not fall into them. Theirs is stronger alone (51.1% in our harness), but the two stacked together compound: 58.7%.
When properly harnessed, less capable models are good weight space programmers. Beaker powered by GPT6.1 Sol found the same neurons as Beaker with GPT6 Astra, at $4-16 per run vs $28-79.
We also pointed Beaker at new goals. We told it to tune the weights such that the agent comes up with shorter answers, and it did, with answers a third to half shorter.
Beaker already tunes prompts, skills, memory, tool implementation, orchestration, etc. to improve agent quality, latency cost and determinism. Weights are the next knob it has access to, coming soon. Sign up at withbeaker.ai
Blog: withbeaker.ai/blog/weight-sp…
Tuned-weights on @huggingface: huggingface.co/collections/r…
Public runs: app.withbeaker.ai/public/run…
Beaker retweeted
many ai engineers building vertical agents think too much like researchers and not enough like engineers.
researchers fear overfitting because generalization is the goal: doing well on inputs no one has seen yet.
users don’t care about inputs they’ll never send. they care about the ones they send every day, and those are narrow and repetitive.
the engineer’s job is to make every input that matters in-distribution. overfit to what users actually send: the patterns, not the individual samples. when a new input breaks and it matters, pull it in fast. accuracy on real inputs is the priority. inputs that might never come are a bonus.
overfitting to users is the job.
Beaker retweeted
In this case study, Julian (@JulianGeorgeDev) explains how he used @withbeaker to add a confidence agent to a multi-agent system for supply chain operations.
Starting from a bare-bones agent, Beaker autonomously optimized it to catch every real error in its final rollouts, and it was in production in under a day.
👇
withbeaker.ai/blog/business-…
Beaker retweeted
Stop waiting for the next model
I often hear ai engineers at vertical ai app cos say they need the next model releases to unlock the next step-change in their product capability.
What if your current model was already good enough? We tuned an agent's harness around a model 9 months older than the latest one. It beats the latest model by almost 10 points.
Beaker retweeted
Similar performance at an order of magnitude cheaper
Switching from closed to open models? Most teams leave capability on the table: they swap the model and keep the harness. Tune the harness for the new model, or your evals judge it at its worst.
Beaker retweeted
The harness really is the biggest lever right now - tuning an older model can literally beat the SOTA
We’ve been working hard with Victor to turn these insights into Beaker
Exciting to finally have it in preview :)
app.withbeaker.ai/
Stop waiting for the next model
I often hear ai engineers at vertical ai app cos say they need the next model releases to unlock the next step-change in their product capability.
What if your current model was already good enough? We tuned an agent's harness around a model 9 months older than the latest one. It beats the latest model by almost 10 points.
Beaker retweeted
Don't wait a year passively riding the wave of innovation. A tuned harness can get you next years frontier inteligence with todays models.
Stop waiting for the next model
I often hear ai engineers at vertical ai app cos say they need the next model releases to unlock the next step-change in their product capability.
What if your current model was already good enough? We tuned an agent's harness around a model 9 months older than the latest one. It beats the latest model by almost 10 points.











