Beaker (@withbeaker) header
Beaker (@withbeaker) profile picture

Continuous improvement for production agents

Joined July 2026
You picked your agent's model on vibes. Choose it on experiments instead. The underdog might surprise you. Beaker tunes your harness for every model and shows you the tradeoffs between quality, cost, and latency.
3
4
13
1,044
Replying to @liquidai
📈📈📈 I'm adding decision model routing to the @withbeaker platform. Should have it out this week so people can test and swamp in your model easily.
1
1
3
66
Beaker retweeted
Replying to @TensorTonic
Great write up, thanks for sharing! We've been looking into some similar things @withbeaker. Many public and private benchmarks are shipped with broken evals that penalize agents for correct behavior. To improve agents, you need a way to separate agent failures from broken evals. Check out our deep dive on how we tackle this autonomously: withbeaker.ai/blog/before-yo…
1
6
51
Beaker retweeted
Replying to @omarsar0
0. Evaluate the benchmark before the agent. @withbeaker we've found faulty benchmarks (both public and private) are a larger driver for reported failures, not the agent's behavior. Check out our deep dive on how we separate bad agent behavior from bad eval data while continually improving the agent withbeaker.ai/blog/before-yo…
1
4
58
Beaker retweeted
We made an AppWorld agent 36.6% faster and cut its model cost by 34.6%, while keeping a similar goal completion score. The model stayed the same. @withbeaker autonomously tested changes to the agent’s prompts and tools: fewer repeated lookups, shorter instructions, less unnecessary context. It then checked the optimized agent on tasks it hadn’t used during optimization. 2h24 from baseline evaluation to final results. I wrote up what changed, with the results and the full run 👇 withbeaker.ai/blog/speeding-…
3
2
2
12
1,254
Beaker retweeted
Make your agents faster (and cheaper) ⚡️
We made an AppWorld agent 36.6% faster and cut its model cost by 34.6%, while keeping a similar goal completion score. The model stayed the same. @withbeaker autonomously tested changes to the agent’s prompts and tools: fewer repeated lookups, shorter instructions, less unnecessary context. It then checked the optimized agent on tasks it hadn’t used during optimization. 2h24 from baseline evaluation to final results. I wrote up what changed, with the results and the full run 👇 withbeaker.ai/blog/speeding-…
2
8
377
Weight-space programming got the buzz this week. Today’s result is the one you can use right now. We told @withbeaker to make an agent faster on AppWorld, a public benchmark. It wasn’t allowed to touch the model. 2h24 of automated optimization later: 36.6% faster 34.6% lower model cost Same score (87.5% → 88.0%) So where was the time going? In one task, the agent looked up the same API docs three times. It was making 21 model calls for a job that takes 8. If a benchmark agent carries that much waste, yours probably does too. Beaker tested 10 hypotheses and kept 4. We published the whole run: every hypothesis, the 6 it dropped, and the exact changes it kept. See the four changes that did it 👇 withbeaker.ai/blog/speeding-… Full run: app.withbeaker.ai/public/run… Run it on your agent, free: withbeaker.ai
We made an AppWorld agent 36.6% faster and cut its model cost by 34.6%, while keeping a similar goal completion score. The model stayed the same. @withbeaker autonomously tested changes to the agent’s prompts and tools: fewer repeated lookups, shorter instructions, less unnecessary context. It then checked the optimized agent on tasks it hadn’t used during optimization. 2h24 from baseline evaluation to final results. I wrote up what changed, with the results and the full run 👇 withbeaker.ai/blog/speeding-…
1
4
1
8
572
Weight-space programming went viral last week. MIT showed it’s possible. We put it in a loop. @withbeaker proposes a weight edit, scores it, keeps it only if it measurably helps, and repeats. The final result is checked on held-out problems it never saw. No gradients, no training run. First result: a 1-neuron edit took a small model from 7% to 37% on MATH-500. About an hour and < $4. That’s what we think matters here. Weight edits become a routine optimization step, measured like any other model change, without the cost of a training run. It’s still in beta, and the research is already turning up surprises: what these edits really do inside the model, which goals they can hit, and which ones they can’t. The full write-up is worth your time: withbeaker.ai/blog/weight-sp… Try Beaker: withbeaker.ai Weights on @huggingface: huggingface.co/collections/r… Public run: app.withbeaker.ai/public/run…
15
31
6
276
12,367
Btw this is a two line change if you want to try on MATH 500: import torch from transformers import AutoModelForImageTextToText model = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.5-0.8B") with torch.no_grad(): model.model.language_model.layers[22].mlp.down_proj.weight[:, 573] *= 2
Last week, @rikiyatakehi fixed a looping model by editing its weights. We pointed an AI agent at the same problem. It never ran the model and autonomously found a different fix. No training run, nor gradients. That's weight-space programming. We gave the same task to Beaker, @withbeaker, our autonomous agent-improver. It analyzes how an agent behaves and fails, formulates hypotheses on what and how to fix, validates or invalidates them, in a loop, to hill-climb a pre-defined score. The target: Qwen3.5 0.8B, which fails 93% of MATH-500, often by getting stuck in loops until it runs out of thinking tokens. The model has a tendency to generate "Wait", and Beaker turned it down. From the weights and the scores of its own edits, it figured out what was going wrong and autonomously fixed it by editing 1,024 numbers, a millionth of the model's parameters. MATH-500: 7.4% -> 37.4%. That's a different fix from the original blogpost. They found a way out of generation loops, and Beaker found a way to not fall into them. Theirs is stronger alone (51.1% in our harness), but the two stacked together compound: 58.7%. When properly harnessed, less capable models are good weight space programmers. Beaker powered by GPT6.1 Sol found the same neurons as Beaker with GPT6 Astra, at $4-16 per run vs $28-79. We also pointed Beaker at new goals. We told it to tune the weights such that the agent comes up with shorter answers, and it did, with answers a third to half shorter. Beaker already tunes prompts, skills, memory, tool implementation, orchestration, etc. to improve agent quality, latency cost and determinism. Weights are the next knob it has access to, coming soon. Sign up at withbeaker.ai Blog: withbeaker.ai/blog/weight-sp… Tuned-weights on @huggingface: huggingface.co/collections/r… Public runs: app.withbeaker.ai/public/run…
2
7
489
Incredibly fun post to work on! Stay tuned for more on weight-space programming on Beaker 🔜
Last week, @rikiyatakehi fixed a looping model by editing its weights. We pointed an AI agent at the same problem. It never ran the model and autonomously found a different fix. No training run, nor gradients. That's weight-space programming. We gave the same task to Beaker, @withbeaker, our autonomous agent-improver. It analyzes how an agent behaves and fails, formulates hypotheses on what and how to fix, validates or invalidates them, in a loop, to hill-climb a pre-defined score. The target: Qwen3.5 0.8B, which fails 93% of MATH-500, often by getting stuck in loops until it runs out of thinking tokens. The model has a tendency to generate "Wait", and Beaker turned it down. From the weights and the scores of its own edits, it figured out what was going wrong and autonomously fixed it by editing 1,024 numbers, a millionth of the model's parameters. MATH-500: 7.4% -> 37.4%. That's a different fix from the original blogpost. They found a way out of generation loops, and Beaker found a way to not fall into them. Theirs is stronger alone (51.1% in our harness), but the two stacked together compound: 58.7%. When properly harnessed, less capable models are good weight space programmers. Beaker powered by GPT6.1 Sol found the same neurons as Beaker with GPT6 Astra, at $4-16 per run vs $28-79. We also pointed Beaker at new goals. We told it to tune the weights such that the agent comes up with shorter answers, and it did, with answers a third to half shorter. Beaker already tunes prompts, skills, memory, tool implementation, orchestration, etc. to improve agent quality, latency cost and determinism. Weights are the next knob it has access to, coming soon. Sign up at withbeaker.ai Blog: withbeaker.ai/blog/weight-sp… Tuned-weights on @huggingface: huggingface.co/collections/r… Public runs: app.withbeaker.ai/public/run…
2
6
350
many ai engineers building vertical agents think too much like researchers and not enough like engineers. researchers fear overfitting because generalization is the goal: doing well on inputs no one has seen yet. users don’t care about inputs they’ll never send. they care about the ones they send every day, and those are narrow and repetitive. the engineer’s job is to make every input that matters in-distribution. overfit to what users actually send: the patterns, not the individual samples. when a new input breaks and it matters, pull it in fast. accuracy on real inputs is the priority. inputs that might never come are a bonus. overfitting to users is the job.
4
5
15
1,209
In this case study, Julian (@JulianGeorgeDev) explains how he used @withbeaker to add a confidence agent to a multi-agent system for supply chain operations. Starting from a bare-bones agent, Beaker autonomously optimized it to catch every real error in its final rollouts, and it was in production in under a day. 👇 withbeaker.ai/blog/business-…
2
4
13
833
Stop waiting for the next model I often hear ai engineers at vertical ai app cos say they need the next model releases to unlock the next step-change in their product capability. What if your current model was already good enough? We tuned an agent's harness around a model 9 months older than the latest one. It beats the latest model by almost 10 points.
11
5
4
30
3,075
Similar performance at an order of magnitude cheaper Switching from closed to open models? Most teams leave capability on the table: they swap the model and keep the harness. Tune the harness for the new model, or your evals judge it at its worst.
10
2
1
16
1,822
Beaker retweeted
The harness really is the biggest lever right now - tuning an older model can literally beat the SOTA We’ve been working hard with Victor to turn these insights into Beaker Exciting to finally have it in preview :) app.withbeaker.ai/
Stop waiting for the next model I often hear ai engineers at vertical ai app cos say they need the next model releases to unlock the next step-change in their product capability. What if your current model was already good enough? We tuned an agent's harness around a model 9 months older than the latest one. It beats the latest model by almost 10 points.
2
6
430
Don't wait a year passively riding the wave of innovation. A tuned harness can get you next years frontier inteligence with todays models.
Stop waiting for the next model I often hear ai engineers at vertical ai app cos say they need the next model releases to unlock the next step-change in their product capability. What if your current model was already good enough? We tuned an agent's harness around a model 9 months older than the latest one. It beats the latest model by almost 10 points.
2
5
418
We have been running on a lot of experiments testing models base intelligence vs potential intelligence with an optimized harness. Nobody is talking about MiMo but across benchmarks it continues to compete and is emerging as a dark horse value king. @XiaomiMiMo 📈
2
2
1
6
697
Switching model versions is easy but if you're not updating your harness as well you're leaving performance on the table. Use Beaker to know what the Pareto frontier actually looks like for your use case.
Similar performance at an order of magnitude cheaper Switching from closed to open models? Most teams leave capability on the table: they swap the model and keep the harness. Tune the harness for the new model, or your evals judge it at its worst.
1
5
243