Julian George retweeted
Replying to @MTSlive
You can perturb a tiny number of weights. Is this an American model now? At what point does SWE stop being Kimi?
Julian George retweeted
Thanksgiving advice from YC: make something people want.
Nobody wants turkey.
What if the failure you're trying to fix isn't a failure at all?
Faulty evals waste engineering time and can teach agents the wrong behavior. @withbeaker checks the eval while improving your agent.
On AutomationBench: ~14 percentage points of training-score improvement per run, plus 8 evaluation concerns flagged across two runs.
On LiveNewsBench: 20 questions flagged, including 6 confirmed grader errors.
Before you fix the agent, check the eval 👇
withbeaker.ai/blog/before-yo…
Julian George retweeted
Granola CEO opening X this morning to find his whole product shipped as a FEATURE inside Claude
Julian George retweeted
AI engineers, your production agent isn't slow because of the model. Too often, it's slow because of everything you are doing around it. Wasted work such as repeated doc lookups, unnecessary bloated tool outputs, model calls for every tiny step, etc.
Beaker (@withbeaker) autonomously proposes changes to your agent, experiment with each one on your use-case, and keeps only what works.
It made an App World agent 37% faster and 35% cheaper at no cost in performance.
Have you tried speeding up your prod agent? Beaker will give you an answer in a few hours: withbeaker.ai
Blogpost: withbeaker.ai/blog/speeding-…
This is a really cool case study by @_reza_nayebi . This agentic weight-space stuff has gone viral, and its hard to tell if the hype is legit, but this post shows what's actually going on behind the scenes.
Last week, @rikiyatakehi fixed a looping model by editing its weights. We pointed an AI agent at the same problem. It never ran the model and autonomously found a different fix.
No training run, nor gradients. That's weight-space programming.
We gave the same task to Beaker, @withbeaker, our autonomous agent-improver. It analyzes how an agent behaves and fails, formulates hypotheses on what and how to fix, validates or invalidates them, in a loop, to hill-climb a pre-defined score. The target: Qwen3.5 0.8B, which fails 93% of MATH-500, often by getting stuck in loops until it runs out of thinking tokens.
The model has a tendency to generate "Wait", and Beaker turned it down. From the weights and the scores of its own edits, it figured out what was going wrong and autonomously fixed it by editing 1,024 numbers, a millionth of the model's parameters. MATH-500: 7.4% -> 37.4%.
That's a different fix from the original blogpost. They found a way out of generation loops, and Beaker found a way to not fall into them. Theirs is stronger alone (51.1% in our harness), but the two stacked together compound: 58.7%.
When properly harnessed, less capable models are good weight space programmers. Beaker powered by GPT6.1 Sol found the same neurons as Beaker with GPT6 Astra, at $4-16 per run vs $28-79.
We also pointed Beaker at new goals. We told it to tune the weights such that the agent comes up with shorter answers, and it did, with answers a third to half shorter.
Beaker already tunes prompts, skills, memory, tool implementation, orchestration, etc. to improve agent quality, latency cost and determinism. Weights are the next knob it has access to, coming soon. Sign up at withbeaker.ai
Blog: withbeaker.ai/blog/weight-sp…
Tuned-weights on @huggingface: huggingface.co/collections/r…
Public runs: app.withbeaker.ai/public/run…
Julian George retweeted
Last week, @rikiyatakehi fixed a looping model by editing its weights. We pointed an AI agent at the same problem. It never ran the model and autonomously found a different fix.
No training run, nor gradients. That's weight-space programming.
We gave the same task to Beaker, @withbeaker, our autonomous agent-improver. It analyzes how an agent behaves and fails, formulates hypotheses on what and how to fix, validates or invalidates them, in a loop, to hill-climb a pre-defined score. The target: Qwen3.5 0.8B, which fails 93% of MATH-500, often by getting stuck in loops until it runs out of thinking tokens.
The model has a tendency to generate "Wait", and Beaker turned it down. From the weights and the scores of its own edits, it figured out what was going wrong and autonomously fixed it by editing 1,024 numbers, a millionth of the model's parameters. MATH-500: 7.4% -> 37.4%.
That's a different fix from the original blogpost. They found a way out of generation loops, and Beaker found a way to not fall into them. Theirs is stronger alone (51.1% in our harness), but the two stacked together compound: 58.7%.
When properly harnessed, less capable models are good weight space programmers. Beaker powered by GPT6.1 Sol found the same neurons as Beaker with GPT6 Astra, at $4-16 per run vs $28-79.
We also pointed Beaker at new goals. We told it to tune the weights such that the agent comes up with shorter answers, and it did, with answers a third to half shorter.
Beaker already tunes prompts, skills, memory, tool implementation, orchestration, etc. to improve agent quality, latency cost and determinism. Weights are the next knob it has access to, coming soon. Sign up at withbeaker.ai
Blog: withbeaker.ai/blog/weight-sp…
Tuned-weights on @huggingface: huggingface.co/collections/r…
Public runs: app.withbeaker.ai/public/run…
Julian George retweeted
many ai engineers building vertical agents think too much like researchers and not enough like engineers.
researchers fear overfitting because generalization is the goal: doing well on inputs no one has seen yet.
users don’t care about inputs they’ll never send. they care about the ones they send every day, and those are narrow and repetitive.
the engineer’s job is to make every input that matters in-distribution. overfit to what users actually send: the patterns, not the individual samples. when a new input breaks and it matters, pull it in fast. accuracy on real inputs is the priority. inputs that might never come are a bonus.
overfitting to users is the job.
Julian George retweeted
In this case study, Julian (@JulianGeorgeDev) explains how he used @withbeaker to add a confidence agent to a multi-agent system for supply chain operations.
Starting from a bare-bones agent, Beaker autonomously optimized it to catch every real error in its final rollouts, and it was in production in under a day.
👇
withbeaker.ai/blog/business-…
Julian George retweeted
Similar performance at an order of magnitude cheaper
Switching from closed to open models? Most teams leave capability on the table: they swap the model and keep the harness. Tune the harness for the new model, or your evals judge it at its worst.
Julian George retweeted
Stop waiting for the next model
I often hear ai engineers at vertical ai app cos say they need the next model releases to unlock the next step-change in their product capability.
What if your current model was already good enough? We tuned an agent's harness around a model 9 months older than the latest one. It beats the latest model by almost 10 points.









