
ai research @withbeaker 👩🔬 prev: codegen @Bloomberg / 🤖🚗 @StanfordAILab @SISLaboratory Lyft L5 (@Woven_Toyota) / ML @GoogleAI @Meta
NYC
Joined September 2021
- Tweets52
- Following489
- Followers135
- Likes563
Ella retweeted
Nobody enjoys staring at data. Yet, it's still one of the biggest levers in machine learning.
Introducing data review in Beaker.
While it improves your agent, Beaker (@withbeaker) analyzes your dataset and flags samples that might be broken: correct answers marked wrong, tasks that contradict their checks, insufficiently specified rubrics, etc.
Try it on your agents and datasets: withbeaker.ai/blog/before-yo…
What if the failure you're trying to fix isn't a failure at all?
Faulty evals waste engineering time and can teach agents the wrong behavior. @withbeaker checks the eval while improving your agent.
On AutomationBench: ~14 percentage points of training-score improvement per run, plus 8 evaluation concerns flagged across two runs.
On LiveNewsBench: 20 questions flagged, including 6 confirmed grader errors.
Before you fix the agent, check the eval 👇
withbeaker.ai/blog/before-yo…
Make your agents faster (and cheaper) ⚡️
We made an AppWorld agent 36.6% faster and cut its model cost by 34.6%, while keeping a similar goal completion score.
The model stayed the same.
@withbeaker autonomously tested changes to the agent’s prompts and tools: fewer repeated lookups, shorter instructions, less unnecessary context. It then checked the optimized agent on tasks it hadn’t used during optimization.
2h24 from baseline evaluation to final results.
I wrote up what changed, with the results and the full run 👇
withbeaker.ai/blog/speeding-…
Ella retweeted
Last week, @rikiyatakehi fixed a looping model by editing its weights. We pointed an AI agent at the same problem. It never ran the model and autonomously found a different fix.
No training run, nor gradients. That's weight-space programming.
We gave the same task to Beaker, @withbeaker, our autonomous agent-improver. It analyzes how an agent behaves and fails, formulates hypotheses on what and how to fix, validates or invalidates them, in a loop, to hill-climb a pre-defined score. The target: Qwen3.5 0.8B, which fails 93% of MATH-500, often by getting stuck in loops until it runs out of thinking tokens.
The model has a tendency to generate "Wait", and Beaker turned it down. From the weights and the scores of its own edits, it figured out what was going wrong and autonomously fixed it by editing 1,024 numbers, a millionth of the model's parameters. MATH-500: 7.4% -> 37.4%.
That's a different fix from the original blogpost. They found a way out of generation loops, and Beaker found a way to not fall into them. Theirs is stronger alone (51.1% in our harness), but the two stacked together compound: 58.7%.
When properly harnessed, less capable models are good weight space programmers. Beaker powered by GPT6.1 Sol found the same neurons as Beaker with GPT6 Astra, at $4-16 per run vs $28-79.
We also pointed Beaker at new goals. We told it to tune the weights such that the agent comes up with shorter answers, and it did, with answers a third to half shorter.
Beaker already tunes prompts, skills, memory, tool implementation, orchestration, etc. to improve agent quality, latency cost and determinism. Weights are the next knob it has access to, coming soon. Sign up at withbeaker.ai
Blog: withbeaker.ai/blog/weight-sp…
Tuned-weights on @huggingface: huggingface.co/collections/r…
Public runs: app.withbeaker.ai/public/run…
Production agents are leaving performance gains on the table.
Our autonomous AI engineer, Beaker, took GPT-5.2 from 30% → 48% on AutomationBench without changing the model, beating OpenAI's newest model (GPT-6 Luna, untuned) at 39%.
Don't wait for the next model 👇
withbeaker.ai/blog/dont-wait…
Stop waiting for the next model
I often hear ai engineers at vertical ai app cos say they need the next model releases to unlock the next step-change in their product capability.
What if your current model was already good enough? We tuned an agent's harness around a model 9 months older than the latest one. It beats the latest model by almost 10 points.
Ella retweeted
Researchers in our #AI Engineering Group & CTO Office are publishing 8 papers at @aclmeeting this week about agentic systems, #responsibleAI & financial NLP; learn why their results are notable + how their work will advance the state-of-the-art
bloom.bg/4eTnAdn
#ACL2026NLP
Ella retweeted
Larry Ellison is right. Models are trained on the same data. The differentiator now is expert human feedback applied to real workflows.
Doctors, lawyers, engineers. People who actually do the work. Observing how models fail in production, providing the cognition to fix it, and feeding that signal back into training.
That loop is the new moat: observe failure → extract expert judgment → build verifiers → post-train → deploy → observe new failures. Each cycle makes the model better at harder, longer-horizon tasks that general training data never covered.
The companies that own that loop will power the next generation of AI.
Larry Ellison $ORCL highlighted something critical: models like ChatGPT, Gemini, Grok, and Llama are all trained on largely the same public internet data.
When everyone trains on the same information, models inevitably converge. That’s why AI is moving toward commoditization.
The real moat isn’t the model itself. It’s the proprietary data behind it.
Companies that can train on exclusive datasets gain an advantage competitors can’t replicate.
Having data that no one else has will allow you to dominate your market.
This is awesome! Go team Flint!! 🚀
When the cost of code goes to zero, marketing is your only advantage.
Introducing Flint. It builds you a unique page for every ad, keyword, and customer.
We’re already doubling conversions for @Cognition and @Graphite. Sign up at @tryflint.
358
Text-to-SQL remains one of the cornerstones of robust agentic systems, especially in domains with low tolerance for error.
One insight: treat SQL generation more like software testing.
Our agentic framework PExA explores this idea and pushed SOTA on the Spider 2.0 leaderboard
A team of researchers from Bloomberg’s #AI Engineering group introduced PExA, an #agenticAI framework that achieved 70.2% execution accuracy on the Spider 2.0 leaderboard, one of the most demanding benchmarks for #text2sql generation
bloom.bg/4af28yB
#CodeGeneration
(1/2)
242
I'm excited to share that our paper "EntSUMv2: Dataset, Models and Evaluation for More Abstractive Entity-Centric Summarization" was accepted to #EMNLP2023 !
Please stop by our poster to learn more about our research on Dec 9th at 11am (GMT+8) in the East Foyer
#NLProc
Researchers from @Bloomberg's #AI Engineering Group co-authored 4 papers at @emnlpmeeting in Singapore this week; learn more about their research, why the results are notable, and how their work will advance the state-of-the-art in #nlproc
bloom.bg/3GTbK2p
#EMNLP2023
1
2
681
Learn more about our paper, "Extractive Entity Centric Summarization as Sentence Selection using Bi-Encoders," and stop by our poster at @aaclmeeting !
Researchers/engineers from our #AI Engineering Group are co-authors on 4 papers at @aaclmeeting this week; learn more about their #NLProc research, why the results are notable, and how their work will advance the state-of-the-art in #NLProc
bloom.bg/3XjNsFr
#AACL22
6
🎉 #aacl22
Replying to @TechAtBloomberg
(2/4) Congrats to Ella Hofmann-Coyle, Mayank Kulkarni, Jane Xie, @gtcomputing’s Mounica Maddela (@mmaddela1005), & @Daniel_Preotiuc for having their paper “Extractive Entity-Centric Summarization as Sentence Selection using Bi-Encoders” accepted for #aacl22
6
Ella retweeted
Stop by our #SIGIR2022 booth to talk with our #AI researchers & engineers about how they tailored #opensource #informationretrieval technologies & built #questionanswering systems from scratch to help @Bloomberg's clients find the information they need, when they need it
#NLProc
Ella retweeted
#CFP: Applications for Bloomberg's #DataScience Ph.D. #Fellowship for the 2022-2023 academic year are now being accepted -- submission deadline: April 15, 2022 (11:59 PM AoE)
bloom.bg/36C8JEN
#MachineLearning #ML #NLProc #AI #informationextraction #annotation #dataforgood
My superior sense of navigation was humbled today. Apparently I'm lost without the nyc grid and low latency maps.
At least I had time to take this photo while pondering how lost I was
2
10
















