A high-signal data ecosystem leveraging data from industry & academia to deliver post-training to frontier labs, neolabs, & enterprises building advanced models
San Francisco, CA
Joined May 2026
- Tweets314
- Following258
- Followers85
- Likes182
Models read raw experimental data badly but write decent code to analyze it. On CME-SpectrumBench, from Jin Gene Wong and Zhi-Xun Shen at Stanford with Berkeley coauthors, a Python interpreter closed most of the gap with handwritten code on simulated spectra, except when noise was severe. So the useful skill is turning data into code, not eyeballing arrays. ReasonCore has 5,000 CritPt-style physics tasks in inventory. arxiv.org/abs/2610.24304
cxbench, from researchers at Lero, University of Limerick, shows a customer-service agent can get the right final answer and still break the rules on the way. About 80% of runs that passed the outcome check failed a process check, like skipping a required disclosure. Scoring only the end state counts those as wins. arxiv.org/abs/2610.25954
Giving an agent tools and memory does not reliably make it more accurate.
A new measurement study ran three open models in three open-source agent setups. The extra machinery rescued some questions the model missed alone, but broke others it had answered correctly without it. Skill instructions and retained history did not reliably help either.
Test the whole agent, not just the model. arxiv.org/abs/2610.20675
ReasonCore AI retweeted
Self-improving research agents are one of the most discussed ideas in AI right now.
(bookmark it)
Épi is a research agent that keeps working on the same research problem across many model sessions. It set 120 records, including an interval for the n-queens constant about 84 times narrower than before.
It has a persistent research history, hierarchical memory, versioned branches it can return to, and an independent reviewer. A fixed verifier scores every output, so the agent cannot change how it is judged.
Other results include exact certificates showing the Cohn-Elkies bound cannot be sharp in dimensions 10 and 11, and a computer-assisted proof of a case of Tuza's conjecture for split graphs.
A lighter version, Épi-turbo, writes solvers for routing and scheduling problems. On 30 FrontierOR tasks, it scores 5.4% higher than Codex at 64% of the model cost, with the same model and reasoning effort.
arxiv.org/abs/2610.24944
Chat with Paper: academy.dair.ai/papers/epi-t…
A plan can read well and still have the steps in the wrong order.
StepBench takes thousands of real how-to procedures and tests whether an AI agent can spot a missing step, a wrong step, or a shuffled sequence. Putting steps back in order was the hardest part. The best model got it right 63% of the time, and only about a third of the time on long procedures with nine or more steps. arxiv.org/abs/2610.26679
Agents that plan should be graded on the order, not just the final summary.
You can make a benchmark harder without changing what the task is.
Researchers kept each task's goal fixed and changed how hard it is to get there: what information the agent can see, how resources connect, what depends on what. Every model they tested did worse on the harder versions. Training on those harder runs beat training on the original ones.
Same goal, tougher world. That is a way to keep tests from going stale. ReasonCore has tasks in the area of recursive self-improvement. arxiv.org/html/2610.16397
ReasonCore AI retweeted
Most AI coding tools prove themselves on one benchmark, built by a small team doing the hard, thankless part.
ReasonCore AI retweeted
Underrated failure mode in RL envs. Overfit tests usually aren't letting hacks through, but they also reject correct answers that just took a different route. A second oracle is a cheap way to catch it. I need to try this out, but double work to write two oracle solutions!
ReasonCore AI retweeted
harbor tasks can now ship multiple oracle solutions.
think of solution/ as the unit tests for tests/.
Single solution only proves your tests accept one approach. Whereas multiple solutions catch tests that are overfit to it, and make verifier easier to debug and robust.
Saying a fix works is not the same as proving it. In CyberArena, a common defender mistake was claiming a patch worked after checking logs from before the patch went live. For security agents, the grade should come from attacking the live fix again. arxiv.org/html/2610.19005
A check you can peek at before you commit stops being a check.
A small Yonsei preprint had an AI agent design machine-learning pipelines and explain why each design works. A verifier tested those explanations by removing what the agent said mattered and rerunning.
When the agent could probe the verifier before committing, it kept the claims that passed. Held-out faithfulness fell, and the in-loop metrics never showed it.
Preliminary: one model, six tasks, and the decline is partly confounded by how the score treats small pipelines. The useful part is the method: stress-test a verifier before you trust it. arxiv.org/html/2610.22779
Customers rarely say what they actually need. FDE-Bench hands agents a vague business request and makes them ask questions first. The best setup matched the expert solution in only about 6% of runs. Good questions are part of the job, not a warm-up. arxiv.org/html/2610.14539
Plotly's new data analytics exam has a useful warning buried in it. The model with the cheaper token price made nearly three times as many calls per question and read four times as many tokens, so its full run cost seven times more than the pricier model's, and took three times longer. Token price lists tell you what a call costs, not what an answer costs. For agents that plan their own calls, the meter that matters is the finished task. plotly.com/blog/introducing-…
A business agent that completes the task but breaks policy along the way hasn't succeeded; it's created a liability. BizBench, built on real procedures from a payments company, grades the path as well as the outcome: whether the agent respected constraints, conflicting guidance, and prior commitments, because a violation midway can carry financial or regulatory costs even when the final answer looks right. One open model reached 42% on goal completion but only about a third once its process was audited. ReasonCore has CheatBench-style tasks in inventory. arxiv.org/html/2610.20424
DeepSeek says its newest Flash model learned to finish whole projects: reinforcement learning on its own attempts, with demos that are complete builds - a 3D Forbidden City, a Portal 2 remake in Godot. Those are vendor demos, but the weights are MIT-licensed, so anyone can run the model on a project of their own. If the approach holds, the interesting shift isn't another benchmark table; it's grading models on whether the project works when they're done. api-docs.deepseek.com/news/n…
ReasonCore AI retweeted
Water, water, every where, Nor any drop to drink.
Coleridge described the RL data market two centuries early.
1. Terminal-Bench-Science 0.1 drew 920 task proposals and shipped 70. @StevenDillmann
2. An audit of five public terminal-agent benchmarks found over 15% of their tasks were reward-hackable.
3. @EpochAIResearch interviewed 18 people across labs and RL-environment startups. Their conclusion: “scaling while maintaining quality is the core operational bottleneck.”
4. Ivan Bercovich (@neversupervised) points at the cause: a fast-growing market rewards shipping tasks quickly, often before anyone has reviewed them adversarially. arxiv.org/abs/2604.28093
From inside the data supply chain: task pipelines run on domain experts racing deadlines and operators managing throughput. The scarcest seat is the adversarial reviewer, the person (or algorithm) paid to break a task before a model does.
Fresh water is an engineering problem. The next leap belongs to whoever builds the desalination plant. We are building that plant at @ReasonCoreAI
Footnote:
A short meeting with @jackclarkSF in SF made me think about AI beyond engineering. I decided to draw, albeit with AI. Let me know if you like "my" painting.
ALT You are right. I made it with AI
Independent testing gives @reflection_ai's Beam a strong start in science. On CritPt, a research-level physics benchmark, Artificial Analysis measured Beam at more than double the two open-weight peers in its comparison, K2 Horizon and Qwen3.8 27B. Reflection says scientific reasoning was a training focus, and this first result points the same way. It is one benchmark from one evaluator, but research-level physics is the right test to lead on. @ArtificialAnlys artificialanalysis.ai/articl…
When top models all ace a test, the usual move is to retire it. That may be too early: how a model solves the items everyone gets right predicts how it handles the ones nobody can. A judge comparing two correct answers ranked models far better than their identical scores did. The catch: it breaks when the paradigm shifts - it missed the first reasoning models. The score saturates before the signal does. ReasonCore has thousands of HLE-style tasks in inventory. arxiv.org/html/2610.20667