Justin (@chojustin) profile picture

Locus @Intology

San Mateo
Joined October 2018
Photo by Justin (@chojustin)Photo by Justin (@chojustin)Photo by Justin (@chojustin)Photo by Justin (@chojustin)Photo by Justin (@chojustin)Photo by Justin (@chojustin)Photo by Justin (@chojustin)Photo by Justin (@chojustin)Photo by Justin (@chojustin)Photo by Justin (@chojustin)
Locus sets SOTA on automated post-training capabilities and produces a model that's in production facing millions of users, completely end-to-end! Here's a glimpse to the endless possibilities we're materializing with our research on automated scientific discovery:
The models are improving the models. Locus, our automated AI research system, is SOTA on PostTrainBench and post-trains Qwen3 base models that surpass the human post-trained Qwen3 model. Today, LLMs post-trained end-to-end by Locus are in production to millions. 🧵👇 PostTrainBench evaluates agents' ability to post-train models on various domains given 10 H100 hours. We extend PostTrainBench via PostTrainBench+, which has a greatly expanded compute budget that provides clearer signal on automated post-training capabilities. We find that thousands of H100 hours help distinguish methods' performance post-training Qwen3 1.7B-Base models, and that Locus scales best. In this setting, modes trained by Locus collectively surpass the perforamce of the offical human post-trained Qwen3 1.7B model. In a test of generalization, we ran Locus on all live Kaggle competitions with prize money and public leaderboards. After 16 days, Locus achieved the 4th highest average rank among all participants.
1
18
1,361
by far the most important result on AI for research this year!
Locus sets the frontier for automating molecular design Our automated research system Locus develops scientific AI models that outperform SOTA expert solutions Unlocking the potential of scientific AI requires leveraging proprietary data. Locus achieves this autonomously 🧵👇 -- We test Locus across a number of tasks related to molecular scoring and design: training molecular property prediction models for a proprietary task, fine-tuning Boltz-2 to predict small-molecule potency, and building antibody design workflows around BoltzGen, Boltz-2, ESMC, and ESMFold2. Each of these settings are real-world AI for science use cases, where tasks are often out-of-distribution for existing foundation models and limited in data, while featuring expensive and noisy evaluators. Compared to state-of-the-art baselines and LLM agents set to optimize these tasks, Locus achieves higher performance. Additionally, Locus generalizes better despite limited data, finds more sample efficient solutions, and explores a higher diversity of solutions.
2
4
473
Justin retweeted
I recently joined @intology because it’s clear that autonomous discovery systems can now accelerate progress on hard research problems, and I’m excited to scale Locus to solve more complex tasks, both in the sciences and beyond!
4
10
237
After 4+ years building models at @AmazonScience (Alexa + AWS AI), I joined @intology a few months ago! We’re automating AI R&D. Here are our results showing Locus building scientific AI models that beat SOTA expert solutions in molecular design
Locus sets the frontier for automating molecular design Our automated research system Locus develops scientific AI models that outperform SOTA expert solutions Unlocking the potential of scientific AI requires leveraging proprietary data. Locus achieves this autonomously 🧵👇 -- We test Locus across a number of tasks related to molecular scoring and design: training molecular property prediction models for a proprietary task, fine-tuning Boltz-2 to predict small-molecule potency, and building antibody design workflows around BoltzGen, Boltz-2, ESMC, and ESMFold2. Each of these settings are real-world AI for science use cases, where tasks are often out-of-distribution for existing foundation models and limited in data, while featuring expensive and noisy evaluators. Compared to state-of-the-art baselines and LLM agents set to optimize these tasks, Locus achieves higher performance. Additionally, Locus generalizes better despite limited data, finds more sample efficient solutions, and explores a higher diversity of solutions.
2
4
24
1,366
We went from SOTA on autonomous post-training to frontier performance on autonomous molecular design (eg antibody design) in just a couple of months! What's exciting about these results is that Locus achieved them via improvements in domain-agnostic auto-research capabilities: - Diverse exploration - Better Generalization to avoid overfitting - Resilience to noise - High sample-efficiency to learn from data-scarce tasks This marks a crucial milestone towards building auto-research systems that can closing the loop for scientific challenges that have even larger search spaces and longer, more expensive, and noisier evaluation cycles.
Locus sets the frontier for automating molecular design Our automated research system Locus develops scientific AI models that outperform SOTA expert solutions Unlocking the potential of scientific AI requires leveraging proprietary data. Locus achieves this autonomously 🧵👇 -- We test Locus across a number of tasks related to molecular scoring and design: training molecular property prediction models for a proprietary task, fine-tuning Boltz-2 to predict small-molecule potency, and building antibody design workflows around BoltzGen, Boltz-2, ESMC, and ESMFold2. Each of these settings are real-world AI for science use cases, where tasks are often out-of-distribution for existing foundation models and limited in data, while featuring expensive and noisy evaluators. Compared to state-of-the-art baselines and LLM agents set to optimize these tasks, Locus achieves higher performance. Additionally, Locus generalizes better despite limited data, finds more sample efficient solutions, and explores a higher diversity of solutions.
1
2
17
866
🚨New work: A SOTA prompt optimization framework + a novel unsupervised measure of prompt performance. Written rubrics and constitutions are crucially important for AI task improvement, safety following, pluralism and adaptation. But some of the things we might want a rubric to do are difficult to put into words; how would you even describe “comic voice”? I’m human, albeit not that funny, and I'm totally not sure what that means. What kinds of written constructs can you expect your judge to do well on and how good can you expect it to be?
6
20
2
91
10,942
So excited for this research! I've always been annoyed at the supremacy of verified rewards, and now we can finally show it. In this work, we develop ways of measuring: how much of a reward signal is verifiable, how much can we put into words, and how much is shared but tacit, or impossible to articulate?
5
43
7
276
28,107
Introducing GameBoyWorlds, a new challenge testbed for self-improvement in embodied video games: Paper: arxiv.org/abs/2609.32093 Website: dhananjayashok.github.io/Gam…
7
31
2
162
310,244
Justin retweeted
Thank you @jackclarkSF for covering our work with Locus! "AI startup Intology, whose goal “is to automate R&D”, has released a new version of Locus, software it has developed to turn LLMs into capable researchers. The new version of Locus is able to get a score of 44.7% on PostTrainBench, a benchmark which sees how well AI systems can take an open weight model and improve its performance above its baseline." "Posts like this highlight how we are under-eliciting today’s AI systems for their ability to automate AI R&D - especially striking is how the company can jump the performance of Opus 5 by 10 absolute percentage points"
Thanks to @jackclarkSF for covering @intology's work on Locus! "Posts like this highlight how we are under-eliciting today’s AI systems for their ability to automate AI R&D - especially striking is how the company can jump the performance of Opus 5 by 10 absolute percentage points" Full blog: importai.substack.com/p/impo…
3
5
1
19
3,119
Justin retweeted
Thanks to @jackclarkSF for covering @intology's work on Locus! "Posts like this highlight how we are under-eliciting today’s AI systems for their ability to automate AI R&D - especially striking is how the company can jump the performance of Opus 5 by 10 absolute percentage points" Full blog: importai.substack.com/p/impo…
2
1
11
3,017
Justin retweeted
Thank you to @tbpn for having us on last week. Ron Arel (@ronusedh, co-founder of @intology) spoke on our research direction — our past work, and our new state-of-the-art results on automating post-training.
2
4
1
21
3,458
Super cool results on PostTrainBench! They also introduce a very interesting setting of PTB where the agent hours are extended by a large margin (4500 GPU hrs!)
The models are improving the models. Locus, our automated AI research system, is SOTA on PostTrainBench and post-trains Qwen3 base models that surpass the human post-trained Qwen3 model. Today, LLMs post-trained end-to-end by Locus are in production to millions. 🧵👇 PostTrainBench evaluates agents' ability to post-train models on various domains given 10 H100 hours. We extend PostTrainBench via PostTrainBench+, which has a greatly expanded compute budget that provides clearer signal on automated post-training capabilities. We find that thousands of H100 hours help distinguish methods' performance post-training Qwen3 1.7B-Base models, and that Locus scales best. In this setting, modes trained by Locus collectively surpass the perforamce of the offical human post-trained Qwen3 1.7B model. In a test of generalization, we ran Locus on all live Kaggle competitions with prize money and public leaderboards. After 16 days, Locus achieved the 4th highest average rank among all participants.
2
3
26
4,353
Justin retweeted
i’m increasingly convinced that the best agent evals will come from mining real agent failure traces. my view is that every failed trace contains a potential eval but not in its raw form. raw traces are messy, long and too specific. the research problem is to distill them into clean reproducible tests. the pipeline i’m interested in is (which i'm currently working on): failure trace → failure attribution → earliest divergence point → minimal reproducible state → targeted eval → regression suite this turns trace data from passive observability into an active improvement loop. like can we extract the exact decision point where the agent should have behaved differently? and can we convert that into an eval that catches the same failure class in the future? i guess this matters because most agent failures are trajectory-level failures and not just output-level failures. personally i think this is much more realistic than relying only on hand-written benchmarks (imo they should look more like failure memory systems). hand-written evals encode what we think agents will fail on. traces encode what agents actually failed on. also once you have the mechanism, you can mutate the trace into variants. that is basically fuzzing for agents.
25
23
10
307
56,400
True progress is built on rigorous evals Contrary to hype, coding agents make little scientific progress when * tasked with long-horizon research * controlled for data contamination * allowed no human interventions NanoGPT-Bench paves an exciting path towards scientific agents
Can coding agents do research? We release NanoGPT-Bench, an internal eval we’ve used to test agents on an AI R&D problem with months of human progress Codex, Claude Code, Autoresearch recover only 9.3% of human progress, mostly tuning hyperparams & ignoring algorithmic research NanoGPT-Bench is built on the NanoGPT Speedrun, a popular LLM pretraining competition to minimize the training time of a GPT-2 style model. Existing human submissions constitute nearly 2 years of work. To control for dependencies and contamination in frontier models, we standardize evaluation to a 5-month window of world records. Evaluation is fully autonomous and end-to-end, with no human intervention or internet access. 🧵
1
11
1,669
great work led by @soren_dunn_ 🚀
1
131
I've joined @intology! I'm excited to push the boundaries of AI-accelerated scientific discovery with an incredibly driven and talented team. Looking forward to dive deep into research on AI-driven automation and creativity!
2
4
12
1,786
New NanoGPT Speedrun WR at 105.9s (-1.0s) from @soren_dunn_ , with a triton kernel to fuse the logit softcap and multi-token prediction cross entropy calc. Interestingly, Soren mentioned that their autonomous system Locus at Intology discovered and implemented the improvement. github.com/KellerJordan/modd…
12
6
138
20,083
Can’t recommend working with Alex enough! It’s hard to think of a positive trait that doesn’t apply to Alex, he’s creative, caring, passionate, and super sharp. You shouldn’t miss out on this opportunitiy if ur planning on doing a PhD
✨ Very overdue update: I'll be starting as an Assistant Professor in CS at University of Minnesota, Twin Cities, Fall 2026. I will be recruiting PhD students!! Please help me spread the word! [Thread] 1/n
1
7
1,109
No better time to learn about that #AI thing everyone's talking about... 📢 I'm recruiting PhD students in Computer Science or Information Science @Cornell_Bowers! If you're interested, apply to either department (yes, either program!) and list me as a potential advisor!
19
104
7
533
124,227
Can you tell what actions are being mimed in this video? If so, you’re smarter than AI models! Check the last tweet in this thread for answers. In a new paper, we present MIME, which evaluates whether vision language models (VLMs) have a robust understanding of human actions. 🧵
1
9
4
26
5,272