AI + bio | Views not always my own, but never more than my own | spicier takes and toddler photos @TrainOnTest | https://xitter.cf/t.co/KfBNuwu51j
Boston
Joined November 2018
- Tweets6.6K
- Following1.7K
- Followers762
- Likes24.8K
much colorado about nothing
this is definitely “polysemantic”. it’s important to note this so critics don’t accuse you of drawing schizophrenic connections to support any theory. but if you read Nick’s paper he claims that some qwen neurons have a second unrelated smaller peak; that is still a much better interp world than we lived in last week, where we thought neurons could be an arbitrary mismash of superposed concepts. hell even if there are 3 or 5 we can read the stack of meanings like Lyra’s alethiometer
few interesting things about this neuron:
- it really likes the Tornado web framework but doesn’t care about tornadoes
- it responds weakly even to fictional software frameworks called Tornado
- it gets somewhat less excited by arbitrary web frameworks
- there is no connection to Ipython, which was invented in Colorado
- there is no connection to Apache, a software license but also a tribe that lives in the American southwest
- -ado connection doesn’t exist, “el dorado” does nothing
update: i realized that @jrkelly and i are in agreement 🫶 -- fun conversation thread below:
to the contrary, single-entity readouts (vs pooled readouts) are the true bottleneck. automating plate-based assays is premature optimization, which Donald Knuth tells us is the root of all evil. (but it's a great business model if you have high time preference!)
I'm no fan of "I'm no fan of ____, but...", but it is one of the signature rhetorical moves of our times.
Calvin McCarter retweeted
we have improved the opendde inference process, allowing more efficient and effective co-folding, from 0.14 to 0.75 top1 success rate with inference guidance.
this week, we will update our codebase with a short tech report for people to review. this new inference algorithm can be used in anthropic x adaptyv protein design challenge ❤
AI Drug Discovery Is Becoming a Bottleneck Trade
I get excited when an industry starts going through a real regime change. AI drug discovery (“AIDD”) increasingly looks like one of those moments.
Two things are happening:
1/ Upstream: the frontier AI labs are piling in.
2/ Downstream: The supply chain is clearly moving.
Supply-chain checks suggest the upstream picks-and-shovels of discovery and early preclinical R&D are starting to feel the increase in experimental volume.
DNA → protein → assays → sequencing → automation → preclinical testing
Names across that stack include $TWST , @GenScript , $ILMN , $TXG , lab-automation vendors and CROs.
$TWST expects triple-digit percentage growth in AI-enabled drug-discovery orders in FY26, and another year of triple-digit order growth in FY27.
@GenScript's AIDD business doubled YoY in 1H26. Its current platform advertises industrial-scale validation of 4,000+ designs/day, with integrated sequence-to-data workflows. Our channel checks suggest the ramp is moving even faster: roughly 8,000 designs/day currently, with a path toward ~16,000/day by YE26.
---
Why does AI drive more wet-lab demand?
1/ AI makes hypothesis generation almost free → way more shots on goal.
The bottleneck is moving from expert-driven design to biological validation.
2/ AI models need continuous experimental feedback — and both good and bad data are useful.
Traditionally, only the highest-conviction A+ candidates might get pushed into expensive validation. With AI, even the B/C candidates can be valuable because failed experiments generate training data.
@GenScript has said its sequence-to-binding workflow can return data in 4–7 days, and that faster cycle times matter because AI models depend on continuous experimental feedback.
@Anthropic is a clean example. @claudeai designed 1,320 protein binders. @adaptyvbio converted those digital sequences into DNA, expressed the proteins and tested binding. Only 354 actually bound. And the 966 failures are not wasted. They are useful negative labels: what does not express, what does not bind, what has poor affinity. Those results help train the next model iteration.
3/ Wet labs are no longer just making drugs. They are making training data.
$TWST / @GenScript increasingly look like biological data foundries.
$TWST explicitly talks about generating model-ready data from AI-designed sequences. In some workflows, the customer may care less about receiving the physical protein than about getting structured experimental results back into the model.
Traditional drug discovery asks: “Does candidate X work?”
AI drug discovery also asks: “What can this experiment teach the model?”
---
TAM of AIDD
If AI is simply a better R&D tool, the relevant spending pool is the $300–400B of annual global pharma R&D. If AI meaningfully increases the number of viable drug programs, the opportunity is larger because it expands downstream demand for DNA synthesis, protein production, assays, and preclinical work.
Near term, we can also size demand from AI-company spending. If Anthropic reaches $80B of ARR in 2026 and spends just 1% on AIDD, that alone would imply ~$800M of annual investment.
---
Trade setup
This is a trade that could have long legs. It’s hard to really stop working until PhaseI/II results (2028+)
It smells a lot like the bottleneck trade we just saw in semis: GPUs → HBM → networking → power/cooling. In biology, It basically follows the drug discovery process downstream: AI models → designs → DNA/protein → assays → preclinical capacity.
After the upstream picks-and-shovels, animal testing could become the next bottleneck. Monkey prices are already near prior highs and CRO capacity is tight. AIDD pushing more candidates into preclinical development would only add demand.
?? But clinical trials are still the bottleneck?
This is the biggest pushback I keep coming back to. No matter how fast discovery becomes, drugs still need to go through preclinical → Phase I → Phase II → Phase III → approval. You still need patients, time and capital.
But that doesn’t mean the bottleneck trade won’t work. More viable candidates — especially with higher success rates — still means more demand throughout the development process.
And who knows: clinical trials themselves may eventually be optimized by AI.
?? What breaks the trade?
Near term, the picks-and-shovels trade breaks if experimental budgets stop growing, AI-generated designs don’t translate into useful wet-lab hits, or capacity catches up too quickly.
Longer term, the thesis breaks if AI drugs look great in discovery / Phase I but fail at normal rates in Phase II/III. That is why Phase II matters so much.
?? Milestones
Late 2026–2027: first Isomorphic-designed drugs enter human trials; more AI-native programs move into IND-enabling work / tox.
2028–2030: clinical trial results start telling us whether AI-designed drugs actually perform better than conventional drugs.
Calling all the "bottleneck bros". :) @jukan05 @zephyr_z9 @aleabitoreddit @ParadisLabs
+++
More comprehensive analysis: robonomics.substack.com/p/ai…
Calvin McCarter retweeted
Short term yes CROs are gonna pump, long term there’s too much alpha in having a model that eats its own cooking. In house labs are going to be table stakes
Strongly agree with the picks and shovels approach and that CROs are the most obvious winners in the short term. We will see the number of compounds being generated by AI explode and the pressure now shift towards physical testing.
The focus of AI Bio is squarely aimed at coming up with drugs right now but I think in the long term the more valuable application is in drug testing & medical diagnostics. Any new technique for testing drugs that increases the likelihood of passing a clinical trial should be more valuable than just the drug themselves, similar logic also applies for identifying targets.
In the future once we know the target creating a drug candidate will probably be as trivial as prompting Codex which means the edge shift towards testing & verifying them more accurately and quicker!
Now faith is the substance of things hoped for, the evidence of things not seen. For by it the elders obtained a new SOTA.
I really enjoyed working on this project. I believe the new flop-aware paradigm of nanoGPT thinking is useful, and it seems helpful in a compute optimization context like this one. It seems like a simple and logical conclusion of optimization: figure out what’s actually needed, and use that. The process was actually fun.
One note regarding this record, as noted in the blog. Some of the techniques that scale to frontier pretraining have been redacted from this record. Most importantly, ANVIL III, the significantly improved version of the ANVIL optimizer lineage, has not been included in this record, as it is proprietary to us at @hyperstition_cc. It replicates Muon’s speed but beats it by 20-28 millinats of loss at all model sizes 124M-1.2B and training depths (up to 8x-Chinchilla), with remarkably consistent hyperparameters and extremely high SNR, as compared to Muon. You can learn more about ANVIL III and Feather-1.7B, the model we trained using internal architectural changes and ANVIL III, here:
hyperstition.cc/cutting-pret…
Using ANVIL III (which allowed me to cut steps) improved the speedrun results by another 0.7 seconds, with lower net loss. This alone proves there’s still more room to improve this record.
One final thing that I’ve had many people ask about is how much of a role autoresearch played in my work. The answer is almost none. While almost all of the implementation was done with Claude Code, I did the ideation almost fully by myself. I spoke about this extensively in a conversation with @METR_Evals.
We had a cluster of H100s while I was working on the record. At night or when I stopped working on it, I’d send out several agents to try to hill-climb the speedrun and improve the record. In total, the agents made shockingly little progress, despite spending more time in aggregate than I did. No matter what I told them, it seemed difficult to get them out of local minimums, stuck performing futile tuning of a method that saved 0.1 seconds and cost ~5 millinats (clearly a bad trade), or just tuning hyperparameters for hours on end trying to make some small change. I saw similar phenomena as those described in this METR report:
metr.org/blog/2026-07-21-exp….
However, as I also discussed with METR, I don’t think the agents lack the knowledge to do this. Instead, I think a lot of it comes from a personality issue with current LLMs. They have very little belief that large jumps like this are possible, or they approach it with the wrong paradigm. Autoresearch was not very helpful in my work, and, from personal experience, I don’t think this current generation of publicly available models, without special harnesses, will do very high-quality autoresearch. However, I think the capability and knowledge is already here, which implies that we could be significantly closer to true autoresearch than recent results would imply.
NanoGPT was a really fun challenge and it led to a number of valuable insights and processes. Finally, I’d like to give @Classiclarryd and @KellerJordan a huge thank you for all of their efforts maintaining the repo.
I'll be at the New England CompBio Symposium! I'm excited to chat about how ML for systems biology (aka virtual cells) has progressed since I switched fields years ago, and on how Aureka Bio is combining AGI and protein design. Reply or DM if you want to meet, or just say hi! 🌊
1/ I'm excited to share that we're launching NECB 2026, the inaugural New England Computational Biology Symposium. Oct 1-2 at Microsoft Research New England, Cambridge. Two days of keynotes, talks, and posters to bring our community together! newenglandcompbio.org
The time has come for a Reconstituted Ad Hoc Committee for Stamping Out Nor’easter.
ALT https://www.newyorker.com/magazine/2005/09/05/noreaster
this paper is going to sydney, atlanta, or paris
How to Make the Most of Your Masked Language Model for Protein Engineering
1 The paper introduces a new sampling method for masked language models (MLMs) in protein engineering that significantly outperforms existing approaches, with the key finding that choice of sampling algorithm matters at least as much as choice of model.
2 The core innovation is temperature-annealed stochastic beam search, which reframes generation as sequence evaluation rather than mutation-centric sampling, exploiting MLMs' efficiency at evaluating pseudoperplexity of entire 1-edit neighborhoods.
3 This sequence-centric approach enables flexible multi-objective optimization with black-box scoring functions, overcoming limitations of mutation-centric methods that struggle with non-differentiable scores like OASis percentile or isoelectric point.
4 The method achieves dramatic computational speedups: O(BEL³) complexity compared to O(EL³) per sequence for traditional approaches, yielding 20EL× speedup for realistic scenarios while producing higher-quality sequences.
5 Extensive in vitro experiments on actual antibody therapeutics campaigns reveal that ESM-2 (trained on generic proteins) and AbLang-2 perform exceptionally well, with beam search consistently outperforming Gibbs sampling across all models.
6 When combined with supervised guidance using smooth Tchebycheff scalarization, the approach achieved 100% success rate in synthesizability and binding QC criteria, while Pareto-based non-dominated sorting showed competitive but slightly lower performance.
7 The work provides practical recommendations: use supervision when possible, prefer AbLang-2 or ESM-2-650M for antibody engineering, adopt stochastic beam search over Gibbs sampling, and carefully consider scalarization methods for multi-objective guidance to avoid undesirable side effects.
📜Paper: arxiv.org/abs/2603.10302
#proteinengineering #antibodydesign #machinelearning #bioinformatics #computationalbiology #drugdiscovery #languagemodels #MLM #beamsearch #multiobjectiveoptimization
I am now at Aureka Biotechnologies, and I could not be more excited. AGI will help cure disease, if we use AGI to empower patients and the entire drug-hunting ecosystem, rather than to concentrate capabilities in one or two companies. Aureka is making the bright future happen.
Calvin McCarter retweeted
Everyone working with AI should read this.
I've got a pretty clear picture now of where we're headed next year, when Fable-class models become ubiquitous and cheap, and every enterprise is flooded with hundreds of new Fable-class AI employees.
Spoiler: They create a constitutional legal system.
yegge.ai/essays/fences-not-s…
I have no work-related obligations for the next 2 days. AMA!
After 2.5 fantastic years, yesterday was my last day at BigHat Biosciences. I'll deeply miss working with all my brilliant and kind colleagues there. To anyone searching, I highly recommend joining the ML team there! I start a new adventure next week, but more on that later. 💜
i found this essay to be much more compelling than machines of loving grace
An actually-serious case that being a “biological bootloader for digital superintelligence” (per @elonmusk) is a credible, dignified, and good-enough “meaning of life.”
I started writing this one partly as a joke (“Elon Musk accidentally solved the meaning of life in a tweet!”) and then it evolved into a long exploration of what humans actually want out of a “meaning of life,” the long religious tradition of “being instrumental,” and why the ultimate manifestation of the thing we call “the good” might not be containable in human beings.
A bit viscerally scarier to revisit this one after recent events with “agent civilizations,” but I hope the perspective here can provide some comfort that the future might be okay (and great, even, and beautiful), even if everything is not necessarily “okay” for us.
Enjoy The Silicon Man.
over the course of one's life, one experiences that
"He that loveth his life shall lose it; and he that hateth his life in this world shall keep it unto life eternal."
is true on an individual level. perhaps this principle is also true of our species