akshay Balsubramani (@_bakshay) profile picture

💻ML/AI |🧬🧪| 📈 • Biotech industry • Ex @StanfordMed, PhD @UCSD, big tech &c • 🇸🇬🇺🇸🇮🇳 • Caveat: humor

SF Bay Area
Joined August 2016
Photo by akshay Balsubramani (@_bakshay)Photo by akshay Balsubramani (@_bakshay)Photo by akshay Balsubramani (@_bakshay)Photo by akshay Balsubramani (@_bakshay)Photo by akshay Balsubramani (@_bakshay)Photo by akshay Balsubramani (@_bakshay)Photo by akshay Balsubramani (@_bakshay)Photo by akshay Balsubramani (@_bakshay)Photo by akshay Balsubramani (@_bakshay)
akshay Balsubramani retweeted
Has nominative determinism gone too far (h/t @bschne)
48
165
24
2,252
149,032
akshay Balsubramani retweeted
What if a [MASK] was all you needed? ModernBERT is great, but we couldn't stop wondering if it could be greater than previous encoders in different ways. Maybe we don't need task-specific heads? Maybe it can do all sort of tasks with only its generative head? Spoilers: Yes
17
101
19
791
165,028
“Shotgun” is a way better name than bootstrap, it isn’t even close.
Replying to @noah_greifer
Regular reminder that we almost had the meat axe.
2
2
65
7,110
akshay Balsubramani retweeted
A central challenge of AI-driven biotech is that in drug discovery a fully validated medicine is worth a million times more than a million early-stage ideas that only have little validation. This is unfortunately at odds with the AI domain where we almost always attempt to "solve" a general class of problems - which even in the optimal case provides us at most with a little bit of validation for a large space of ideas, with most questions remaining unanswered and relegated to further downstream steps. Because of the large number of steps involved in drug discovery, solving even a general class of problems within just one of these steps does not massively impact the overall enterprise. This dichotomy is at the core of the challenges that first-generation data-generating AI-driven biotechs face - platforms produce a million low-validation ideas that then further clog up the validation bottleneck. Day to day, swimming in an ever-expanding lake of early-stage ideas may feel like progress, but the impact, measured in new medicines available for patients, is limited. Moreover, this approach to solve the general before the specific neglects the time factor - if you set out to solve a general class of problems before embarking on any specific idea, you will be overtaken by those that elect to instead address the valuable but specific problems in that space immediately. By the time your general capability is ready for action, you may find that the best ideas have already been pursued by others that did wait to first solve the entire class of problems. In short, we perhaps were too busy building platforms that we forgot to make medicines. What could we do differently to help address this central challenge? For one, there may be potential to focus more on the capabilities that enable us to produce fully validated or nearly fully validated ideas (depicted as a full glass of water) more quickly, that produce ideas of greater quality when fully validated, or perhaps even capabilities that open up new avenues that could not have been pursued otherwise. Secondly, it may help to mindfully balance general capability building with actually developing a medicine *today* to ensure we remain on track for realized and not only envisioned impact. The future of AI-driven drug discovery is bright but needs to be grounded in actual innovative medicines available for patients.
4
16
3
136
34,595
akshay Balsubramani retweeted
I wrote about some recent studies identifying widespread gene-environment interactions (GxE) acting on common traits, but through an unexpected mechanism. A few highlights 🧵:
3
60
4
270
31,971
akshay Balsubramani retweeted
Short, interesting antibody design paper using available tools (EvoDiff, ColabFold, HADDOCK, OpenMM). No wet-lab validation, but interesting to see the pipelines people are putting together. Appears to be related to silico.bio biorxiv.org/content/10.1101/…
2
8
1
53
3,598
This last chapter "Horoscopes" is only 3 pages but I keep thinking it should be at the start of the book, because it is about scientific and social context of statistics. Anyway here it is:
7
93
11
461
71,427
akshay Balsubramani retweeted
Then we repeated the experiment, but we dehydrated the tissue first. The architecture was fine. We left it at room temperature for a year. Still fine. Ran over it with a car, had a former Astros starting pitcher throw a fastball at it, fired a shotgun at it. Still fine. (15/18)
3
50
48
353
94,502
akshay Balsubramani retweeted
📈 One standard graph in all bio deep learning papers should be: max similarity to anything in training set vs performance. (Reviewers shouldn't have to guess if there might be overfitting issues).
12
26
3
236
24,100
akshay Balsubramani retweeted
Does more compute equate with greater risk? What is our track record at predicting what risks emerge with scale? I don't have much time anymore to write, but this felt important enough to write down some thoughts on. arxiv.org/pdf/2407.05694v1
9
77
20
322
144,336
👇👇 for biotech
we all want to and need to be prepared to train our own large-scale language models from scratch. why? 1. transparency or lack thereof 2. maintainability or lack thereof 3. compliance or lack thereof and because we can, thanks to amazing open-source and open-platform ecosystem. (1/12)
1
3
477
akshay Balsubramani retweeted
Replying to @Thom_Wolf
I'm surprised the MuMath-Code paper wasn't more discussed on Twitter. It is awesome! Super dense in information. Only reference I found was: teapaw.org/gm8xx8/status/17903650… who incidentally tweets about a lot of really interesting papers that don't get much attention.
MuMath-Code: Combining Tool-Use Large Language Models with Multi-perspective Data Augmentation for Mathematical Reasoning ↓ arxiv.org/abs/2405.07551
1
7
84
21,842
akshay Balsubramani retweeted
🧵Today is an exciting day! We have just announced ESM3, the design of a new green fluorescent protein, and the launch of our company. Here's a short thread about the scaling and infrastructure work involved in making ESM3 happen! 🧵
We have trained ESM3 and we're excited to introduce EvolutionaryScale. ESM3 is a generative language model for programming biology. In experiments, we found ESM3 can simulate 500M years of evolution to generate new fluorescent proteins. Read more: evolutionaryscale.ai/blog/es…
5
17
1
70
11,563
akshay Balsubramani retweeted
read the license before using ESM-3; github.com/evolutionaryscale…. you're welcome, pharmas and biotechs.
3
5
18
7,594
akshay Balsubramani retweeted
I think it's a relevant metric, it's just not very hard. If someone could take a GFP and make it 10000x brighter than any known ones with 192 shots I'd be far more impressed, even if it was at 3 mutations. ESM3 excites me most because it is a relatively high success rate model that lowers the bar for design for a non-ML person. I don't think it would beat an expert designer on its own. I'm willing to bet a high ED GFP can be made with a <1M parameter model and a couple of experts. The controllability is the key value add to me. The steam engine didn't go faster than a horse (at first), but it added a whole world of possibilities.
look, seq identity to some database is a bad proxy of a vanity metric. Protein engineering is about making big molecules do what we want. Nothing wrong with expanding libraries, “novelty” (independent of function) is just not the right metric to care about
6
9
65
22,629
akshay Balsubramani retweeted
What if we could universally recombine, insert, delete, or invert any two pieces of DNA? In back-to-back @Nature papers, we report the discovery of bridge RNAs and 3 atomic structures of the first natural RNA-guided recombinase - a new mechanism for programmable genome design
189
1,151
260
4,969
1,877,703
akshay Balsubramani retweeted
Taking a look at the ESM3 GFP, there is a similar but less pronounced pattern to the Profluent OpenCRISPR protein. The "nearest match" is 58% (from the paper) but you can get 74% from mixing and matching two near proteins. gistpreview.github.io/?81c10…
5
23
2
144
27,249