Research @EngramLab. previously CS PhD @StanfordAILab @HazyResearch, undergrad @princeton. Working on all things data! she/her 🎃
Stanford, CA
Joined February 2020
- Tweets731
- Following775
- Followers2.7K
- Likes6.1K
Pinned Tweet
I've spent my PhD figuring out how a model learns from data, digging through piles of Spanish math problems, fanfiction, and the depths of StackExchange. Engram opens a new dimension: how a model learns from you - your world, how it evolves, and the threads that tie it all together. Figuring out how to do this well is a wide-open research problem. Join us :)
Mayee Chen retweeted
Introducing Omnii-1, our most powerful genome language model, and a major step toward general biological intelligence
In a world first, Omnii-1 designed the first AI-generated bacteriophages to be administered to a human patient with an antibiotic-resistant infection under compassionate use
Mayee Chen retweeted
🚀 Introducing Next-Concept Modeling (NCM): our new paradigm for fast, parallel, coarse-to-fine text generation.
Compared to strong diffusion language models (DLMs) at similar parameter size, NCM achieves:
⚡ Up to 31× faster generation on downstream tasks like summarization and machine translation with comparable or better quality.
🏎️ 10–21× faster unconditional generation across TinyStories, LM1B, and OpenWebText, with comparable or better generative perplexity and using 3–5× fewer training tokens.
🧩 Better performance on reasoning tasks like Sudoku, outperforming both AR and diffusion baselines with fewer sampling steps.
With @ishanskhare, @IddahMlauzi, @ekellbuch, @simran_s_arora, @Azaliamirh, and @HazyResearch.
More details in the thread 🧵👇
Mayee Chen retweeted
AI researchers' idea of a good time: playing HORSE and debating continual learning between shots.
@oneill_c (co-head of training at Baseten) and @jxmnop (co-founder and head of research at @EngramLab) discuss:
- when to put knowledge in weights vs. the KV cache,
- why forgetting matters less for specialized models,
- on-policy self-distillation vs. RL, and
- what research publishing should look like today.
Watch the full game (and discussion) here: youtu.be/q_-_t5AmKz0
Mayee Chen retweeted
1/5 What does it take to scale pretraining to 20+ data modalities at once? Improved data mixing methods are one important ingredient.
Today, we share how we scale data mixing to thousands of sources for DNA and natural language pretraining of Omnii, our multimodal genome model.
Mayee Chen retweeted
Today, we’re excited to launch the Datology Curation Studio, customizable frontier data curation for every team training its own models.
Data quality is the ultimate compute multiplier, and Curation Studio makes it easy for everyone.
Mayee Chen retweeted
At @datologyai we run thousands of data experiments. Two things keep biting us:
- Same run on 16 vs 8 GPUs → different evals
- Every pipeline change → re-materialize the dataset
Introducing Zephon, the data loader we built to process data on the fly, deterministically. 🧵
Mayee Chen retweeted
Today we're unveiling Trillium Labs @trillium_labs, a new non-profit to foster the open science of frontier AI. We're building open post-training recipes and will expand into open infra to study RSI, reward-hacking, multi-agent systems, and whatever comes next.
We're built around the theory of change that you need more eyes to solve hard technical problems. We have faith in the scientific methods and communities that humanity has built, and worry that AI is becoming too closed to utilize them.
Trilliums are wildflowers that bloom briefly in the spring, before the forest canopies fill out. Though they are small, they lay the foundation for the cycles of growth and nourishment through the rest of the year. At Trillium Labs, the recipes will be the slow nutrients for the seasons and the model releases will be the blooms. Building an institution dedicated to this is needed because, much as nature’s trilliums are slow to expand and grow, the open-ecosystem needs time and dedicated resources to catch up.
I co-founded with with a long-time friend and collaborator Tom Zick (@thesezickbeats). We're hiring (full time + student collabs/interns), we're fundraising, and we're looking for compute. Please get in touch if you're interested in helping out. Offices based in the Bay Area and Cambridge MA, remote okay.
I’m in the Bay Area until for The Curve and COLM to connect with people who are interested. We’re thankful to have initial support from Halcyon Futures and Schmidt Sciences with more funding en route to enable our ambitions of scaling. Our advisors @Thom_Wolf, @HannaHajishirzi, @gneubig and @ctnzr have been instrumental to building the ecosystem that exists today, and I’m stoked to get to keep working with them.
Mayee Chen retweeted
Releasing PhantomEnvironments today! Fully synthetic RL environments that turn off-the-shelf LLMs into strong search agents: a 7B LLM trained with RL in PhantomEnvs performs like an agent 10x its size. You can generate environments at $0 cost, free of distillation, and fully open-source.
And we are just getting started: we won a large GPU compute grant from @NVIDIAAI to continue our work on synthetic data and self-improving agents. We'll be "on tour" in SF next week:
- @SnorkelAI Frontier Data Summit (Oct 8)
- Oral talk at the @COLM_conf LSEI workshop and poster at LLA workshop (Oct 9)
Drop me a message if you'd like to chat!
Paper: arxiv.org/abs/2609.40221
Code: github.com/kilian-group/phan… (coming soon; we're finishing a hero run on our new NVIDIA compute)
---
Agent training is bottlenecked by RL environments. Our bet: the biggest lever is data, and the way to scale it is synthetic environments.
Earlier this year we asked a simple question: can an LLM learn the generalizable skill of agentic search (decomposing a question, retrieving documents, composing knowledge) from environment interaction alone? And to push it to the extreme: can it learn this in worlds where the questions are templated ("Who is the mother of Alice's father?") and the Wikipedia is generated by rules?
Yes. Agents learn the skill and transfer to real-world search remarkably well. PhantomEnvs lets you scale that data axis in agent training for free.
PhantomEnvs is a milestone in my PhD, and the latest in a line of Phantom* projects. When I started the PhD two years ago, OpenAI o1 had just come out and Claude 3.x was getting its first tool-use abilities. I became obsessed with training these systems myself, squeezing every GPU I could find to understand what goes into building something so cool. I thought it would take 5 years to get a grip, but it's been 2.
- PhantomWiki: evaluating LLM reasoning in isolation from memorization
- PhantomReasoning: showing LLMs can be post-trained on synthetic data
- PhantomEnvironments: training LLM search agents on rule-generated synthetic RL environments
There's always another Phantom* project on the way... much to my lab's annoyance because of the number of phantom-* slack channels. RL and synthetic data are two deep gold mines, and we (are starting to) understand how to dig.
Joint work with Swathi Saravana Selvam, @GongAlbert, @ChaoWan0331, Christian Belardi, Dongyoung Go, @katielulula, and @KilianQW. Big thanks to @NVIDIAAI for the continued support!
Mayee Chen retweeted
AI agents now seem to solve the hardest problems in math and science, but they haven't yet deepened our understanding at the same rate. They are great problem-solving machines when the objective and optimization process are clear, but discovering insights is a lot fuzzier!
We build EurekaBench to measure this gap using 6 natural science domains (neuroscience, geophysics, plasma physics, astrophysics, computer science, and chemistry). We test if AI agents can complete the whole scientific discovery loop from understanding the observations, explaining the data by describing the mechanism (e.g., equations), and eventually discovering insights.
Results show that the agents don't advance on both improving accuracy and finding insights at the same pace! A lot more details in the thread. Fantastic work led by @JiayiiGeng.
AI scientist agents are great at optimizing and finding solutions through endless trial and error. But is that really what science is about?
As Terrance Tao eloquently put, the role of math and science should be more than that – they are "lighthouses" that guide and inspire exploration through understanding and insights. Can AI agents discover novel insights through experimentation?
To study these gaps, we worked with domain experts to introduce EurekaBench, a benchmark spanning 6 science domains that evaluates an agent’s ability to discover genuinely new scientific insights.
(1/n)
Mayee Chen retweeted
CLM is one of the most elegant ideas we've worked on together.
To me, it starts with a simple interaction need: there are many cases where I just want the agent to make surgical edits to the context. For example, when the agent finishes one task and prepares for another, it should ideally be able to clean up the context for the next one, just as a human would. And there are many such cases!
Instead of hard-coding such rules into the harness (I doubt that's even possible), what would a first-principles solution to this problem look like? It turns out to be shockingly simple: ask the LLM to manage the context itself! Not only can this help the agent more efficiently solve the next task, but it can also be applied recursively to many more complex cases.
This unlocks new agent capabilities across a variety of challenging, long-running, and open-ended tasks. For example, equipped with CLM, the agents can run continuously for hours on challenging tasks like EdgeBench or our new software world repository optimization tasks, and achieve really good results. Check out the CLM paper for more method details and many other cool results!
Mayee Chen retweeted
i was diagnosed with thyroid cancer many years ago purely by chance, because i had to get blood work for my US green card application (thank you, USCIS!). i was in my mid-30’s and was reasonably healthy (or so i believed.) this was shocking and totally unpredictable news to me. thanks to the amazing team of clinicians at NYU Langone, i successfully went through a total thyroidectomy followed by radioactive iodine therapy. but, it was truly the most uncertain and confusing period of my life.
this is when i first realized that no one can opt out of healthcare. being sick isn’t what you choose to or choose not to. it is something that unpredictably happens to all of us at some point for one reason or another. this is why it is a healthcare system that we collectively build and use to support each other at the society level. because it must support everyone, the complexity of creating, maintaining and improving this system is a Herculean effort that calls for all the help in the world.
today, i am announcing @OrtetAI together with my dear co-founders, @keunwoochoi, @HenriDwyer, @elmanmansimov, @causalclaudia and Jeff, to lend our support to improving the health of each and every one of us. Ortet’s goal is to create a technological foundation that will enable health systems to deliver predictably better care to each and every patient. this will only be possible by working closely with all stakeholders in the broader ecosystem, which is enabled in part by Ortet’s close partnership with Thoreau.
Ortet is taking the very first step of a long journey ahead, and at the end of this journey, i strongly believe we would be able to say that Ortet had contributed to building better, more predictable healthcare and thus to improving health of each and every one of us.
see ortet.ai/ for more!
Mayee Chen retweeted
There’s an enormous amount of open training data on Hugging Face. What does it take to make it work together 🤗?
For Marin’s 535B run, we built on 25T tokens from 152 datasets with licenses permitting training.
Here’s the work between downloading those and training a model 🧵
ALT Flow diagram of the Marin data pipeline. Starting with 25.25 trillion tokens, deduplication removes 2.13 trillion and benchmark decontamination removes another 13.66 billion, leaving 23.11 trillion. Documents are organized into 40 topics and five quality bands, yielding 200 sampling groups. Results from 872 small-model experiments feed a regression used to choose sampling weights.
Mayee Chen retweeted
I'm excited to announce @SnorkelAI's $350M Series E at $3.5B, led by @insightpartners and @S32_VC.
We've grown 18x+ in the last 12 months since launching our Data-as-a-Service offering, passing $375M ARRR this week.
As AI advances to superhuman capabilities, AI data & environment development must advance with it - and basic staffing and crowdsourcing approaches are not enough.
AI progress now requires deep research and technology work that combines human expertise with specialized AI in compounding ways. @SnorkelAI is building the RSI data engine and frontier data lab for this next phase.
We're honored to have the support of existing investors Addition, @lightspeedvp, @GreylockVC, @GVteam, P7, Factory, @WellsFargo, Walden Catalyst Ventures, and new investors @ThirdPointLLC, @MarchCPs, @BlumbergCapital, @AllegisCapital, @Frontlinevc, and @standard_vc.
–
@SnorkelAI started as a research project a decade ago at @StanfordAILab.
Our thesis was simple: AI progress would become increasingly data-centric – and therefore data development should be studied as a true research and technology problem, not just a staffing and crowdsourcing one.
Today, as AI capabilities verge on superhuman, building the data and environments to safely measure and train AI is becoming too hard for even the smartest human experts to do alone.
Only humans and AI agents, collaborating together in compounding ways, can meet the accelerating needs of the frontier, and keep humans in the driver’s seat of AI progress for decades to come.
At @SnorkelAI, we are building the data lab to define the shape of this new “Data 2.0” frontier, and the new paradigms of human-computer interaction needed to advance it.
Our key focus is building the RSI engine for data, where specialized AI models accelerate and improve human expert output, and in turn, scaled human supervision is used to continuously evaluate and improve these models – creating a powerful compounding loop to keep pace with an accelerating RSI frontier.
With this round of funding, we are also doubling down on our commitments to support data development for open benchmarking and evaluation (more news here soon!); an increasingly diverse ecosystem of general and specialized intelligence; and a path to safe, well-aligned AI built on robust training and evaluation data.
Data development will guide and drive the next stages of AI – and must do so in a human-centric, AI accelerated, open, diverse, and safe way. We are excited to support this mission in the next decade of research ahead at @SnorkelAI.
More thoughts here: snorkel.ai/blog/data-2-0-and…
Mayee Chen retweeted
Pretraining progress seems to be coming mostly from data improvements.
@who_is_jerbear and I pretrained combinations of year-representative open model recipes and data corpuses across 2019 to 2025 at various small scales.
Data improvements contributed 3.24x as many compute multipliers as model improvements did (12.0x vs 3.7x).
And the gains stack independently - a better dataset helps every architecture about equally, and vice versa.
Here are full results, plus what we think this means for the future of AI progress:
dwarkesh.com/p/pretraining-p…
Mayee Chen retweeted
Really delighted to share that I've been named one of the "Top 50 Legal Innovators in Academia" by @natlawreview and Wickard. It’s a privilege to be amongst friends, colleagues, and other individuals I’ve long looked up to!
natlawreview.com/article/202…
Mayee Chen retweeted
Introducing Tenet, our first model post-trained for legal.
Tenet is a Kimi K3 base that we post-trained with @FireworksAI_HQ on a corpus of publicly available legal data, synthetic data, and human expert data simulating long-horizon legal work.
Training increases Tenet's all-pass rate by 82% on LAB and 22% on LAB Contracts relative to the Kimi K3 base model. It achieves state-of-the-art performance on LAB Contracts and places second on LAB.
These gains generalize to other leading agentic benchmarks including @mercor's Apex Agents - Corporate Law, @crosbylegal's Redline Bench, and @scale_AI's Professional Reasoning Bench.
Tenet is also optimized for token efficiency, operating at less than a fourth the cost of leading foundation models.
We additionally post-trained three specialist models for Tenet to use as subagents:
1) M&A Diligence: post-trained with @baseten on our LAB Diligence environment in an RLM harness, this model is optimized for high-scale, long-horizon tasks.
2) Review Tables: trained with @appliedcompute on our Review Table environment, this model is state-of-the-art and cost-effective at high-volume document review and structured data extraction.
3) Firm Knowledge: trained with @EngramLab on our synthetic law firm environment, this model is optimized to learn and search over a firm's knowledge via memory and structured notes.
More details on model training, environment design, benchmarking, results, and more in the article by @gabepereyra below.
What's next for Harvey’s research?
- Scaling LAB to more jurisdictions, practice areas and workflows
- Scaling compute to bring new generalist models and capabilities to Harvey
More to come soon.
Mayee Chen retweeted
Awesome research by @engramlab using the synthetic law firm we built.
They trained a 27B Qwen model on synthetic client matters and used the learned knowledge to do online search more efficiently. The trained model outperforms frontier models at 10x lower cost per query.
The results are promising for scaling up enterprise search and personalization.
Mayee Chen retweeted
Really interesting result here from @EngramLab on @harvey's LAB Firm Knowledge dataset.
They introduce studying as a learning mechanism that both improves quality and reduces the need for extensive inference-time reasoning, leading to Pareto efficient search over really large knowledge bases.
Very bullish on leveraging upfront / offline compute to reduce online token use!
Awesome to collaborate with @dan_biderman @realJessyLin and the Engram team on building more efficient long-horizon agents.
