data lab to train frontier models & evaluate agents
San Francisco, CA
Joined August 2022
- Tweets1.2K
- Following0
- Followers21.5K
- Likes3.1K
Today we’re announcing the micro1 Medical Index: a suite of medical benchmarks built and graded by working doctors, biostatisticians, regulatory experts, and psychiatrists.
It brings all of our medical research into one place, with a combined score across models and a living leaderboard that grows as we publish new benchmarks. We’re starting with three, each built around a fundamental question:
- RealmPathology Benchmark: can a model reason over a real pathology report, staying exact on what it says, within its diagnostic limits, and without escalating beyond what the evidence supports?
- PharmaBench: can a model work through the regulatory and clinical documentation behind an FDA drug program with the precision a clinical development team would expect?
- WellnessBench: when a young person brings a sensitive health question, can a model stay safe without shutting the conversation down, and hold that line under pushback?
Together, these three benchmarks cover a meaningful range of medical work, from diagnostic reasoning to pharmaceutical research and patient safety. While they don't yet capture the full scope of the entire medical industry, they establish a strong starting point that will become more comprehensive as we expand the index.
Huge shoutout to every expert who built and graded these tasks. Full reports linked in the comments below!
The most advanced AI models still get nearly half of expert contract review judgments wrong.
We partnered with @heyivoai to examine why, and where even the most advanced models fall short when reviewing and negotiating real contracts.
Introducing the Ivo-micro1 ContractBench, a benchmark built by practicing attorneys to test how well AI handles the judgment calls that come with reviewing a contract, not just whether it can draft.
No model scored above 59%. Models did well on the straightforward calls, like leaving acceptable language alone or applying a clear playbook rule, but they struggled most on the issues attorneys considered highest priority. Knowing what to push back on, what to leave alone, and when to escalate proved challenging for the models evaluated.
Explore the full results and methodology below.
Introducing the Ivo-micro1 ContractBench: a new benchmark from Ivo and @micro1_ai for measuring the judgment behind contract review.
Most legal AI benchmarks focus on the edits a model makes. But an effective review requires more than knowing what to change. It means understanding things like what to leave alone, when to adapt a standard position to the deal, and when to escalate an issue to a lawyer.
ContractBench evaluates these decisions across 350 tasks, 35 scenarios, and five agreement types. We tested seven frontier models against five dimensions of contract-review judgment.
What we found:
→ Restraint is the strongest skill across every model tested.
→ Models struggle to reconcile standard playbook rules with deal-specific context.
→ Escalation remains a major gap, even when models have clear approval boundaries.
As AI takes on more of the review process, we need to measure not just whether a model can make an edit, but whether it can make the right decision.
We’re publishing our methodology, results, and public dataset to provide a blueprint for how to improve models for contract work.
Explore ContractBench: ivo.ai/research/ivo-micro1-c…
Read the full findings: ivo.ai/blog/ivo-micro1-contr…
Models have become great at sounding and writing like attorneys, but the gap remains in reasoning like one, and sustaining that reasoning across an entire matter.
View the full report: micro1.ai/benchmark/legal-re…
Designed to mirror real legal work, each matter unfolds over time as new facts and filings emerge. Models must build on their earlier analysis, adapt to new information, and research the law themselves when necessary.
Across 8 models and 1,200 full runs, Gemini 4 Argon scored highest at 50.8%, followed closely by Claude Fable 5.1 at 50.6%. No model achieved all-pass on any matter.
Every model was strongest at identifying the issue and weakest at stating and applying the rule, especially when it had to research the law itself.
AI models can be as convincing as a lawyer, but can they complete the complex multi-step reasoning a real legal matter requires?
Today we're excited to introduce LegalReasonBench, a new benchmark to explore this question and test models on realistic, long-horizon legal matters authored by practicing attorneys.
More info in thread 🧵
We’re sharing more information about flow qc, our system for reviewing the tasks used to train and evaluate frontier models across domains, with multimodal support including audio and video.
A task can ask for the right work and still reward the wrong behavior. Grading rules can reject valid answers, miss real mistakes, or leave room for reward hacking. flow qc red teams tasks, grading rules, and model rollouts to investigate these issues before human review, providing experts with evidence for each finding and a proposed fix.
Our latest article shares how we built this process and what we’ve learned from running it in production.
Read more below.
micro1.ai/blog/flow-qc
As agents evolve, evaluations need to keep up. Knowing whether an agent is improving requires consistent measurement and evidence that progress carries through to real-world performance.
Our Economic Research Unit explores this in its first paper. Learn more:
Most companies can’t determine whether their AI agents are genuinely improving and our findings reveal why. The Economics Research Unit (ERU) at @micro1_ai , which I am privileged to lead, has released its inaugural paper, co-authored with @liuzhang11 : “Reliability of AI Agents: Rater Effects, Drift, and the Return to an Evaluation Program.”
Key Insights from 2,611 Real Production Interviews
By tracking a live AI interviewing agent in production, we uncovered two major dynamics:
1) Your evaluation metrics may reflect reviewer bias rather than AI performance. The identity of the reviewer was a stronger predictor of scores than the timing of the interview. In fact, a single overly strict evaluator masked a genuinely improving agent, making it appear stagnant. Once adjusted, reliability metrics increased by half a standard deviation.
2) AI evaluations go stale after roughly five weeks. Performance drifted even without code updates or new deployments. Without ongoing testing, after five weeks you lose the ability to detect a change as significant as your largest release. A one-time evaluation tells you about the agent you had, not the agent you are running.
Agents drift, so evaluation cannot be a launch-day event. It has to run continuously for as long as the agent is live. That is the principle behind Cortex, micro1's reliability layer for agents in production, and this paper shows why it matters.
Measuring AI agents is not a mere compliance checkbox; it serves as core economic infrastructure. This research marks just the beginning for the ERU, with further studies on the horizon.
micro1.ai/research/are-ai-ag…
We recently observed a frontier model repeatedly trying to hack its way towards the right answer, displaying misaligned behavior that optimizes for cheating.
Gemini 3.8 Flash searched its sandbox for hidden grading files. micro1’s flow qc detected this reward hacking behavior before the task was published.
GPT-6.1 Sol, GPT-6 Astra, Claude Opus 5.5 and Grok 4.7 never did this.
While the leak was open, Gemini 3.8 Flash read the grading files in 32 of 245 rollouts.
The other models in the same environment:
- GPT-6.1 Sol: 0 of 394
- Claude Opus 5.5: 0 of 224
- Grok 4.7: 0 of 109
- GPT-6 Astra: 0 of 103
With today’s public release of Gemini 4, we rebuilt the same leak on the same task and ran it 50 times. Gemini 4 read the grading files in 0 of 50 rollouts.
As models get better at recognizing when they’re being evaluated, data quality matters more than ever.
Bad tasks don’t just produce bad scores. They produce models that are misaligned & optimize for cheating, which poses safety risks as we put more trust into these systems.
On October 30th, we’re kicking off the micro1 Eval Hackathon with a $10,000 prize pool!
The challenge:
Pick a task in your field that requires expert judgment, challenge a frontier AI model with tough, realistic scenarios to uncover its mistakes, then build a repeatable test in code with clear criteria for scoring the model’s answers.
Bring the edge cases and judgment calls you encounter across software, finance, healthcare, law, science, engineering, or any field you know deeply.
The hackathon is fully virtual and free to enter worldwide, with a $10k prize pool.
🥇 First place: $5,000
🥈 Second place: $3,000
🥉 Third place: $2,000
Compete solo, with coding skills or proficiency in AI coding tools. Open to experts with at least two years of professional experience.
Free registration is now open. Join us via the link in replies.
We’re hiring across every function from research and engineering to ops, sales, and marketing.
Join us: micro1.ai/careers
about 3 years ago, micro1 lost 60% of its revenue overnight and almost went bankrupt.
today, i'm proud to share we've surpassed $1B annual run rate while remaining historically profitable.
it’s been a special journey. we’re grateful for every struggle, moment, and person who has been part of it.
we've taken a very untraditional path to get here. our core team is only ~180 people, spanning 14+ countries, and fully remote.
many of our decisions sacrifice hundreds of millions of revenue and slow us down short term. but we believe they compound into advantages that are impossible to replicate long term.
the long term is now.
these are the 8 principles that made micro1 a leader in the data industry and will soon make us number one:
1.) human 1st AI: treat the tens of thousands of experts helping train models with absolute best-in-class care. our goal is to make AI training one of the largest, highest paying, and most enjoyable job sectors in the world.
2.) U.S. first: we refuse to work with adversarial nations, or export data that improves models against the interests of Americans.
3.) long-term research over short-term ARR: invest in deep research, products, and infrastructure for years from now, not just the next customer pipeline.
4.) proper incentives for intensely hard work: no arbitrary requirements like 996. inspire people to work extremely hard with outlier incentive structures instead of forcing weekend work. we’ve maintained close to 100% core team retention as a result.
5.) stay lean and global: we’ve reached $1B run rate with ~180 people, and intend to slow team growth significantly from here. exceptional talent is found around the world, and pedigree is the wrong thing to over-index on.
6.) execution alignment: company to retain a streamlined board so we can optimize for the next several decades, remain light-weight and dynamic, and take bold & calculated risks.
7.) abundant intelligence improvement: invest in building data generation models to make high-quality datasets and RL environments dramatically less expensive and more abundant.
8.) scale data on realism and horizon: realism brings the complexity of the real world into training models, while horizon extends how long models can work and how much complexity they can handle. both have enormous room to scale, and we’re pushing the limits of both.
additionally: after surpassing $500M run rate this summer, we priced our Series B at a $4B valuation, alongside a ~$25M tender to give our early employees liquidity as we build for the long term. the round was led by Dream Ventures, a longtime supporter and one of our earliest investors, alongside Microsoft, Javelin, Flybridge, G Squared, Up Partners, and LG.
we’re also grateful to be backed by incredible industry leaders, including Guodong Zhang (co-founder of xAI), Andy Fang (CTO of DoorDash), Igor Babuschkin (co-founder of xAI), Behnam Neyshabur (founder of Mirendil), Jerry Tworek (ex-VP of Research at OpenAI), and many more.
data is on its way to becoming the largest industry humanity has ever seen, and we're certain this is just the beginning.
best,
Ali
micro1 retweeted
We have two additions to PersonalAgentBench: @hark_labs and @OpenAI's Dots!
Dots has taken the lead, with Hark close behind while Gemini Spark moves to third, followed by Instinct, Grok Bot, and Muse.
We also doubled the number of tasks this week, adding research, image understanding, editing, and everyday communication. The wider set changed the picture. Gemini Spark was strongest on research but weaker on image and editing tasks. Hark and Gemini tied on tasks involving user authority. Dots was the most consistent across task types, although it was weaker when it had to infer personal context. No assistant led every category; the overall ranking rewarded consistent performance across the full set.
This is still an early snapshot, and we’re excited to see how results evolve as we add more agents, expand the tasks, and re-evaluate after product updates.
If you’re using any of these six personal agents, follow along by visiting the link in the original post.
Which personal agent is the best life assistant?
Today we're releasing the micro1 PersonalAgentBench to help answer that, and we're paying real personal agent users to contribute to the benchmark.
Personal agents can now book your flights, answer emails, and move your meetings around. That’s great, until one messages the wrong person or makes a purchase without confirming.
Capability is not the same as trust. So we put four personal agents to the test: Instinct, Muse, Grok Bot, and Gemini Spark. We found that even when agents found the right information, they still often miss what matters, overshare, or make things up.
To find out for yourself how your agent compares, you can join our benchmark. We’re paying the first 100 users to run our prompts and share their results.
We're committing to spend $1B on enterprise data acquisition over the next 12 months. Learn more:
Today micro1 is announcing a commitment to spend $1B over the next 12 months on the acquisition & licensing of enterprise data for AI training, with financial backing from Citi and Hercules.
micro1.ai/blog/micro1-commit…
