Our current benchmarks focus on domain-specific bioinformatics, therapeutics capabilities or biosecurity task categories (you can find the full herd at benchmarks.bio). While domain-specific benchmarks provide rigorous treatment of complex areas of science and help organize failure modes, we believe that we have now measured enough data types to warrant a meta-benchmark useful as a high-level barometer on model progress. We introduce OmicsBench v1, a collection of 784 tasks from eight benchmarks whose primary concern is the analysis of molecular (“omics”) data: SpatialBench, ScBench, EpiBench, VariantBench, BioSecBench-Surveillance, BioSecBench-Function, SpatialBench-Long, and ScBench-Long. Chris wrote an awesome essay below:

We’ve released quite a few frontier biology benchmarks this year. While focusing on individual scientific domains provides rigorous treatment of complex task categories, we have now covered enough ground to take a broader view of model progress. Here we introduce OmicsBench v1: 784 tasks from eight prior benchmarks focused on analyzing molecular data, covering spatial omics, single-cell transcriptomics, epigenomics, variant analysis, microbial surveillance and protein function. Domain experts constructed these tasks by analyzing published experimental datasets, reproducing reported results or generating new findings. The collection includes 741 short-horizon tasks covering individual analysis steps and 43 long-horizon tasks covering complete workflows. For this release, we added further human verification, revised prompts and research environments, disabled internet access, and removed additional identifying information from datasets. We tested 31 model-harness combinations. The strongest short-horizon configurations from xAI, OpenAI and Anthropic scored similarly: Grok 4.7 at 49.4%, GPT-6 Astra at 49.3% and Opus 5 at 47.7%, with overlapping confidence intervals. GPT-5.6 Sol with Codex led on long-horizon tasks at 32.8%. Native harnesses gave no large advantage on average. Agents have gotten better at omics analysis. Most of the improvement we measured came between February and July, with smaller gains in subsequent releases. No later model we tested exceeded GPT-5.6 Sol on long-horizon tasks. Agents also continue to make mistakes. They apply conventions from one assay to another, choose inappropriate scales for cellular neighborhoods, and mistake background microbial sequences for signal. Across domains, we see agents picking thresholds without validation, ignoring their own results and inaccurately reimplementing tools. We also see attempts to identify the studies behind sanitized datasets, retrieve benchmark answers and reverse-engineer the outputs a grader expects. Combining these benchmarks has given us a chance to revisit earlier tasks with more capable models and look closely at how their behavior has changed. It has also raised questions we want to explore: 1/ Models may solve problems differently from humans. Is it desirable for models to memorize biological studies and reference information, such as genomic coordinates? 2/ When we cannot see a model’s full thinking trace, how often are we missing attempts to game the task? 3/ How can we design evaluations that require models to provide interpretable scientific traces? 4/ How can we reduce contamination while retaining the real-world relevance of our data? We are excited to try benchmark construction ideas from other efforts, including GeneBench-Pro, Terminal-Bench Science, RareBench and BioMysteryBench. Benchmarking + evaluation is a rapidly evolving and challenging category of engineering. Biology provides a rich landscape of knowledge and its own interesting challenges for this work. With wet lab verification and drug development programs in the pipeline, we’re excited to bring these benchmarks closer to doing science. The future of biology comes from engineers and scientists working closely together. More to come.

We published results for Claude through LSVP on our therapeutics benchmarks. Considerable uplift, especially for Opus 5.5, on tasks drawn from small molecule, antibody and oligonucleotide development programs. benchmarks.bio/txbench-oligo

Very unique time in biotech history. Come sit down and talk shop. Drinks + good food with engineers, drug hunters and veteran scientists.

Biology lags code by about a year. But there are strong + unintuitive technical parallels.

Hiring for Members of Technical Staff to work on rigorous evaluation of AI in biology. Our mission is to advance humanity's ability to engineer biology. We approach this by understanding and improving AI’s application to biology across basic research, drug discovery, and biosecurity. We are a team of engineers, scientists, and operations specialists who are driven by improving capabilities and interpretability. Some of our work is showcased on benchmarks.bio. About this role You’ll turn your understanding of a biological domain into a roadmap of benchmarks and design them with engineers. Then build and manage your own scientific team using your network and internal talent infrastructure. You have (1) deep scientific expertise and experience using AI tools (2) or strong coursework in math/CS and experience working with scientists This is a high-intensity, founder-type role. Our Members of Technical Staff have built companies, run drug programs and scaled complex technical projects. Projects you might work on - Design benchmarks with engineers and scientists, from selecting scientific questions and datasets to developing reliable graders. - Work with engineers on large-scale computing and AI inference infrastructure to evaluate models and investigate their failures. - Research new methods for evaluating scientific capabilities and measuring improvements from training. - Work with our internal data generation and drug development teams to build tasks around real experiments and research decisions. How we work Our MTS team works in person in Mission Bay, San Francisco. We keep meetings to a minimum and are flexible about when you arrive or leave. Tight deadlines and customer commitments sometimes require late hours. Our work in use Frontier AI labs use our benchmarks to assess biological capabilities and safeguards. Our work appears in research announcements and model cards, including: - anthropic.com/claude-fable-5… - anthropic.com/claude-opus-5-… - anthropic.com/news/healthcar… - media.x.ai/v1/website/card4p… - x.ai/news/biosafety-at-the-f… Hiring Process Our process is typically completed within two weeks. - Round 1: Introduction - Round 2: Technical project matching - Round 3: Take-home evaluation project (~2 hours) - Round 4: Technical interview / live engineering session (1 hour) - Round 5: Paid onsite project and presentation (2–3 days) - Offer Compensation & Benefits - 350K-500K+ OTE - Equity - Unlimited PTO - Health, dental and vision insurance fully covered - Visa sponsorship and relocation assistance - 401(k) - Office in Mission Bay, San Francisco - Lunch and dinner provided Apply at jobs.ashbyhq.com/latchbio/9b…. Or send an email to kenny[at]latch[dot]bio

We introduce OncologyBench, a verifiable benchmark of 25 multistep oncology evaluations that tests the ability of multi-agent systems to produce clinically relevant decisions in cancer. 25 evaluations, built on 92 component tasks, span six stages of an oncology journey: (1) Translational and basic research (2) Risk and detection (3) Diagnosis and tumor characterization (4) Treatment (5) Response and resistance (6) Outcome Evaluations draw on published datasets, case studies, public databases, and patient data graciously donated to Latch for the express purpose of benchmarking. Clinical and translational decisions in oncology are rarely made by one person working from one dataset. They are built on several analyses, each carried out by a specialist on a given data type, whose reports are then weighed by someone who does not typically repeat the underlying work. OncologyBench is built to benchmark agents working in that setting. Agents received experimental data and worked without internet access. The strongest of the 31 model-harness combinations we tested were Claude Opus 5, Opus 5.5 and Sonnet 5.5, passing 29 to 40% of attempts, followed by Kimi K3 at 24%. We found that the models with the strong interpretation and analysis skills still showed weakness in reasoning. Two-thirds of the attempts that got every analysis right still failed the final decision. In a separate experiment we found decision and answer quality varied depending on the persona of the question asker. An example evaluation: A 76-year-old woman has undergone resection of stage II mucinous cecal cancer. The evaluation asks whether she should receive adjuvant chemotherapy. Five specialist agents analyze her tumor mutations, pharmacogenomics, gene expression, T-cell infiltration and clinical record. A decision agent receives their reports without access to the underlying data. The tumor is MSI-high, which predicts no benefit from 5-FU monotherapy. CDX2 loss and low T-cell infiltration do not establish that chemotherapy would improve her outcome. DPYD*2A is detected in 6 of 58 tumor-sequencing reads, but these data neither confirm nor exclude an inherited variant associated with severe 5-FU toxicity. BRAF V600E and MLH1 silencing support a sporadic origin. The expected recommendation is observation, with missing pathology requested and germline DPYD testing before any future 5-FU. --- Incredible work from researchers + former biotech founders at Latch. Gratitude to the patients that donated their data. More to come.

See young engineers questioning assumptions, building physical infrastructure in the real world and learning. Perhaps re-learning things biologists with decades of experience find trivial. But also questioning CRO and equipment prices/timelines, outfitting an empty room, building their own plate readers, and understanding data generating processes by doing the work. There is a lot of good intent here. There are legitimate criticisms of the claims and lab practices. We should maintain the highest bar for knowledge creation, safety and technical communication while encouraging scrappy teams to try new things. Let them fail technically, get humbled by the physical world, then encourage them to try something else. The history of technology teaches us that interesting discoveries and processes can come from projects like these. Many of you might be their heroes. Your publications may have encouraged them to try this. The antagonistic framing of “SV tech bro” fighting “biologist” that always pops up on the timeline is a terrible way to think about a field where progress depends on scientists and engineers working in lockstep on hard problems. There are real issues with biotech. I’ve watched seasoned academics and expert drug hunters burn billions of venture capital within the existing biotech formation structure. Expertise is essential, but conventional thinking deserves scrutiny too. We need new ways of thinking about engineering biology, and room for people to try them.

We introduce MetagenomicsBench, a benchmark for testing whether AI agents can analyze microbial communities and make scientific decisions from experimental data. 100 evaluations span five domains: (1) community structure (2) host-microbiome associations (3) microbial function (4) longitudinal dynamics (5) microbiome interventions Tasks draw on published datasets from human, animal, and environmental microbiomes. Metagenomic analysis requires interpreting microbial abundance alongside functional potential, host biology, and experimental design. Sequencing depth, taxonomic resolution, and confounding all affect observed results. Some concrete examples: - A higher abundance of antibiotic-resistance genes may reflect a change in community composition rather than a greater burden of acquired resistance. - A decline in one annotated butyrate pathway may not imply a loss of overall butyrate-synthesis capacity. - An association between bacterial abundance and reduced host-cell proliferation may reflect tissue context rather than a direct microbial effect. Agents received experimental data and worked without internet access. Across 34 model–harness configurations, Claude Sonnet 5.5 achieved the highest mean pass rate at 60.0% with both Claude Code and Pi, followed by Claude Opus 5 with Pi at 54.0%. GPT-6 Astra with OpenAI Codex was the highest-scoring non-Anthropic configuration at 49.7%. Performance also varied by scientific question. Host-microbiome associations were generally harder for the Opus and GPT families, while longitudinal dynamics and microbial function had higher pass rates across model families. Across judged failures, 47.1% involved unverified assumptions about task interpretation, thresholds, or metadata. Agents often produced plausible analyses while making choices that changed correct end biological conclusions. Manuscript: latch.bio/metagenomicsbench GH: github.com/latchbio/metageno… Blog: blog.latch.bio/p/metagenomic…

Biology benchmarks must evolve with agent capabilities and behavior. We updated benchmarks.bio after observing agents investigate benchmark identities, browse unrelated material and supply fabricated API contact details. The update includes: - Restricted or disabled internet access. - Stronger anonymization of datasets, files and tasks. - More natural prompts and research environments, with fewer evaluation cues and artificial completion requirements. - Live trajectory monitoring to flag unexpected tool use and behavior. Scores fell by 3.3 percentage points per benchmark on average, with the largest decline in EpiBench at roughly 15 points. The changes also removed useful web access, so these declines cannot be attributed entirely to reduced shortcut use. We have not observed substantial misaligned behavior in the updated runs so far. We will continue auditing trajectories and revising tasks, graders and sandboxes to keep our benchmarks focused on scientific reasoning.