Evaluation
BioMysteryBench is a benchmark developed by Anthropic to measure agentic biological analysis performance. Tasks are constructed by stripping real-world biological data of metadata (e.g. sample origin) and evaluating whether models can recover this information via computational analysis. Tasks may require agents to inspect files, run command-line tools, write analysis code, or query biological databases. Tasks cover areas including sequencing, expression analysis, variant calling, epigenomics, metagenomics, proteomics, and metabolomics.
Up to five domain experts attempted each problem, and the tasks solved by at least one expert were labeled human-solvable. Tasks that no expert solved were labeled human-difficult.
Tasks refused by biology content filters are counted as task failures for the purposes of evaluation.
BioMysteryBench was developed by Anthropic. Its task materials are released under CC BY 4.0 with restrictions against their use for model training, while the biological data remain subject to their original repositories’ policies.
Model behavior
We analyze failure modes using an openai/gpt-5.6-terra judge that reads only the final message from each scored-incorrect trajectory. The view shows each failure type as a share of that model’s incorrect attempts, including provider refusals and filters.
How incorrect attempts end
Share of each model's scored-incorrect task-trials
We find that models overwhelmingly choose to give confidently wrong answers instead of expressing uncertainty. OpenAI models are comparatively more likely to remain uncertain or end without a definitive answer. Kimi K3 is also prone to confidently wrong answers. This is directionally consistent with Kimi’s own report of a “noticeable gap in user experience” compared with other models.
How Opus outperformed on hard tasks
Which Ensembl IDs were knocked down? GENE1: samples 1, 3, 4, 11. GENE2: 7, 9, 10, 12. Controls: 2, 5, 6, 8.
What parent drug was administered to this patient, given the untargeted mass-spectrometry data?
Methodology
Models run with the Terminus 2 agent harness inside sandboxes configured for bioinformatics work. The environment includes common alignment, sequence-processing, quality-control, workflow, R/Bioconductor, and Python tooling. Network access is restricted to the scientific package repositories and biological databases required by the benchmark. Tasks were limited to 200 agent turns and four hours of wall-clock execution.
Each displayed score is the mean of three independent full-benchmark runs, and error bars show the standard error across those three run-level scores. Models use their registry-default reasoning or compute effort; the exact settings used for each row are available from the leaderboard’s settings control.
Each task is evaluated from the agent’s submitted answer by a separately configured judge (we use openai/gpt-5.6-terra). The judge is additionally instructed to flag when models arrive at the answer via cheating, though we don’t observe this in our environment. Task scores are binary: a fully accepted answer scores 1, while judge-rejected answers and model/provider failures or refusals score 0.
Dataset Provider
