Harvey's Legal Agent Benchmark

Partners in Evaluation

Key Takeaways

Muse Spark 1.2 is the current top model by Harvey final score at 25.42%, followed by Muse Spark 1.1 at 20.00% and Grok 4.6 at 15.83%.
Kimi K3 is the strongest open-weight model in this run at 10.83%, while DeepSeek V4 matches GPT 5.5 on final score at 3.75%.
Top models satisfy most individual criteria, with criteria pass rates around 90%.

Benchmark

The Legal Agent Benchmark is a benchmark recently released by Harvey to test the ability of models to support legal work in an agentic setting. There are two datasets as of today: the public set and the held-out test set. The results here are from the held-out set, and initial results have already been released by Harvey.

Each task asks an agent to produce legal work against a set of task-specific criteria. The agent is provided with six tools: Read File, Edit File, Write File, Glob, Bash, and Grep. It also has three skills: docx, pptx, and xlsx.

Reported results use the same methodology as Harvey’s initial leaderboard. Criteria pass rate is included to show how often models satisfy individual requirements, even when they do not fully resolve the task.

Results

Among models with a nonzero final score, Ling 3.0 Flash, DeepSeek V4 Flash 0731, Muse Spark 1.1, and Muse Spark 1.2 form the cost/performance frontier.

Muse Spark 1.2 leads on the overall Harvey final score at 25.42%, with Muse Spark 1.1 second at 20.00% and Grok 4.6 third at 15.83%. Claude Fable 5, fifth at 11.25%, fell back to Claude Opus 4.8 on 4 tasks; counting those as failures gives a no-fallback score of 10.42%. The criteria pass rates are much higher: Muse Spark 1.2 reaches 94.52%, Muse Spark 1.1 reaches 92.86%, and Grok 4.6 reaches 92.52%.

Criteria Pass Rate by Task TypePercent of criteria passed per task type

Task type	Muse Spark 1.294.52% avg	Muse Spark 1.192.86% avg	Grok 4.692.52% avg	Kimi K390.79% avg	Grok 4.590.55% avg
Intellectual Property	95.8%	95.4%	94.7%	95.1%	95.1%
Corporate M&A	95.8%	96.2%	95.2%	93.3%	95.2%
Banking Finance	97.8%	94.8%	94.8%	92.2%	95.5%
Data Privacy/Cybersecurity	95.8%	93.7%	97.9%	94.0%	95.1%
Trusts & Estates/Private Client	93.5%	91.6%	92.7%	89.7%	90.8%
Capital Markets	96.9%	92.5%	94.1%	92.2%	94.5%
Corporate Governance	96.2%	96.9%	93.2%	92.5%	90.8%
International Trade Sanctions	93.1%	90.0%	93.3%	92.1%	89.7%
Real Estate	96.5%	95.6%	95.0%	94.8%	93.6%
Energy/Natural Resources	98.7%	96.4%	96.7%	96.7%	95.0%

0%99% criteria passed

The leaderboard can be filtered by task type and switched between task pass rate and criteria pass rate. Models perform best on task resolution in Energy/Natural Resources and Healthcare/Life Sciences, while criteria pass rates averaged across all models are highest in Intellectual Property, Corporate M&A, and Data Privacy/Cybersecurity.

Criteria Pass Rate vs. Task Resolution

CRITERIA PASS RATETASK RESOLUTION

Muse Spark 1.2

94.5%/25.4%

Muse Spark 1.1

92.9%/20.0%

Grok 4.6

92.5%/15.8%

Kimi K3

90.8%/10.8%

Grok 4.5

90.5%/12.9%

Claude Fable 5

90.5%/11.3%

Qwen 3.8 Max

90.0%/10.4%

DeepSeek V4 Flash 0731

88.8%/8.3%

Harvey grades a task as resolved only if every criterion passes. A model can satisfy most individual criteria and still miss task resolution credit.

There is a clear trend: strong models and agents satisfy most criteria, around 90% for top models. The remaining gaps are large enough that task resolution stays low even when criterion-level performance looks strong.

Tool Calling Statistics5/52 models

Average tool count per item, across the six tools in the benchmark harness.

Models heavily prefer Bash and Read File. Write File appears regularly for some models, while Edit File, Glob, and Grep are lower-volume.

The available skills rely on shell commands in their instructions and to run their scripts. The tools also often encourage models to read files through the harness.

The skills do not directly emphasize Edit File, Write File, Glob, or Grep. Those tools still appear in traces, but less consistently than Bash and Read File. Grep is not well-suited to the binary format of .docx, .pptx, and .xlsx files. Likewise, Edit is useful for text-based files such as .md files, not for these filetypes.

Skill Invocation Statistics5/52 models

Average skill invocations per item, across the three skills available to the agent.

Skill usage is dominated by docx, followed by xlsx. pptx is used less often, but it is not absent.

Methodology

We use Harvey’s generation and grading protocol in the same environment, with internet access disabled.

Harvey grades each submission with two LLM judges. Each judge computes a task pass rate. A task passes only if 100% of its criteria pass, and Harvey’s final score is the average of the two judge task pass rates.

The two judges were GPT 5.5 and Claude Sonnet 4.6. GPT 5.5 used medium reasoning and Claude Sonnet 4.6 was not modified.

While running the benchmark, we found that redline criteria needed DOCX tracked changes preserved when reading submitted files so judges could see inserted and deleted text. We fixed that bug and merged it upstream in harveyai/harvey-labs#76. The scoring rubric is unchanged.

The benchmark was modified to use our model library, an abstraction over various LLM provider APIs, and to run on Valkyrie, our framework for running agentic benchmarks. These are infrastructure changes and do not impact model performance.

To improve judge performance and reduce cost, we split the instruction prompt provided to each judge so common elements could be cached. This did not modify prompt content outside of caching.