• We evaluated Anthropic’s new Claude Sonnet 5 across our benchmark suite. It comes in at #3 on the Vals Index (68.61%), behind only Claude Fable 5 (75.15%) and Claude Opus 4.8 (70.36%), and narrowly ahead of GPT 5.5 (67.95%).

• It’s a substantial generational step: +8.5 points on the Vals Index over Claude Sonnet 4.6 (60.07%) — and almost all of that gain comes from coding.

• The models main strength is in coding. Compared with Sonnet 4.6, Claude Sonnet 5 jumps +30.7 points on Vibe Code Bench (56.22% → 86.90%) and +17.2 points on Terminal-Bench 2.1 (57.30% → 74.53%), while SWE-bench Verified actually slips slightly (77.45% → 75.49%). The gains are concentrated in sustained, multi-step agentic coding rather than one-shot patch generation.

• Outside coding it’s a quieter step. Sonnet 5 posts 67.95% on CorpFin v2 (+1.3 vs Sonnet 4.6) and 51.98% on Finance Agent (+0.9). Its index score clears last-generation Claude Opus 4.7 (66.10%).

We noticed a small but noticeable number of refusals on CorpFin v2 — 15 of 858 tasks, mostly flagged as “bio.”

The model has a 1M-token context window. Evaluations were run with compute effort set to “max” on all benchmarks except Terminal-Bench 2.1 (run at “high”), 128k max output tokens, and default temperature and top-p. Terminal-Bench 2.1 was averaged over 3 runs (71.9%–78.7%).

Congrats to the Anthropic team on the release!

We’ve released CyberBench, a private benchmark that tests whether autonomous agents can reproduce and patch real OSS-Fuzz-style crashes in open-source projects. The benchmark has two tracks: PoC, where agents produce raw inputs that trigger vulnerable builds, and Patch, where agents edit source code to stop the crash while preserving behavior.

• GPT 5.5 leads overall. On the PoC track it scores 79.7%, showing the strongest ability to generate crashing inputs without being given the original crash.
• Open-weight models including GLM 5.2, MiniMax-M3, and Kimi K2.6 perform quite competitively; GLM 5.2, for instance, is second overall on the leaderboard.
• Patch is tightly clustered at the top, with Claude Opus 4.8, GPT 5.4 (xhigh), and GPT 5.5 tied at 81.4%.
• Provider refusals materially affect results. We count refusals in the authorized benchmark sandbox as measurable failures, and several models refused or filtered vulnerability-style tasks.

We’ve released Legal Research Bench, a private benchmark that tests whether agents can handle realistic US legal research across eight practice areas. Each task uses case-law search, web search, and document retrieval to answer a question, graded against rubrics authored and peer-reviewed by practicing lawyers.

• Claude Opus 4.8 leads at 43.75% all-pass, followed by GPT 5.5 (40.39%) and Claude Sonnet 4.6 (38.46%).
• GLM 5.2 is the top open-weight model at 31.25%, ahead of MiniMax-M3 (29.81%).
• GLM 5.2 is the cost-efficiency leader at $0.89 per test, versus $2.82 for Claude Opus 4.8 and $7.40 for GPT 5.5.

We evaluated models on Harvey’s Legal Agent Benchmark, which tests an agent’s ability to complete legal work. Each task asks an agent to answer a client inquiry using shell and file-editing tools, plus skills for Word, Excel, and PowerPoint.

• Claude Fable 5 ranks #1 at 11.25% with Claude Opus 4.8 fallback, followed by Claude Opus 4.8 at 9.58% and Claude Sonnet 4.6 at 5.00%. Without fallback, Fable 5 still ranks #1 at 10.42%.
• MiniMax-M3 is the top open-weight model at 4.17%, ahead of GPT 5.5 at 3.75%.
• Claude Fable 5 ran at $19.23 per test, compared with $10.22 for Claude Opus 4.8, $3.04 for Claude Sonnet 4.6, and $1.46 for MiniMax-M3.