We’ve released the Excel Modeling Benchmark (EMB), a private benchmark that tests whether agents can build the complex Excel financial models used in investment banking and private equity — LBOs, DCFs, M&A, and more — graded against expert-authored gold models.

• Claude Opus 4.8 leads at 69.4%, ahead of Claude Sonnet 5 (66.3%) and GPT 5.5 (64.5%).
• Numerical accuracy is the bottleneck: the leading model passes 87% of formula checks but only 61% of number checks — structurally sound models whose computed values drift from the reference.
• Claude Opus 4.8 is the most accurate at ~$12 per task, while Claude Sonnet 5 ranks second but is the priciest of all at $15.44; MiMo V2.5 Pro stays above 50% at just $0.22.

• We evaluated Anthropic’s new Claude Sonnet 5 across our benchmark suite. It comes in at #3 on the Vals Index (68.61%), behind only Claude Fable 5 (75.15%) and Claude Opus 4.8 (70.36%), and narrowly ahead of GPT 5.5 (67.95%).

• It’s a substantial generational step: +8.5 points on the Vals Index over Claude Sonnet 4.6 (60.07%) — and almost all of that gain comes from coding.

• The models main strength is in coding. Compared with Sonnet 4.6, Claude Sonnet 5 jumps +30.7 points on Vibe Code Bench (56.22% → 86.90%) and +17.2 points on Terminal-Bench 2.1 (57.30% → 74.53%), while SWE-bench Verified actually slips slightly (77.45% → 75.49%). The gains are concentrated in sustained, multi-step agentic coding rather than one-shot patch generation.

• Outside coding it’s a quieter step. Sonnet 5 posts 67.95% on CorpFin v2 (+1.3 vs Sonnet 4.6) and 51.98% on Finance Agent (+0.9). Its index score clears last-generation Claude Opus 4.7 (66.10%).

We noticed a small but noticeable number of refusals on CorpFin v2 — 15 of 858 tasks, mostly flagged as “bio.”

The model has a 1M-token context window. Evaluations were run with compute effort set to “max” on all benchmarks except Terminal-Bench 2.1 (run at “high”), 128k max output tokens, and default temperature and top-p. Terminal-Bench 2.1 was averaged over 3 runs (71.9%–78.7%).

Congrats to the Anthropic team on the release!

We’ve released CyberBench, a private benchmark that tests whether autonomous agents can reproduce and patch real OSS-Fuzz-style crashes in open-source projects. The benchmark has two tracks: PoC, where agents produce raw inputs that trigger vulnerable builds, and Patch, where agents edit source code to stop the crash while preserving behavior.

• GPT 5.5 leads overall. On the PoC track it scores 79.7%, showing the strongest ability to generate crashing inputs without being given the original crash.
• Open-weight models including GLM 5.2, MiniMax-M3, and Kimi K2.6 perform quite competitively; GLM 5.2, for instance, is second overall on the leaderboard.
• Patch is tightly clustered at the top, with Claude Opus 4.8, GPT 5.4 (xhigh), and GPT 5.5 tied at 81.4%.
• Provider refusals materially affect results. We count refusals in the authorized benchmark sandbox as measurable failures, and several models refused or filtered vulnerability-style tasks.

We’ve released Legal Research Bench, a private benchmark that tests whether agents can handle realistic US legal research across eight practice areas. Each task uses case-law search, web search, and document retrieval to answer a question, graded against rubrics authored and peer-reviewed by practicing lawyers.

• Claude Opus 4.8 leads at 43.75% all-pass, followed by GPT 5.5 (40.39%) and Claude Sonnet 4.6 (38.46%).
• GLM 5.2 is the top open-weight model at 31.25%, ahead of MiniMax-M3 (29.81%).
• GLM 5.2 is the cost-efficiency leader at $0.89 per test, versus $2.82 for Claude Opus 4.8 and $7.40 for GPT 5.5.