• We evaluated Anthropic’s new Claude Opus 4.8 across our benchmark suite. It is the new #1 model on the Vals Index (70.17%) and the Vals Multimodal Index (70.71%), edging out GPT 5.5 on both.

• Agentic work is the headline. The model is #1 on SWE-bench Verified (88.60%), #1 on Vibe Code Bench (82.72%), and #1 on ProofBench (69.00%). It also places #2 on Terminal Bench 2 (70.04%) and #2 on Finance Agent v2 (53.92%).

• Knowledge benchmarks land near the top: LCB 87.82% (#3), GPQA 92.42% (#4), MMLU Pro 89.58% (#4), and MMMU 86.59% (#9).

• Domain-specific benchmarks hold up: MortgageTax 69.91% (#2), MedCode 53.22% (#5), MedScribe 85.75% (#6), TaxEval v2 75.63% (#7), and CorpFin v2 66.71% (#8). Claude Opus 4.8 also picks up #3 on Sage (54.79%).

• The model still lags behind the field on LegalBench, where it ranks #27 at 83.57%.

The model has a 1M-token context window. Evaluations were run with compute effort set to “max”, a temperature of 1.0, and max output tokens set to 128k.

Congrats to the Anthropic team on the release!

We’ve upgraded our Terminal-Bench evaluation to version 2.1, a verified iteration of 2.0. Tasks were cleaned up for environment bugs, flaky timeouts, and reward hacking to make the benchmark a more reliable baseline. Task count and structure are unchanged.

Here are some key takeaways from our results:

• GPT 5.5 leads at 76.40%, edging out Gemini 3.5 Flash by 2.2 points. Both come in under $1 per test.
• Gemini 3.1 Pro Preview (02/26) is the most cost-effective top-5 model at $0.58 per test — less than half the cost of the Claude models.
• Open-source models like GLM 5.1 and Kimi K2.6 remain competitive at a fraction of the cost.

• We evaluated Alibaba’s new Qwen 3.7 Max on the Vals Index, where it places #5 with a score of 57.29%.

• On SWE-bench Verified, Qwen 3.7 Max scores 68.8% overall and 66.7% on our Vals Index subset.

• On CorpFin v2, the model scores 65.4% on the Vals Index task. Finance Agent v2 results are mid-tier at 48.4%.

• On coding benchmarks, Qwen 3.7 Max scores 59.2% on Terminal Bench 2.0, 52.9% on the Vibe Code Bench Vals Index subset, and 26.0% on ProofBench.

The model has a ~1M-token context window. Evaluations were run with temperature 0.7 and preserve_thinking enabled.

• We evaluated Google’s new Gemini 3.5 Flash across our benchmark suite. It is the new #1 model on our Finance Agent Benchmark v2, dethroning GPT 5.5 by 6 points.

• The model places #3 on the Vals Index, scoring 62.05%, and #3 on the Vals Multimodal Index, scoring 62.29%. That is an ~8-point increase over Gemini 3.1 Pro Preview (02/26) on the Vals Index and a ~7-point increase on the Vals Multimodal Index.

• We also saw a meaningful increase on Vibe Code Bench, our benchmark measuring how well models can create web applications from scratch. Gemini 3.5 Flash ranked #10, scoring 48.68% — a 16-point jump relative to Gemini 3.1 Pro Preview (02/26). The model also did well on other coding benchmarks, ranking #3 on SWE-bench Verified and #3 on Terminal Bench 2.0.

• On MedCode, our benchmark that assesses whether models can support the medical billing process, Gemini 3.5 Flash ranks #3 overall — just 3.2 points behind the top model, Gemini 3.1 Pro Preview (02/26). It also had a slight 3-point increase on ProofBench.

• The model still lags behind Gemini 3.1 Pro Preview (02/26) on some of our benchmarks: LegalBench, MortgageTax, and academic benchmarks including MMLU Pro, GPQA.

The model has a 1M-token context window. Evaluations were run using a reasoning effort of “high”, a temperature of 1.0, and max output tokens set to 65k.

Congrats to the Google team on the strong release!