The newly-released Claude Opus 4.7 has taken first place on both our Vals Index and Vals Multimodal Index, both by substantial margins. Here are our key takeaways:

• Claude Opus 4.7 perfoms excellently on coding tasks, placing first on Terminal Bench 2 and taking the top spot by several percent on SWE-bench and Vibe Code Bench.
• Claude Opus 4.7 also excels across other domains, placing first on our Finance Agent Benchmark and our math education benchmark, SAGE.
• The model is very consistent, placing in the top ten of all of our benchmarks so far, with the sole exception of Medscribe where it is within 1% of the top ten.
• We also encountered a relatively large number of safety refusals on Claude Opus 4.7, with the model refusing to answer certain questions on benchmarks such as GPQA and Corpfin.

Congrats to Anthropic on the release!

We evaluated Qwen 3.6 Plus on our suite of benchmarks. Here are the key takeaways:

• On MedCode and MedScribe, the model ranked 30/50 and 27/50 respectively, indicating room for improvement.
• Qwen 3.6 Plus places #17 overall on Vals Index (58.8% accuracy), just above GPT-5.4 Mini and just below Qwen 3.5 Plus (#14, 59.1%).
• The model performs well on knowledge benchmarks, placing #6 on MMLU Pro (87.7%) and #8 on MMMU (84.2%).
• It lands in the top 15 on most other benchmarks, with notable results on Mortgage Tax (#9, 68.0%) and Finance Agent (#11, 54.6%).
• Its weakest showing is on Case Law (v2), where it ranks #42 (51.4%).
• On Vibe Code Bench, it scores 25.9%, placing #13 and demonstrating a ~10% improvement over Qwen 3.5 Plus (15.74%, ranked #18).

All evals were run via the Alibaba API.

We evaluated Gemma 4 31B IT — a 31B dense open model in Google DeepMind’s Gemma 4 family, with a 262k context window and 33k max output.

Key takeaways:

• Its best result is #1 on SAGE at 55.03%, which makes Gemma 4 31B IT look strongest on education and exam-style reasoning.
• It is also competitive on structured document and finance work: 61.37% on MortgageTax, 52.63% on Case Law (v2), and 50.79% on Finance Agent (v1.1).
• On the subset tasks used in Vals Index, it posted 59.67% on the Corp Fin (v2) shared-max-context split and 53.92% on the SWE-bench subset.
• Its latency is strongest on structured tasks like Case Law (28.7s) and MortgageTax (41.9s), then rises sharply on heavier agentic workloads like Vals Index (323.2s), Terminal-Bench 2.0 (855.3s), and Finance Agent (5360.3s).
• Overall, Gemma lands at 38.94% on Vals Index and 45.12% on Vals Multimodal Index. The main story here is efficiency for size, not broad frontier dominance.

We’ve completed our evaluation of Muse Spark — Meta’s first frontier release in a year.

It debuts at #3 on the Vals Index (65.66%), re-entering the top tier alongside Claude Opus 4.6 (Thinking) and GPT 5.4.

Key takeaways:

• #1 on TaxEval (77.68%), dethroning Claude Sonnet 4.6. It also takes #2/41 on Finance Agent (v1.1) (60.60%), showing standout strength in finance.
• Broad performance across hard evals — top placements on AIME, GPQA, and MedScribe — not isolated to a single domain.
• Obtains third place on Terminal Bench 2, scoring 59.55%.

Evaluations were run via the Meta API. Congrats to the team at Meta on a standout release.