We’ve evaluated GPT 5.1 Codex Max on our coding benchmarks. It boasts a +9.5% performance boost on VibeCodeBench (#3), +1% performance on SWE Bench (#4), and a slight regression on Terminal Bench.

This is not the fastest model in the shed. It’s 3x slower on SWE Bench, 10x on VCB. This is a model for when you need the absolute best, not when you need it quickly.

Correlation isn’t causation… …but the more suffixes we tack onto these models, the longer they seem to take. Excited to test 5.25 Codex Max Ultra High Supreme Edition

The model was run with reasoning high and verbosity medium through OpenAI’s API. Results on IOI and LCB will be released soon!

We found that Mistral Large 3 struggles across domains:

• We found a variety of difficulties with agentic use-cases involving tool-calling. We’re working with the Mistral team to address one such issue.
• The model places second-to-last on our Vals Multimodal Index, ahead of Llama 4 Maverick.

However, there are a few caveats:

• the model represents a meaningful improvement over predecessors like Magistral Medium 1.2 (09/2025) and Mistral Medium 3.1 (05/2025).
• The model is open-weight and performs comparably to other open-weight models like GPT OSS 120B.

We look forward to further improvements from Mistral to come!

Mistral Large 3 is #20 on our Vals Index (out of 24).

A main source of failure was the model’s difficulty in calling tools - a common error mode was the model making a function call with a malformed tool name (e.g. inserting the function arguments into the tool name). (2/3)

However, it achieves a six percentage point increase over Magistral Medium 1.2 (09/2025), with comparable performance on the Vals Index to GPT OSS 120B.

Full results will be posted soon.

Claude Opus 4.5 (Thinking) is the new leader on our Vals Index, surpassing its cheaper variant, Claude Sonnet 4.5 (Thinking).

Like other Anthropic models, Claude Opus 4.5 (Thinking) demonstrates strong performance on coding tasks - the thinking and nonthinking variants take the top 2 spots on SWE-Bench. However, the model performs worse than Claude Sonnet 4.5 (Thinking) on our new VibeCodeBench.

Unlike other Anthropic models, Claude Opus 4.5 (Thinking) showcases strong multimodal capabilities, placing first on our Multimodal Vals Index. It also places first on SAGE, surpassing the previous leader, Gemini 3 Pro (11/25). It’s also cheaper than before - it’s a third of the price of Claude Opus 4.1 (Nonthinking) and on some benchmarks, more token efficient than Claude Sonnet 4.5 (Thinking)!

Along with Claude Sonnet 4.5 (Thinking) and Claude Opus 4.5 (Thinking), the top 3 models on our Vals Index are now all from Anthropic. We look forward to other providers catching up!