TaxEval (v2) Benchmark

Key Takeaways

• GPT 5 Mini , OpenAI’s latest reasoning model, takes the #1 spot.
• Seven of the top ten models are reasoning models, highlighting the advancements made possible by this new technology.
• The top models are all tightly clustered around 80%

Dataset and Context

TaxEval v2 evaluates models’ abilities to answer hard tax-related questions. This version focuses on both answer correctness and structured reasoning capabilities. This dataset was created in collaboration with financial and tax experts, who have both created and double-checked all questions and answers.

Some key features:

• 1,500+ total questions across validation and test sets
• A balanced distribution of topics and question types
• Comprehensive evaluation of both answers and reasoning steps

The benchmark consists of three main components:

• Public Validation Set: 20 samples, available upon request (contact contact@vals.ai).
• Private Validation Set: 300 samples for model evaluation, available for purchase to evaluate and improve models.
• Test Set: 1,223 samples. These samples are never shared.

The benchmark is composed of two tasks (each task uses the same questions, but they are evaluated differently):

1. Answer Correctness: The factual correctness of the answer, as compared to a ground truth.
2. Stepwise Reasoning: The quality and structure of the reasoning process as compared to the reasoning process of human experts. This ensures models not only provide correct answers but also demonstrate a clear thinking process.

The benchmark includes a diverse range of question types:

• Application and Compliance (18.3%)
• Comparative Analysis (16.2%)
• Numerical Reasoning (16.7%)
• Problem Solving and Critical Thinking (16.5%)
• Semantic Analysis (18.0%)
• Updates and Current Affairs (15.9%)

Each category is carefully balanced between the private validation and test sets to ensure representative sampling. The evaluation process uses Claude 3.5 Sonnet Latest as judge for both answer correctness and stepwise reasoning assessment.

Results

Model Output Examples

The hardest questions for the models typically require complex, multi-step calculations, reasoning on what laws and numbers to use given the information, or accessing more recent data.

Below is an example where only GPT 5 gets it right ($18,280), while the other models perform the calculations incorrectly. All models used a significant amount of reasoning tokens for this answer.