CorpFin (v2) Benchmark

Key Takeaways

• GPT-4.1 is the best performing model on this benchmark achieving 72.3% accuracy.
• Grok 3 Beta is the second best performing model on this benchmark achieving 69.1% accuracy.
• Grok 3 Mini Fast Beta High Reasoning is just behind at 68.6% accuracy.
• o3 Mini seems to struggle with large context windows, performing poorly on the Max Fitting Context task. It tends to lose the question if it is provided at the beginning of a large context window (around 150k tokens and more).

Dataset

In both the finance and legal industry, it is a common task to ask a specific question or understand a piece of information from a very long document. A common document type is a credit agreement - contracts, often over 200 pages, used when large corporations receive a line of credit from a banking entity (you can find an example agreement here).

For this dataset, we worked with a team of experts, including financial analysts, legal professionals, and academics,to create a set of questions and answers about these credit agreements. The dataset is divided into:

• Public Validation: 20 questions from 1 document, accessible to anyone on request (email contact@vals.ai for access).
• Private Validation: 340 questions from 17 documents, available for purchase to evaluate and improve your own models.
• Test: 858 questions from 43 documents. This is the privately held set for Vals AI benchmarking and is never shared.

The dataset is also organized into three distinct tasks, which all use the entire set of questions, but pass a different subset of information into each models’ context window. The three tasks are:

1. Exact Pages: This task provides only the necessary pages required to answer each question, typically resulting in a small context of only a few pages.
2. Shared Max Context: This task includes approximately 80 PDF pages while ensuring all information needed for the correct answer is included. The subselection doesn’t necessarily start at the first page, which can make document structure comprehension challenging for models.
3. Max Fitting Context: This task includes the largest possible chunk of the document, starting from the first page, that fits within the model’s context window. This means longer-context models have more information.

The types of questions written by the experts include:

• Basic extraction of terms and numbers: Some examples include “Who is the borrower’s legal counsel?” or “What currencies are the USD 325mm RCF first lien available in?”
• Summarization and interpretation questions: Some examples include, “Is there an erroneous payment provision?” and “How will the loan proceeds be used?”
• Numeric reasoning or calculation-based questions: Some examples include “How much initial debt capacity is available to the Borrower on day one?”
• Questions involving referring to multiple sections of the provided text, especially to previous definitions: For instance, one example is “What is the minimum required amount of Available Liquidity that the company must maintain?” The model must refer to a previous definition of Available Liquidity.
• Giving opinions of terms based on market standards: For instance, “Are there any unusual terms used to define or adjust EBITDA?” These questions require the models to make a judgment call, rather than just make a statement of fact.
• Questions making use of industry jargon: There are several terms like “baskets”, which have a commonly understood meaning in the industry, but are almost never explicitly used in the agreement itself. An example is “Does the contract contain a Chewy Blocker?” (a type of clause meant to prevent a subsidiary from being released from its debt obligations).

Overall Results

Although the bigger models, like those from OpenAI and Anthropic, are performing the best, the gap between the smaller models is not as big as in other benchmarks. When the latency and price are taken into account, it means that sometimes the smaller models provide a better accuracy for a given cost.

Model highlights

GPT 4.1

Released date : 4/14/2025

Accuracy :

71.2%

Latency :

10.3s

Cost :

$2.00 / $8.00

• GPT-4.1 (2025-04-14) shows exceptional performance on this benchmark, taking the #1 spot with the highest accuracy.
• It demonstrates superior ability to process financial documents and extract relevant information across all task types.
• The model does particularly well on the Max Fitting Context task, partially explanable by its large context window.

View Model

Model Output Examples

The questions that require synthetizing content and calculating values across multiple sections in the document are some of the hardest. The parsing from the original PDFs also adds a lot of special characters which can make it difficult to find information, in tables for example.

Below is an example of a number retrieval question, on the Max Fitting Context task. This question can be difficult for models because in a long financial document, there can be several similar tables that relate to the question, but only one is correct.

In this case, the expected answer is 3.25 to 1.00.

Additional Notes

Context Window

Our testing across different context lengths reveals that the context window is an extremely significant factor in the model’s performance. Models have different window sizes across providers:

• Claude 3.5 Sonnet Latest and Claude 3.5 Haiku Latest can handle up to 200k tokens
• GPT-4o 2024-11-20 and Llama 3.1 Instruct Turbo (8B) are limited to 128k tokens
• Gemini 1.5 Flash (002) effectively has no limit for our test documents

Evaluation Methodology

All evaluations were conducted using Claude 3.5 Sonnet Latest as the judge, with temperature set to 0. Even with fixed temperature, OpenAI models showed slight non-determinism (1-2% variance). The evaluation process revealed that both the judging prompt and the judge model can significantly impact results.

Retrieval-Augmented Generation (RAG)

As an alternative to including the entire document in the context window, RAG techniques are extremely common. This method breaks up long documents and databases into “chunks” which are first retrieved, and then passed to the model as context. This is another area of evaluation we will explore further.