Partners in Evaluation
Key Takeaways
- As a single number, LegalBench is largely saturated: the top models are bunched near the ceiling (led by Claude Fable 5 at 88.56%), so overall accuracy barely separates the frontier.
- The useful signal is the type of legal reasoning: even the best models handle issue-spotting and drawing conclusions well (~92%) but are markedly weaker at rhetorical analysis (top score only 84%) — so a model’s fitness depends heavily on which legal task you put in front of it.
Results
The results per question type are summarized in the graph below.
Model Output Example
We show some examples from one of the LegalBench free response tasks (Rule QA) here. The models were asked “Where in the Federal Rules of Civil Procedure are notice requirements described?”
This question is tricky because both Rules 4 and 5 talk about the service process and notice requirements. However, Rule 4 is the main rule that defines the first notice in a proceeding, whereas Rule 5 gives the procedures for filing pleadings, etc.
This question is also difficult because implicit in the question is that it’s “notice” of a suit. GPT correctly determines that the question is talking about the initial notice, whereas Opus does not. Interestingly, Gemini gets the answer wrong, but it is much more terse than the other two models. Llama 3.1 makes the same mistake as Opus, citing Rule 5 instead of Rule 4.
It’s important this is one question from a very large evaluation data set and is not representative of overall performance.
Where in the Federal Rules of Civil Procedure are notice requirements described?
INCORRECT
Dataset and Context
There has been a considerable effort to measure language model performance in academic tasks and chatbot settings but these high-level benchmarks are not applicable to specific industry use cases. Here we start to remedy this by reporting our application-specific findings and live leaderboard results on LegalBench, a large crowd-sourced collection of legal reasoning tasks. The data set is quite comprehensive, covering six major categories.
- Issue-spotting, where a model must determine if a fact has relevance to a particular law or legal entity.
- Rule-recall, where a model must identify a relevant rule or state its characteristics.
- Rule-conclusion, where a model must predict a legal outcome.
- Rule-application, where a model must analyze how a rule was applied to reach a conclusion.
- Interpretation, where a model must parse and understand legal text.
- Rhetorical understanding, where a model must determine whether a legal argument performs a certain function.