Key Takeaways
-
Seven of the 86 models evaluated reach 95% or better, and the leader, Claude Opus 5 at 97.00%, sits 3.00 percentage points from a perfect score — little room is left to separate frontier models on this benchmark alone.
-
Open-weight models have reached the very top: DeepSeek V4 Pro 0813 is second overall at 96.40%, within 0.60 points of the closed leader, with Kimi K3 (93.40%) also ahead of closed models such as Claude Opus 4.8 (88.60%) and Grok 4.5 (86.60%).
Background
SWE-bench, introduced by Jimenez et al. in their seminal paper “Can Language Models Resolve Real-World GitHub Issues?”, has emerged as a prominent benchmark for evaluating Large Language Models (LLMs) in software engineering contexts.
The benchmark comprises 500 tasks, each executed within an isolated Docker container. These tasks represent real-world GitHub issues from various repositories. Models must generate a “patch” to resolve each issue. The success of a model’s solution is determined by running unit tests against the generated patch.
A notable complexity of SWE-bench lies in its dual evaluation of both the agentic harness and the underlying foundation model. This leads to different methodologies adopted by foundation model labs when they report their results. Additionally, the benchmark’s computational requirements make it resource-intensive to reproduce results.
To enable fair and consistent comparisons across foundation models, we use a minimal bash-tool-only agent harness. Models are given a single tool — bash — and must navigate, search, edit, and solve tasks using standard command-line tools. This puts the evaluation burden squarely on the model rather than the harness.
Results
Models that perform well on SWE-bench Verified tend to be proficient with bash and standard command-line tools for code navigation and editing.
There is also a clear trend that closed-source models perform better on SWE-bench Verified than open-source models. The clearest performance differences appear among tasks that take between 15 minutes and 1 hour to complete. Models that perform well on these tasks tend to score higher overall on the benchmark.
Resolution Rate by Task Difficulty
Tool Use
All models are given a single tool: bash. Models must use standard command-line tools (grep, find, sed, etc.) to navigate codebases, search for relevant files, and apply edits. This means tool usage differences between models reflect their command-line fluency and problem-solving strategy rather than how they interact with specialized harness tooling.
Methodology
We use a minimal bash-tool-only agent harness, mini-swe-agent, for all evaluations. Models are given a single tool — bash — and a system prompt describing the task. They must use standard command-line tools to navigate the codebase, identify the relevant code, and produce a patch.
We use the SWE-bench Verified subset of the dataset. SWE-bench Verified is a human-validated section of the SWE-bench dataset released by OpenAI in August 2024. Each task in the split has been carefully reviewed and validated by human experts, resulting in a curated set of 500 high-quality test cases from the original benchmark. You can find more information about the Verified split of the dataset here.
All models have access to the same tool to ensure a fair, apples-to-apples comparison. Models are run with the default configuration given by the provider, except for the max token limit, which is always set to the highest value possible.
All experiments are run on isolated cloud sandboxes. Latency is calculated starting from the first step the model takes within each task.
It may be possible to build better harnesses for a given model. However, our aim is to adopt a fair framework with which to evaluate all models.