vercel/next.js
Languages
The React Framework
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 55 / 74 | $2.51 | $5/$30 | 8m28s | ||
2 | Mini-SWE-agent | 54 / 74 | $11.73 | $3/$15 | 16m44s | ||
3 | Mini-SWE-agent | 51 / 74 | $8.80 | $10/$50 | 17m24s | ||
4 | Mini-SWE-agent | 50 / 74 | $0.99 | $2/$6 | 5m22s | ||
5 | Mini-SWE-agent | 44 / 74 | $5.02 | $5/$25 | 18m05s | ||
6 | Mini-SWE-agent | 43 / 74 | $0.92 | $1.4/$4.4 | 19m31s | ||
7 | Mini-SWE-agent | 43 / 74 | $4.78 | $3/$15 | 20m06s | ||
8 | Mini-SWE-agent | 40 / 74 | $1.27 | $1.5/$9 | 5m40s | ||
9 | Mini-SWE-agent | 39 / 74 | $0.15 | $0.2/$1.2 | 6m08s | ||
10 | Mini-SWE-agent | 38 / 74 | $2.32 | $5/$30 | 6m04s | ||
11 | Mini-SWE-agent | 38 / 74 | $4.38 | $5/$25 | 11m34s | ||
12 | Mini-SWE-agent | 34 / 74 | $0.48 | $2/$12 | 3m07s | ||
13 | Mini-SWE-agent | 34 / 74 | $0.75 | $2/$12 | 4m18s | ||
14 | Mini-SWE-agent | 32 / 74 | $0.39 | $1/$5 | 5m09s |
Key Takeaways
- Kimi K3 with Mini-SWE-agent resolves 54 of 74 tasks, one fewer than GPT-5.6 Sol with Mini-SWE-agent, at $11.73 per test.
- Grok 4.5 with Mini-SWE-agent is the fastest result at 321.86 seconds, resolving 50 of 74 tasks for $0.99 per test.
- Claude Haiku 4.5 (Nonthinking) with Mini-SWE-agent records the lowest score, resolving 32 of 74 tasks at $0.39 per test.
Model Comparison
Accuracy
74.32%
GPT-5.6 Sol
72.97%
Kimi K3
Task outcomes
74 tasks
Cost / test
$2.51
GPT-5.6 Sol
$11.73
Kimi K3
Cost distribution
Latency
8m 28s
GPT-5.6 Sol
16m 44s
Kimi K3
Latency distribution
Cost Analysis
Average Token Use / Test
Cost is the clearest tradeoff in this comparison. GPT-5.6 Sol leads at 74.32% for $2.51 per test. No other model in this comparison is cheaper.
Latency Analysis
Average Response Time / Test
Latency separates several models with similarly strong scores. GPT-5.6 Sol leads at 74.32%, while GPT-5.6 Terra is fastest at 3m 7s with 45.95% accuracy.
Tasks with failures
| Models | |||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5.6 Sol | |||||||||||||||||||||||||||||||||||||||||||||||||||||
| Kimi K3 | |||||||||||||||||||||||||||||||||||||||||||||||||||||
| Claude Fable 5 | |||||||||||||||||||||||||||||||||||||||||||||||||||||
| Grok 4.5 | |||||||||||||||||||||||||||||||||||||||||||||||||||||
| Claude Opus 4.8 | |||||||||||||||||||||||||||||||||||||||||||||||||||||
| GLM 5.2 | |||||||||||||||||||||||||||||||||||||||||||||||||||||
| Claude Sonnet 5 | |||||||||||||||||||||||||||||||||||||||||||||||||||||
| Gemini 3.5 Flash | |||||||||||||||||||||||||||||||||||||||||||||||||||||
| GPT-5.6 Luna | |||||||||||||||||||||||||||||||||||||||||||||||||||||
| GPT 5.5 | |||||||||||||||||||||||||||||||||||||||||||||||||||||
| Claude Opus 4.7 | |||||||||||||||||||||||||||||||||||||||||||||||||||||
| GPT-5.6 Terra | |||||||||||||||||||||||||||||||||||||||||||||||||||||
| Gemini 3.1 Pro Preview (02/26) | |||||||||||||||||||||||||||||||||||||||||||||||||||||
| Claude Haiku 4.5 (Nonthinking) |
Task detail
66182ebIssue statement
When a Next.js project sets output: 'export' (or output: 'standalone') and also enables the experimental supportsImmutableAssets flag, the flag is incorrectly left enabled in the resolved configuration.
supportsImmutableAssets is designed to work together with the adapter code path, which controls how static assets are emitted (e.g. under a versioned _next/static/immutable/ directory). The output: 'export' and output: 'standalone' modes currently use a non-adapter code path, so honoring supportsImmutableAssets there produces incorrect output.
Expected behavior: while resolving/normalizing the Next.js configuration, if experimental.supportsImmutableAssets is enabled but output is set to 'export' or 'standalone', the flag must be force-disabled (experimental.supportsImmutableAssets === false) in the final resolved config, even when the user explicitly requested true. When output is not one of those values, an explicitly enabled supportsImmutableAssets must remain enabled.
This resolution should happen as part of the same config finalization that applies other framework defaults, so it is reflected in the config returned by loading the configuration.
View Hidden Tests
diff --git a/test/unit/isolated/supports-immutable-assets-output.test.ts b/test/unit/isolated/supports-immutable-assets-output.test.tsnew file mode 100644index 00000000..e80a1c84--- /dev/null+++ b/test/unit/isolated/supports-immutable-assets-output.test.ts@@ -0,0 +1,45 @@+/* eslint-env jest */+import { PHASE_PRODUCTION_BUILD } from 'next/constants'++// force require usage instead of dynamic import in jest+// x-ref: https://github.com/nodejs/node/issues/35889+process.env.__NEXT_TEST_MODE = 'jest'++describe('config - supportsImmutableAssets with output', () => {+ let loadConfig: typeof import('next/dist/server/config').default++ beforeEach(async () => {+ jest.resetModules()+ const configModule = await import('next/dist/server/config')+ loadConfig = configModule.default+ })++ it('force-disables supportsImmutableAssets when output is "export"', async () => {+ const config = await loadConfig(PHASE_PRODUCTION_BUILD, '<rootDir>', {+ customConfig: {+ output: 'export',+ experimental: { supportsImmutableAssets: true },+ },+ })+ expect(config.experimental.supportsImmutableAssets).toBe(false)+ })++ it('force-disables supportsImmutableAssets when output is "standalone"', async () => {+ const config = await loadConfig(PHASE_PRODUCTION_BUILD, '<rootDir>', {+ customConfig: {+ output: 'standalone',+ experimental: { supportsImmutableAssets: true },+ },+ })+ expect(config.experimental.supportsImmutableAssets).toBe(false)+ })++ it('keeps supportsImmutableAssets enabled when output is not set', async () => {+ const config = await loadConfig(PHASE_PRODUCTION_BUILD, '<rootDir>', {+ customConfig: {+ experimental: { supportsImmutableAssets: true },+ },+ })+ expect(config.experimental.supportsImmutableAssets).toBe(true)+ })+})