arduino/arduino-ide
Languages
Arduino IDE 2.x
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 13 / 30 | $0.03 | N/A | 2m15s | ||
2 | Mini-SWE-agent | 12 / 30 | $1.91 | $1.5/$7.5 | 16m32s | ||
3 | Mini-SWE-agent | 10 / 30 | $0.64 | $1.4/$4.4 | 12m29s |
Key Takeaways
- GPT-5.6 Luna (High) with Mini-SWE-agent scores 43.33%, ahead of Gemini 3.6 Flash with Mini-SWE-agent at 40% and GLM 5.2 (Fireworks) with Mini-SWE-agent at 33.33%.
- GPT-5.6 Luna (High) with Mini-SWE-agent has the lowest cost per test at $0.03 and shortest latency at 135.11 seconds among the supplied runs.
Model Comparison
Accuracy
43.33%
Gpt 5.6 Luna High
40.00%
Gemini 3.6 Flash
Task outcomes
30 tasks
Cost / test
$0.03
Gpt 5.6 Luna High
$1.91
Gemini 3.6 Flash
Cost distribution
Latency
2m 15s
Gpt 5.6 Luna High
16m 32s
Gemini 3.6 Flash
Latency distribution
Cost Analysis
Average Token Use / Test
Cost is the clearest tradeoff in this comparison. openai/gpt-5.6-luna-high leads at 43.33% for $0.03 per test. No other model in this comparison is cheaper.
Latency Analysis
Average Response Time / Test
openai/gpt-5.6-luna-high is both the most accurate and fastest model in this comparison at 43.33% and 2m 15s.
Tasks with failures
| Models | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gpt 5.6 Luna High | |||||||||||||||||||||
| Gemini 3.6 Flash | |||||||||||||||||||||
| GLM 5.2 |
Task detail
b4a3a33Issue statement
Add support for forcing a clean sketch verification/compile. Users must be able to trigger a clean verify with Ctrl/Cmd+Shift+R or by Shift-clicking the Verify toolbar button; the toolbar should show a clean-verify tooltip while Shift is held. The clean choice must be propagated through the compile options to Arduino CLI's compile request so it performs a full recompile. Normal verification must continue to use the non-clean behavior.
View Hidden Tests
diff --git a/arduino-ide-extension/src/test/node/core-service-impl.test.ts b/arduino-ide-extension/src/test/node/core-service-impl.test.tsindex ac419dd2..a07feddb 100644--- a/arduino-ide-extension/src/test/node/core-service-impl.test.ts+++ b/arduino-ide-extension/src/test/node/core-service-impl.test.ts@@ -5,9 +5,39 @@ import { Port, } from '../../common/protocol/boards-service'; import { Port as RpcPort } from '../../node/cli-protocol/cc/arduino/cli/commands/v1/port_pb';+import { Instance } from '../../node/cli-protocol/cc/arduino/cli/commands/v1/common_pb'; import { CoreServiceImpl } from '../../node/core-service-impl'; describe('core-service-impl', () => {+ describe('compileRequest', () => {+ const options = (clean?: boolean) =>+ ({+ sketch: { uri: 'file:///tmp/CleanCompile/CleanCompile.ino' },+ verbose: false,+ optimizeForDebug: false,+ sourceOverride: {},+ clean,+ } as any);++ it('should request a clean compile when clean is enabled', () => {+ const request = new CoreServiceImpl()['compileRequest'](+ options(true),+ new Instance()+ );++ expect(request.getClean()).to.equal(true);+ });++ it('should not request a clean compile by default', () => {+ const request = new CoreServiceImpl()['compileRequest'](+ options(),+ new Instance()+ );++ expect(request.getClean()).to.equal(false);+ });+ });+ describe('createPort', () => { it("should map the 'undefined' port object to an 'undefined' gRPC port value", () => { const actual = new CoreServiceImpl()['createPort'](undefined);