pallets/click
Languages
Python composable command line interface toolkit
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 27 / 30 | $1.65 | $5/$25 | 6m34s |
Key Takeaways
- This single 30-task run provides no comparison with other models or harnesses.
- Average latency was 393.84 seconds per test.
Cost Analysis
Average Token Use / Test
No token usage data available.
Cost is the clearest tradeoff in this comparison. Claude Opus 5 leads at 90.00% for $1.65 per test. No other model in this comparison is cheaper.
Latency Analysis
Average Response Time / Test
Claude Opus 5 is both the most accurate and fastest model in this comparison at 90.00% and 6m 34s.
Tasks with failures
| Models | |||
|---|---|---|---|
| Claude Opus 5 |
Task detail
19fd4d6Issue statement
When a Click group is invoked with an unknown command, it should provide useful typo suggestions based on the group's registered commands. A single close match should be presented as a direct suggestion, while multiple close matches should be presented together in a deterministic order. The failure should be represented by a public click.NoSuchCommand usage exception that exposes the attempted command name and its computed close matches. Unknown commands with no close matches should retain the normal No such command '...'. diagnostic without a suggestion.
View Hidden Tests
diff --git a/tests/test_valsmith_unknown_command.py b/tests/test_valsmith_unknown_command.pynew file mode 100644index 0000000..9286b3f--- /dev/null+++ b/tests/test_valsmith_unknown_command.py@@ -0,0 +1,31 @@+import click+++def test_unknown_command_exception_and_suggestions(runner):+ cli = click.Group()++ @cli.command()+ def push():+ pass++ @cli.command()+ def declare():+ pass++ @cli.command()+ def refine():+ pass++ single = runner.invoke(cli, ["pause"], standalone_mode=False)+ assert isinstance(single.exception, click.NoSuchCommand)+ assert single.exception.command_name == "pause"+ assert single.exception.possibilities == ["push"]+ assert "Did you mean 'push'?" in single.exception.format_message()++ multiple = runner.invoke(cli, ["decline"], standalone_mode=False)+ assert isinstance(multiple.exception, click.NoSuchCommand)+ assert set(multiple.exception.possibilities) == {"declare", "refine"}+ assert "(Did you mean one of: 'declare', 'refine'?)" in multiple.exception.format_message()++ unrelated = runner.invoke(cli, ["unknown"], standalone_mode=False)+ assert isinstance(unrelated.exception, click.NoSuchCommand)+ assert unrelated.exception.possibilities == []+ assert "No such command 'unknown'." in unrelated.exception.format_message()+ assert "Did you mean" not in unrelated.exception.format_message()