astral-sh/uv
Languages
An extremely fast Python package and project manager, written in Rust.
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 35 / 53 | $7.60 | $10/$50 | 17m30s | ||
2 | Mini-SWE-agent | 34 / 53 | $2.26 | $5/$30 | 9m07s | ||
3 | Mini-SWE-agent | 34 / 53 | $8.50 | $3/$15 | 15m26s | ||
4 | Mini-SWE-agent | 33 / 53 | $4.61 | $5/$25 | 17m50s | ||
5 | Mini-SWE-agent | 32 / 53 | $0.45 | $2/$12 | 3m10s | ||
6 | Mini-SWE-agent | 32 / 53 | $0.11 | $0.2/$1.2 | 4m53s | ||
7 | Mini-SWE-agent | 32 / 53 | $2.41 | $5/$30 | 7m02s | ||
8 | Mini-SWE-agent | 30 / 53 | $0.94 | $2/$6 | 7m02s | ||
9 | Mini-SWE-agent | 29 / 53 | $4.60 | $3/$15 | 19m55s | ||
10 | Mini-SWE-agent | 28 / 53 | $0.81 | $1.4/$4.4 | 18m20s | ||
11 | Mini-SWE-agent | 27 / 53 | $4.59 | $5/$25 | 15m56s | ||
12 | Mini-SWE-agent | 25 / 53 | $0.91 | $2/$12 | 5m20s | ||
13 | Mini-SWE-agent | 24 / 53 | $1.56 | $1.5/$9 | 6m18s | ||
14 | Mini-SWE-agent | 17 / 53 | $0.45 | $1/$5 | 6m14s |
Key Takeaways
- Kimi K3 with Mini-SWE-agent and GPT-5.6 Sol with Mini-SWE-agent each resolve 34 of 53 tasks, one behind Claude Fable 5.
- GPT-5.6 Luna with Mini-SWE-agent, GPT-5.6 Terra with Mini-SWE-agent, and GPT 5.5 with Mini-SWE-agent each resolve 32 of 53 tasks; GPT-5.6 Luna is lowest-cost at $0.11 per test.
- GPT-5.6 Terra with Mini-SWE-agent is fastest among the 32-task results at 189.52 seconds, while GPT-5.6 Luna with Mini-SWE-agent costs $0.11 per test.
Model Comparison
Accuracy
66.04%
Claude Fable 5
64.15%
GPT-5.6 Sol
Task outcomes
53 tasks
Cost / test
$7.60
Claude Fable 5
$2.26
GPT-5.6 Sol
Cost distribution
Latency
17m 30s
Claude Fable 5
9m 7s
GPT-5.6 Sol
Latency distribution
Cost Analysis
Average Token Use / Test
Cost is the clearest tradeoff in this comparison. Claude Fable 5 leads at 66.04% for $7.60 per test. GPT-5.6 Sol is the lower-cost option at 64.15% for $2.26 per test.
Latency Analysis
Average Response Time / Test
Latency separates several models with similarly strong scores. Claude Fable 5 leads at 66.04%, while GPT-5.6 Terra is fastest at 3m 10s with 60.38% accuracy.
Tasks with failures
| Models | ||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Fable 5 | ||||||||||||||||||||||||||||||||||||||||||
| GPT-5.6 Sol | ||||||||||||||||||||||||||||||||||||||||||
| Kimi K3 | ||||||||||||||||||||||||||||||||||||||||||
| Claude Opus 4.8 | ||||||||||||||||||||||||||||||||||||||||||
| GPT-5.6 Luna | ||||||||||||||||||||||||||||||||||||||||||
| GPT-5.6 Terra | ||||||||||||||||||||||||||||||||||||||||||
| GPT 5.5 | ||||||||||||||||||||||||||||||||||||||||||
| Grok 4.5 | ||||||||||||||||||||||||||||||||||||||||||
| Claude Sonnet 5 | ||||||||||||||||||||||||||||||||||||||||||
| GLM 5.2 | ||||||||||||||||||||||||||||||||||||||||||
| Claude Opus 4.7 | ||||||||||||||||||||||||||||||||||||||||||
| Gemini 3.1 Pro Preview (02/26) | ||||||||||||||||||||||||||||||||||||||||||
| Gemini 3.5 Flash | ||||||||||||||||||||||||||||||||||||||||||
| Claude Haiku 4.5 (Nonthinking) |
Task detail
615a11cIssue statement
When a URL that contains embedded credentials (for example https://user:password@example.com/simple/) is stored in a VerbatimUrl, the struct keeps the original, user-provided string in its given field so that the verbatim form can be reproduced later.
The parsed URL held by VerbatimUrl already redacts credentials when it is formatted (its username/password are shown as ****). However, the Debug output of VerbatimUrl prints the given field as-is, so any credentials that were part of the original string are exposed in plain text.
This is a problem because VerbatimUrl values end up in debug/diagnostic output in several places (for example when settings containing index or publish URLs are dumped, or via verbose logging). A URL such as https://user:password@example.com/simple/ is rendered with the parsed url field correctly masked, but the given field still shows the literal password.
Formatting a VerbatimUrl with {:?} must not reveal credentials that were embedded in the original URL. The credentials in the given field should be masked the same way they are for the parsed URL, while URLs without credentials should continue to be shown unchanged.
View Hidden Tests
diff --git a/crates/uv-pep508/src/verbatim_url.rs b/crates/uv-pep508/src/verbatim_url.rsindex 730ef8a..0bfc9f1 100644--- a/crates/uv-pep508/src/verbatim_url.rs+++ b/crates/uv-pep508/src/verbatim_url.rs@@ -948,4 +948,17 @@ mod tests { }, ); }++ #[test]+ fn debug_redacts_given_credentials() {+ // The `given` field preserves the URL exactly as the user provided it, including any+ // embedded credentials. The `Debug` output must not leak those credentials.+ let raw = "https://user:s3cr3t-token@example.com/simple/";+ let url = VerbatimUrl::parse_url(raw).unwrap().with_given(raw);+ let debug = format!("{url:?}");+ assert!(+ !debug.contains("s3cr3t-token"),+ "credentials leaked in Debug output: {debug}"+ );+ } }