vals-ai/valkyrie
Languages
Scalable, cloud-native infrastructure for evaluating AI agents across any benchmark.
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 | 41 / 46 | $1.44 | $5/$30 | 9m14s | |||
2 | 40 / 46 | $0.37 | $2/$12 | 3m35s | |||
3 | 40 / 46 | $0.07 | $0.2/$1.2 | 6m23s | |||
4 | 40 / 46 | $4.48 | $10/$50 | 10m26s | |||
5 | 40 / 46 | $3.38 | $5/$25 | 12m13s | |||
6 | 39 / 46 | $0.53 | $1.4/$4.4 | 9m27s | |||
7 | 39 / 46 | $4.73 | $3/$15 | 11m51s | |||
8 | 39 / 46 | $3.43 | $3/$15 | 15m30s | |||
9 | 38 / 46 | $0.47 | $2/$6 | 3m48s | |||
10 | 38 / 46 | $1.74 | $5/$30 | 8m31s | |||
11 | 38 / 46 | $3.10 | $5/$25 | 9m53s | |||
12 | 36 / 46 | $1.19 | $1.5/$9 | 6m15s | |||
13 | 34 / 46 | $0.62 | $2/$12 | 3m17s | |||
14 | 31 / 46 | $0.37 | $1/$5 | 4m41s |
Key Takeaways
- GPT-5.6 Luna with Mini-SWE-agent matches the 40-task tier at $0.07 per test and 382.95 seconds, the lowest cost in that tier.
- GPT-5.6 Terra with Mini-SWE-agent also resolves 40 tasks, with 214.54 seconds and $0.37 per test, faster than the other 40-task results.
Model Comparison
Accuracy
89.13%
GPT-5.6 Sol
86.96%
GPT-5.6 Luna
Task outcomes
46 tasks
Cost / test
$1.44
GPT-5.6 Sol
$0.07
GPT-5.6 Luna
Cost distribution
Latency
9m 14s
GPT-5.6 Sol
6m 23s
GPT-5.6 Luna
Latency distribution
Cost Analysis
Average Token Use / Test
Cost is the clearest tradeoff in this comparison. GPT-5.6 Sol leads at 89.13% for $1.44 per test. GPT-5.6 Luna is the lower-cost option at 86.96% for $0.07 per test.
Latency Analysis
Average Response Time / Test
Latency separates several models with similarly strong scores. GPT-5.6 Sol leads at 89.13%, while Gemini 3.1 Pro Preview (02/26) is fastest at 3m 17s with 73.91% accuracy.
Tasks with failures
| Models | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5.6 Sol | ||||||||||||||||||
| GPT-5.6 Luna | ||||||||||||||||||
| GPT-5.6 Terra | ||||||||||||||||||
| Claude Opus 4.8 | ||||||||||||||||||
| Claude Fable 5 | ||||||||||||||||||
| GLM 5.2 | ||||||||||||||||||
| Claude Sonnet 5 | ||||||||||||||||||
| Kimi K3 | ||||||||||||||||||
| Grok 4.5 | ||||||||||||||||||
| GPT 5.5 | ||||||||||||||||||
| Claude Opus 4.7 | ||||||||||||||||||
| Gemini 3.5 Flash | ||||||||||||||||||
| Gemini 3.1 Pro Preview (02/26) | ||||||||||||||||||
| Claude Haiku 4.5 (Nonthinking) |
Task detail
cc3fd33Issue statement
The tracker service exposes a POST /retry-or-resume-benchmark/{benchmark_id} endpoint that re-enqueues a previously created benchmark run using the arguments it was originally created with, including the agent contract stored on the benchmark row. That stored contract carries a secrets mapping (environment variable name -> secret name) that is passed through to the tasks when they run.
Today there is no way to change those secrets when resuming or retrying a run. If a secret in the stored contract is wrong, stale, or missing, the only option is to recreate the benchmark from scratch. Users need to be able to override secrets at resume/retry time so the resumed tasks pick up the corrected values.
Please extend the resume/retry endpoint so the request body accepts an optional secrets mapping (defaulting to an empty mapping) of environment-variable name to secret name. When a non-empty secrets mapping is provided, it must be merged into the agent contract's existing secrets before the run is re-enqueued:
- Keys already present in the stored contract are replaced by the values from the request.
- Keys not present in the stored contract are added.
- Keys in the stored contract that are not mentioned in the request are left untouched.
The merge must be persisted to the stored benchmark row (so a later refresh from the database reflects the merged secrets) and must also be reflected in the contract carried by the request payload that is used to re-enqueue the resumed tasks. When no secrets mapping is supplied, behavior must be unchanged. Supplying secrets must not otherwise change existing resume/retry behavior (status codes, task selection, concurrency handling, header forwarding, etc.).
View Hidden Tests
diff --git a/services/tracker/tests/unit/test_stop_and_resume.py b/services/tracker/tests/unit/test_stop_and_resume.pyindex c59a0bbf..7fd88b2c 100644--- a/services/tracker/tests/unit/test_stop_and_resume.py+++ b/services/tracker/tests/unit/test_stop_and_resume.py@@ -518,6 +518,63 @@ class TestStopAndResume: assert observed_headers["X-Descope-Api-Key"] == "tracker-api-key" assert captured_request_json["service_headers"]["X-Descope-Api-Key"] == "tracker-api-key" + async def test_retry_or_resume_applies_secrets_to_stored_contract(+ self,+ example_benchmark_object: Benchmark,+ database_session: Session,+ monkeypatch: MonkeyPatch,+ ):+ """Resume secrets should update the contract used by resumed tasks.++ Test cases:+ - Existing env var mappings are replaced by resume overrides.+ - New env var mappings are added to the stored contract before enqueue.+ """+ benchmark_row = example_benchmark_object+ benchmark_row.status = BenchmarkStatus.STOPPED+ benchmark_row.arguments.contract.secrets = {+ "ANTHROPIC_API_KEY": "old-secret",+ "OPENAI_API_KEY": "openai-secret",+ }+ database_session.add(benchmark_row)+ database_session.commit()++ captured_request_json: dict[str, Any] = {}++ async def _mock_reset_to_in_progress_status(*_args: Any, **_kwargs: Any):+ return ["task_0"]++ class _MockKicker:+ def with_labels(self, **_kwargs: Any) -> "_MockKicker":+ return self++ async def kiq(self, **kwargs: Any) -> None:+ captured_request_json.update(kwargs["start_benchmark_request_json"])++ monkeypatch.setattr("main.reset_to_in_progress_status", _mock_reset_to_in_progress_status)+ monkeypatch.setattr("main.process_benchmark.kicker", lambda: _MockKicker())++ response = client.post(+ f"/retry-or-resume-benchmark/{benchmark_row.id}",+ json={+ "task_ids": [],+ "service_headers": {},+ "secrets": {+ "ANTHROPIC_API_KEY": "new-secret",+ "GEMINI_API_KEY": "gemini-secret",+ },+ },+ )++ assert response.status_code == 200+ assert captured_request_json["contract"]["secrets"] == {+ "ANTHROPIC_API_KEY": "new-secret",+ "OPENAI_API_KEY": "openai-secret",+ "GEMINI_API_KEY": "gemini-secret",+ }+ database_session.refresh(benchmark_row)+ assert benchmark_row.arguments.contract.secrets == captured_request_json["contract"]["secrets"]+ async def test_running_retry_noops_without_error_tasks( self, example_benchmark_object: Benchmark,