18jeffreyma/Valkyrie
Languages
Scalable, cloud-native infrastructure for evaluating AI agents across any benchmark.
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 13 / 30 | $4.23 | $5/$25 | 13m23s |
Key Takeaways
- This single 30-task run provides no comparison with another model or harness.
- The run completes in 803.29 seconds on average; the result is directional rather than a statistically significant benchmark conclusion.
Cost Analysis
Average Token Use / Test
No token usage data available.
Cost is the clearest tradeoff in this comparison. Claude Opus 4.8 leads at 43.33% for $4.23 per test. No other model in this comparison is cheaper.
Latency Analysis
Average Response Time / Test
Claude Opus 4.8 is both the most accurate and fastest model in this comparison at 43.33% and 13m 23s.
Tasks with failures
| Models | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 4.8 |
Task detail
4b5e91fIssue statement
Force-stopping an entire benchmark must finalize its database state immediately, without waiting for sandbox-provider teardown. A forced stop should transition all active tasks—including tasks currently in progress—to STOPPED, leave already finished tasks unchanged, transition the benchmark directly to STOPPED once no active tasks remain, and terminalize any active executor dispatch. Provider cleanup may happen afterward and must not be responsible for completing these database transitions. Graceful (non-forced) stop behavior and task-scoped stop behavior should remain unchanged.
View Hidden Tests
diff --git a/services/tracker/tests/unit/utils/test_force_stop_immediate_task.py b/services/tracker/tests/unit/utils/test_force_stop_immediate_task.pynew file mode 100644index 0000000..7038831--- /dev/null+++ b/services/tracker/tests/unit/utils/test_force_stop_immediate_task.py@@ -0,0 +1,61 @@+from typing import Any++from sqlmodel import Session, select++from tests.utils import TEST_ORG_ID+from tracker.database.models import (+ Benchmark,+ BenchmarkStatus,+ ExecutorDispatch,+ ExecutorDispatchStatus,+ Org,+ Task,+ TaskStatus,+)+from tracker.utils import initiate_stop_benchmark+++async def test_force_stop_immediately_finalizes_run_and_active_tasks(+ example_benchmark_object: Benchmark,+ database_session: Session,+ executor_authority: Any,+) -> None:+ benchmark = example_benchmark_object+ benchmark.status = BenchmarkStatus.IN_PROGRESS+ database_session.add(benchmark)+ database_session.add_all(+ [+ Task(org_id=TEST_ORG_ID, task_id="pending", benchmark=benchmark.id, status=TaskStatus.PENDING),+ Task(org_id=TEST_ORG_ID, task_id="building", benchmark=benchmark.id, status=TaskStatus.BUILDING),+ Task(org_id=TEST_ORG_ID, task_id="running", benchmark=benchmark.id, status=TaskStatus.IN_PROGRESS),+ Task(org_id=TEST_ORG_ID, task_id="evaluating", benchmark=benchmark.id, status=TaskStatus.EVALUATING),+ Task(org_id=TEST_ORG_ID, task_id="finished", benchmark=benchmark.id, status=TaskStatus.FINISHED),+ ]+ )+ database_session.commit()+ authority = executor_authority(benchmark, session=database_session)++ await initiate_stop_benchmark(+ benchmark,+ database_session,+ force=True,+ org=Org(id=TEST_ORG_ID, name="default"),+ )++ database_session.refresh(benchmark)+ statuses = {+ task.task_id: task.status+ for task in database_session.exec(select(Task).where(Task.benchmark == benchmark.id)).all()+ }+ dispatch = database_session.get(ExecutorDispatch, authority.dispatch_id)++ assert benchmark.status == BenchmarkStatus.STOPPED+ assert statuses == {+ "pending": TaskStatus.STOPPED,+ "building": TaskStatus.STOPPED,+ "running": TaskStatus.STOPPED,+ "evaluating": TaskStatus.STOPPED,+ "finished": TaskStatus.FINISHED,+ }+ assert dispatch is not None+ assert dispatch.status == ExecutorDispatchStatus.FAILED