Chen-Oliver/verifiers
Languages
Our library for RL environments + evals
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 18 / 30 | $3.60 | $5/$25 | 12m50s |
Key Takeaways
- Claude Opus 5 with Mini-SWE-agent scores 60% across 30 tasks, making this more than a smoke test.
- The run reports 770.04 seconds of latency per task; token metrics were not supplied.
Cost Analysis
Average Token Use / Test
No token usage data available.
Cost is the clearest tradeoff in this comparison. Claude Opus 5 leads at 60.00% for $3.60 per test. No other model in this comparison is cheaper.
Latency Analysis
Average Response Time / Test
Claude Opus 5 is both the most accurate and fastest model in this comparison at 60.00% and 12m 50s.
Tasks with failures
| Models | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5 |
Task detail
a506236Issue statement
Host-side runtime coordination and workspace files must not use predictable, shared top-level paths under /tmp. Store prepared PEP 723 scripts, subprocess runtime workspaces, and the OpenClaw gateway lock under Verifiers' cache directory, creating required parent directories as needed. Environment server pools should use a uniquely owned temporary IPC directory, place each worker socket inside it, and remove that directory during shutdown after cleaning up worker sockets. Preserve the existing content-addressed script reuse and runtime/pool behavior.
View Hidden Tests
diff --git a/tests/v1/test_cache_scoped_host_paths.py b/tests/v1/test_cache_scoped_host_paths.pynew file mode 100644index 00000000..6812c3f6--- /dev/null+++ b/tests/v1/test_cache_scoped_host_paths.py@@ -0,0 +1,57 @@+from pathlib import Path++import pytest++from verifiers.v1.harnesses.openclaw import harness as openclaw_harness+from verifiers.v1.runtimes import base+from verifiers.v1.runtimes.subprocess import SubprocessConfig, SubprocessRuntime+from verifiers.v1.serve.pool import EnvServerPool+from verifiers.v1.utils.paths import CACHE_DIR+++def test_prepared_scripts_use_verifiers_cache_directory() -> None:+ scripts_dir = Path(base.SCRIPTS_DIR)++ assert scripts_dir == CACHE_DIR / "runtimes" / "scripts"+ assert not scripts_dir.is_relative_to(Path("/tmp"))+++@pytest.mark.asyncio+async def test_openclaw_gateway_lock_is_cache_scoped(+ monkeypatch: pytest.MonkeyPatch, tmp_path: Path+) -> None:+ class FakeDockerRuntime:+ network_restricted = False++ cache_dir = tmp_path / "home-cache" / "verifiers"+ monkeypatch.setattr(openclaw_harness, "DockerRuntime", FakeDockerRuntime)+ monkeypatch.setattr(openclaw_harness, "CACHE_DIR", cache_dir, raising=False)++ async with openclaw_harness._gateway_start_lock(FakeDockerRuntime()):+ assert (cache_dir / "harnesses" / "openclaw" / "gateway.lock").is_file()+++@pytest.mark.asyncio+async def test_subprocess_runtime_workspace_is_cache_scoped(+ monkeypatch: pytest.MonkeyPatch, tmp_path: Path+) -> None:+ cache_dir = tmp_path / "home-cache" / "verifiers"+ monkeypatch.setattr("verifiers.v1.runtimes.subprocess.CACHE_DIR", cache_dir)+ runtime = SubprocessRuntime(SubprocessConfig(), name="unit-runtime")++ await runtime.start()++ assert runtime.workdir == cache_dir / "runtimes" / "subprocess" / "unit-runtime"+ assert runtime.workdir.is_dir()+++def test_env_server_pool_owns_and_removes_private_ipc_directory() -> None:+ pool = EnvServerPool({}, 2, "tcp://127.0.0.1:0")+ ipc_dir = Path(pool._ipc_dir)+ try:+ assert ipc_dir.is_dir()+ assert Path(pool._worker_path(3)) == ipc_dir / "3"+ finally:+ pool._shutdown()++ assert not ipc_dir.exists()