abhishek203/SfAgent
Languages
🙌 OpenHands: Code Less, Make More
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 | 24 / 30 | $0.23 | N/A | 57.08s | |||
2 | 21 / 30 | $0.18 | $1/$5 | 2m01s |
Key Takeaways
- Claude Opus 4.8 (High Effort) with Mini-SWE-agent scores 80%, compared with 70% for Claude Haiku 4.5 (Thinking) with Mini-SWE-agent.
- Claude Opus 4.8 (High Effort) with Mini-SWE-agent costs $0.23 per test and takes 57.08 seconds; Claude Haiku 4.5 (Thinking) costs $0.18 and takes 120.54 seconds.
Model Comparison
Accuracy
80.00%
Claude Opus 4 8 High
70.00%
Claude Haiku 4.5 (Thinking)
Task outcomes
30 tasks
Cost / test
$0.23
Claude Opus 4 8 High
$0.18
Claude Haiku 4.5 (Thinking)
Cost distribution
Latency
57s
Claude Opus 4 8 High
2m 1s
Claude Haiku 4.5 (Thinking)
Latency distribution
Cost Analysis
Average Token Use / Test
Cost is the clearest tradeoff in this comparison. anthropic/claude-opus-4-8-high leads at 80.00% for $0.23 per test. Claude Haiku 4.5 (Thinking) is the lower-cost option at 70.00% for $0.18 per test.
Latency Analysis
Average Response Time / Test
anthropic/claude-opus-4-8-high is both the most accurate and fastest model in this comparison at 80.00% and 57s.
Tasks with failures
| Models | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 4 8 High | ||||||||||
| Claude Haiku 4.5 (Thinking) |
Task detail
1354675Issue statement
Remote runtime startup can take long enough that callers currently receive no progress feedback. When a status callback is supplied, report the startup lifecycle in order: runtime startup begins, container image construction begins (when an image must be built), the runtime is waiting for the remote client after preparing the start request, and a blank status clears the message once initialization is complete. Supplying no callback must remain supported.
View Hidden Tests
diff --git a/tests/valsmith/test_remote_runtime_status.py b/tests/valsmith/test_remote_runtime_status.pynew file mode 100644index 00000000..2430de2d--- /dev/null+++ b/tests/valsmith/test_remote_runtime_status.py@@ -0,0 +1,136 @@+import ast+import os+import threading+import types+import uuid+from pathlib import Path+++def _load_remote_runtime_class():+ source = Path('openhands/runtime/remote/runtime.py').read_text()+ tree = ast.parse(source)+ remote = next(+ node+ for node in tree.body+ if isinstance(node, ast.ClassDef) and node.name == 'RemoteRuntime'+ )+ methods = [+ node+ for node in remote.body+ if isinstance(node, (ast.Assign, ast.AnnAssign))+ or (+ isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))+ and node.name in {'__init__', 'send_status_message'}+ )+ ]+ isolated = ast.Module(+ body=[+ ast.ClassDef(+ name='RemoteRuntime',+ bases=[ast.Name(id='Runtime', ctx=ast.Load())],+ keywords=[],+ body=methods,+ decorator_list=[],+ )+ ],+ type_ignores=[],+ )+ ast.fix_missing_locations(isolated)++ class Runtime:+ def __init__(+ self,+ config,+ event_stream,+ sid,+ plugins,+ env_vars,+ status_message_callback,+ ):+ self.plugins = plugins or []+ self.status_message_callback = status_message_callback++ class Session:+ def __init__(self):+ self.headers = {}++ class Response:+ status_code = 200+ text = ''++ def __init__(self, payload):+ self.payload = payload++ def json(self):+ return self.payload++ def send_request(_session, method, url, **_kwargs):+ if url.endswith('/registry_prefix'):+ return Response({'registry_prefix': 'registry.example'})+ if url.endswith('/image_exists'):+ return Response({'exists': True})+ if url.endswith('/start'):+ response = Response({'runtime_id': 'runtime-1', 'url': 'http://runtime'})+ response.status_code = 201+ return response+ raise AssertionError(f'unexpected request: {method} {url}')++ namespace = {+ 'Runtime': Runtime,+ 'AppConfig': object,+ 'EventStream': object,+ 'PluginRequirement': object,+ 'Optional': __import__('typing').Optional,+ 'Callable': __import__('typing').Callable,+ 'RemoteRuntimeBuilder': lambda *_args: object(),+ 'build_runtime_image': lambda *_args, **_kwargs: 'built-image',+ 'send_request': send_request,+ 'requests': types.SimpleNamespace(Session=Session),+ 'logger': types.SimpleNamespace(+ warning=lambda *_args: None,+ info=lambda *_args: None,+ debug=lambda *_args: None,+ ),+ 'os': os,+ 'threading': threading,+ 'uuid': uuid,+ }+ exec(compile(isolated, '<isolated RemoteRuntime>', 'exec'), namespace)+ return namespace['RemoteRuntime']+++def test_remote_runtime_reports_startup_status_in_order():+ remote_runtime = _load_remote_runtime_class()+ messages = []+ sandbox = types.SimpleNamespace(+ api_key='secret',+ remote_runtime_api_url='https://sandbox.example',+ runtime_container_image=None,+ base_container_image='python:3.12',+ runtime_extra_deps=None,+ force_rebuild_runtime=False,+ browsergym_eval_env=None,+ user_id=1000,+ timeout=120,+ )+ config = types.SimpleNamespace(+ sandbox=sandbox,+ workspace_base=None,+ workspace_mount_path_in_sandbox='/workspace',+ debug=False,+ run_as_openhands=True,+ )++ remote_runtime(+ config=config,+ event_stream=object(),+ sid='session',+ status_message_callback=messages.append,+ )++ assert messages == [+ 'STATUS$STARTING_RUNTIME',+ 'STATUS$STARTING_CONTAINER',+ 'STATUS$WAITING_FOR_CLIENT',+ ' ',+ ]