Public

skypilot-org/skypilot

Updated: 8/13/2026

Languages

Python89.4%JavaScript7.6%Jinja1.2%Shell1.1%HTML0.4%Go0.1%Other0.2%

4 Models30 Tasks

The AI Compute Platform for frontier teams. SkyPilot turns fragmented AI compute into one AI supercomputer, so frontier AI teams build custom intelligence faster.

Add models

		Harness			Input / Output Cost
1	GLM 5.2	Mini-SWE-agent	9 / 30	$0.76	$1.4/$4.4	10m01s
2	Claude Opus 4.8	Mini-SWE-agent	9 / 30	$4.20	$5/$25	13m17s
3	Claude Opus 5	Mini-SWE-agent	9 / 30	$4.20	$5/$25	13m48s
4	DeepSeek V4 Flash 0731	Mini-SWE-agent	8 / 30	$0.07	$0.44/$1.32	5m39s

Key Takeaways

The 30-task results are directional: Claude Opus 4.8, Claude Opus 5, and GLM 5.2 (Fireworks) tie at 30%, ahead of DeepSeek V4 Flash at 26.67%.
GLM 5.2 (Fireworks) with Mini-SWE-agent has the lowest cost per test at $0.76 and shortest latency at 601.19 seconds among the tied leaders.

Model Comparison

Accuracy

30.00%

GLM 5.2

30.00%

Claude Opus 5

Task outcomes

30 tasks

Both: 9 tasks

Neither: 21 tasks

Both

GLM 5.2 only

Claude Opus 5 only

Neither

Not attempted

Cost / test

$0.76

GLM 5.2

$4.20

Claude Opus 5

Cost distribution

Latency

10m 1s

GLM 5.2

13m 48s

Claude Opus 5

Latency distribution

Cost Analysis

Cost / Test vs. Accuracy

ACCURACYCOST

Average Token Use / Test

Token Usage

InputOutputReasoningCache readCache write

Claude Opus 5

5.0M

Claude Opus 4.8

4.7M

DeepSeek V4 Flash 0731

2.0M

Cost is the clearest tradeoff in this comparison. Claude Opus 4.8 leads at 30.00% for $4.20 per test. DeepSeek V4 Flash 0731 is the lower-cost option at 26.67% for $0.07 per test.

Latency Analysis

Latency vs. Accuracy

ACCURACYLATENCY

Average Response Time / Test

Response Time

Claude Opus 5

13m 48s

Claude Opus 4.8

13m 17s

GLM 5.2

10m 1s

DeepSeek V4 Flash 0731

5m 39s

Latency separates several models with similarly strong scores. Claude Opus 4.8 leads at 30.00%, while DeepSeek V4 Flash 0731 is fastest at 5m 39s with 26.67% accuracy.

Tasks with failures

Models
GLM 5.2
Claude Opus 5
Claude Opus 4.8
DeepSeek V4 Flash 0731

Task detail

2949e1a

Issue statement

Managed-job controllers currently skip retaining local logs whenever an external logging agent and a reader are configured. That configuration does not prove that the agent actually ran on the particular cluster, so a short job or a cluster where deployment failed can end up with no readable logs anywhere. Add a public plugin extension point that lets the component operating the logging agent register a fast per-cluster callback. The callback receives both the display cluster name and provider-side cluster name, and returns a human-readable reason when delivery is known not to have occurred, or None otherwise. During job finalization, when an agent and reader are configured, consult this callback before discarding the local copy: retain/download the local logs when a reason is returned, and preserve the existing skip behavior when delivery is not disputed. With no callback registered, behavior must remain unchanged. Failures raised by the callback must be contained and treated like no reported delivery problem.

View Hidden Tests

diff --git a/tests/unit_tests/test_sky/jobs/test_log_delivery_fallback.py b/tests/unit_tests/test_sky/jobs/test_log_delivery_fallback.pynew file mode 100644index 00000000..424e59af--- /dev/null+++ b/tests/unit_tests/test_sky/jobs/test_log_delivery_fallback.py@@ -0,0 +1,46 @@+"""Behavioral checks for managed-job external log delivery fallback."""++from unittest.mock import MagicMock+from unittest.mock import patch++import sky.jobs.controller as controller_module+from sky.jobs.controller import JobController+++def test_keeps_local_logs_when_external_delivery_is_unconfirmed(monkeypatch):+    """A readable store is insufficient when this cluster missed delivery."""+    delivery_source = MagicMock()+    delivery_source.undelivered_reason.return_value = (+        'logging agent was not deployed on the cluster')+    monkeypatch.setattr(controller_module,+                        'LogDeliverySource',+                        delivery_source,+                        raising=False)++    controller = MagicMock(spec=JobController)+    controller._job_id = 1+    controller._backend = MagicMock()+    controller.download_log_and_stream = (+        JobController.download_log_and_stream.__get__(controller,+                                                       JobController))+    handle = MagicMock()+    handle.cluster_name = 'display-name'+    handle.cluster_name_on_cloud = 'provider-name'++    with patch.object(controller_module.logs,+                      'is_logging_agent_configured',+                      return_value=True), \+         patch.object(controller_module.logs,+                      'get_log_reader',+                      return_value=MagicMock()), \+         patch.object(controller_module, 'managed_job_state'), \+         patch.object(controller_module,+                      'managed_job_runtime') as mock_runtime, \+         patch.object(controller_module, 'controller_utils') as mock_utils:+        mock_runtime.is_registered.return_value = False+        mock_utils.download_and_stream_job_log.return_value = '/tmp/run.log'+        controller.download_log_and_stream(0, handle, None)++    delivery_source.undelivered_reason.assert_called_once_with(+        cluster_name='display-name', cluster_name_on_cloud='provider-name')+    mock_utils.download_and_stream_job_log.assert_called_once()