apache/spark
Languages
Apache Spark - A unified analytics engine for large-scale data processing
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 | 49 / 58 | $2.46 | $5/$30 | 9m48s | |||
2 | 48 / 58 | $7.44 | $10/$50 | 16m48s | |||
3 | 48 / 58 | $4.55 | $5/$25 | 17m18s | |||
4 | 47 / 58 | $0.71 | $2/$6 | 3m47s | |||
5 | 47 / 58 | $0.88 | $1.4/$4.4 | 15m39s | |||
6 | 47 / 58 | $9.82 | $3/$15 | 16m32s | |||
7 | 45 / 58 | $4.96 | $3/$15 | 21m11s | |||
8 | 44 / 58 | $4.92 | $5/$25 | 13m26s | |||
9 | 41 / 58 | $0.12 | $0.2/$1.2 | 5m21s | |||
10 | 38 / 58 | $0.48 | $2/$12 | 3m30s | |||
11 | 38 / 58 | $1.26 | $1.5/$9 | 5m14s | |||
12 | 38 / 58 | $2.02 | $5/$30 | 6m19s | |||
13 | 33 / 58 | $0.70 | $2/$12 | 3m23s | |||
14 | 33 / 58 | $0.41 | $1/$5 | 4m36s |
Key Takeaways
- Claude Fable 5 and Claude Opus 4.8 with Mini-SWE-agent each resolve 48 of 58 tasks (82.76%), while GLM 5.2 (Fireworks), Kimi K3, and Grok 4.5 each resolve 47.
- Grok 4.5 with Mini-SWE-agent resolves 47 of 58 tasks for $0.71 per test in 226.98 seconds, compared with GPT-5.6 Sol at $2.46 and 588.17 seconds.
Model Comparison
Accuracy
84.48%
GPT-5.6 Sol
82.76%
Claude Opus 4.8
Task outcomes
58 tasks
Cost / test
$2.46
GPT-5.6 Sol
$4.55
Claude Opus 4.8
Cost distribution
Latency
9m 48s
GPT-5.6 Sol
17m 18s
Claude Opus 4.8
Latency distribution
Cost Analysis
Average Token Use / Test
Cost is the clearest tradeoff in this comparison. GPT-5.6 Sol leads at 84.48% for $2.46 per test. Grok 4.5 is the lower-cost option at 81.03% for $0.71 per test.
Latency Analysis
Average Response Time / Test
Latency separates several models with similarly strong scores. GPT-5.6 Sol leads at 84.48%, while Gemini 3.1 Pro Preview (02/26) is fastest at 3m 23s with 56.90% accuracy.
Tasks with failures
| Models | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5.6 Sol | ||||||||||||||||||||||||||||||||
| Claude Opus 4.8 | ||||||||||||||||||||||||||||||||
| Claude Fable 5 | ||||||||||||||||||||||||||||||||
| Grok 4.5 | ||||||||||||||||||||||||||||||||
| GLM 5.2 | ||||||||||||||||||||||||||||||||
| Kimi K3 | ||||||||||||||||||||||||||||||||
| Claude Sonnet 5 | ||||||||||||||||||||||||||||||||
| Claude Opus 4.7 | ||||||||||||||||||||||||||||||||
| GPT-5.6 Luna | ||||||||||||||||||||||||||||||||
| GPT-5.6 Terra | ||||||||||||||||||||||||||||||||
| Gemini 3.5 Flash | ||||||||||||||||||||||||||||||||
| GPT 5.5 | ||||||||||||||||||||||||||||||||
| Claude Haiku 4.5 (Nonthinking) | ||||||||||||||||||||||||||||||||
| Gemini 3.1 Pro Preview (02/26) |
Task detail
fe0e502Issue statement
Correlated scalar and lateral subqueries reference columns from the enclosing query through outer references. When such an outer reference appears as a bare, top-level output column of the subquery's Project or Aggregate (for example SELECT (SELECT a) FROM t1, or a lateral subquery whose GROUP BY produces the grouping column directly), the analyzer currently leaves it as a bare OuterReference in the operator's output list.
Because OuterReference.toAttribute strips the outer wrapper, the attribute exposed through Project.output / Aggregate.output reuses the outer attribute's ExprId. That outer expression id then leaks into the subquery's scope, so downstream operators that reference the subquery's output end up sharing an expression id with an attribute from the outer plan. This breaks the invariant that each operator produces attributes with fresh, self-contained expression ids and can lead to incorrect expression-id tracking.
Expected behavior: when the resolved output of a subquery's Project or Aggregate is a bare top-level OuterReference (i.e. a NamedExpression that is exactly an OuterReference, not already wrapped in an Alias), the analyzer should wrap it in an Alias so that the output attribute gets a fresh ExprId while still carrying the original column name. Outputs that are already an Alias (such as those produced by struct star expansion or explicit user aliases) must be left unchanged. The alias name should match the referenced column's name, and the wrapped OuterReference must keep pointing at the original outer attribute (same ExprId on the inner OuterReference) while the surrounding Alias carries a new ExprId.
View Hidden Tests
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveSubquerySuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveSubquerySuite.scalaindex 30ef9532..48286359 100644--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveSubquerySuite.scala+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveSubquerySuite.scala@@ -254,7 +254,8 @@ class ResolveSubquerySuite extends AnalysisTest { // SELECT (SELECT a) FROM t1 checkAnalysis( Project(ScalarSubquery(t0.select($"a")).as("sub") :: Nil, t1),- Project(ScalarSubquery(Project(OuterReference(a) :: Nil, t0), Seq(a)).as("sub") :: Nil, t1)+ Project(ScalarSubquery(+ Project(OuterReference(a).as(a.name) :: Nil, t0), Seq(a)).as("sub") :: Nil, t1) ) // SELECT (SELECT a + b + c AS r FROM t2) FROM t1 checkAnalysis(@@ -274,6 +275,27 @@ class ResolveSubquerySuite extends AnalysisTest { ) } + test("SPARK-58171: alias a bare outer reference that is an Aggregate output") {+ // SELECT * FROM t1, LATERAL (SELECT a, count(b) AS cnt FROM t2 GROUP BY a)+ // The bare outer reference `a` in the Aggregate output must be aliased so it gets a fresh+ // ExprId, instead of leaking the outer attribute's ExprId through the Aggregate output.+ // The query is only resolved (not passed through CheckAnalysis) because a correlated+ // aggregate output is otherwise rejected as an unsupported correlated reference.+ val plan = lateralJoin(t1, t2.groupBy($"a")($"a", Count($"b").as("cnt")))+ val resolved = getAnalyzer.execute(plan)+ val aggregate = resolved.collectWithSubqueries { case a: Aggregate => a }.head+ // The first output is the aliased outer reference; the outer attribute's ExprId is not+ // leaked through the alias (the alias carries a fresh ExprId).+ aggregate.aggregateExpressions.head match {+ case alias @ Alias(o: OuterReference, name) =>+ assert(name == a.name)+ assert(o.exprId == a.exprId)+ assert(alias.exprId != a.exprId)+ case other =>+ fail(s"Expected the outer reference to be wrapped in an Alias, but got: $other")+ }+ }+ test("SPARK-47509: Incorrect results for subquery expressions in LambdaFunctions") { val data = LocalRelation(Seq( $"key".int,