chrisanderton/tyk-gateway
Languages
Tyk Open Source API Gateway written in Go, supporting REST, GraphQL, TCP and gRPC protocols
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 23 / 30 | $1.42 | $5/$30 | 5m38s | ||
2 | Mini-SWE-agent | 23 / 30 | $3.00 | $5/$25 | 10m40s | ||
3 | Mini-SWE-agent | 22 / 30 | $3.21 | $5/$25 | 10m45s | ||
4 | Mini-SWE-agent | 20 / 30 | $0.49 | $1.4/$4.4 | 7m26s |
Key Takeaways
- GPT-5.6 Sol matches Claude Opus 5 while reporting lower cost per test ($1.42 versus $3.00) and lower latency (337.56 versus 639.94 seconds) with Mini-SWE-agent.
- Claude Opus 4.8 resolves 22 of 30 tasks for 73.33% with Mini-SWE-agent, while GLM 5.2 (Fireworks) resolves 20 for 66.67%; these 30-task results are directional.
Model Comparison
Accuracy
76.67%
GPT-5.6 Sol
76.67%
Claude Opus 5
Task outcomes
30 tasks
Cost / test
$1.42
GPT-5.6 Sol
$3.00
Claude Opus 5
Cost distribution
Latency
5m 38s
GPT-5.6 Sol
10m 40s
Claude Opus 5
Latency distribution
Cost Analysis
Average Token Use / Test
Cost is the clearest tradeoff in this comparison. Claude Opus 5 leads at 76.67% for $3.00 per test. GPT-5.6 Sol is the lower-cost option at 76.67% for $1.42 per test.
Latency Analysis
Average Response Time / Test
Latency separates several models with similarly strong scores. Claude Opus 5 leads at 76.67%, while GPT-5.6 Sol is fastest at 5m 38s with 76.67% accuracy.
Tasks with failures
| Models | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5.6 Sol | ||||||||||
| Claude Opus 5 | ||||||||||
| Claude Opus 4.8 | ||||||||||
| GLM 5.2 |
Task detail
5ef54e3Issue statement
GraphQL analytics must correctly handle upstream responses compressed with gzip. When a GraphQL client advertises gzip via Accept-Encoding and the upstream response carries Content-Encoding: gzip, the gateway should extract the same GraphQL statistics it does for an uncompressed response, without emitting a body decompression error. Reading the response for analytics must also leave the decoded response body available for the remainder of response processing.
View Hidden Tests
diff --git a/gateway/handler_success_gzip_test.go b/gateway/handler_success_gzip_test.gonew file mode 100644index 00000000..ad629835--- /dev/null+++ b/gateway/handler_success_gzip_test.go@@ -0,0 +1,49 @@+package gateway++import (+ "bytes"+ "compress/gzip"+ "io"+ "net/http"+ "strings"+ "testing"++ "github.com/TykTechnologies/tyk-pump/analytics"+ "github.com/TykTechnologies/tyk/apidef"+ "github.com/stretchr/testify/require"+)++func TestRecordGraphDetailsHandlesGzipResponse(t *testing.T) {+ requestBody := `{ "query": "{ hello(name: \"World\") httpMethod }" }`+ responseBody := `{"data":{"hello":"World","httpMethod":"POST"}}`++ var compressed bytes.Buffer+ writer := gzip.NewWriter(&compressed)+ _, err := writer.Write([]byte(responseBody))+ require.NoError(t, err)+ require.NoError(t, writer.Close())++ req, err := http.NewRequest(http.MethodPost, "http://gateway/", strings.NewReader(requestBody))+ require.NoError(t, err)+ req.Header.Set("Accept-Encoding", "gzip")+ resp := &http.Response{+ Header: http.Header{"Content-Encoding": []string{"gzip"}},+ Body: io.NopCloser(bytes.NewReader(compressed.Bytes())),+ }+ spec := &APISpec{APIDefinition: &apidef.APIDefinition{GraphQL: apidef.GraphQLConfig{+ Enabled: true,+ ExecutionMode: apidef.GraphQLExecutionModeProxyOnly,+ Version: apidef.GraphQLConfigVersion2,+ Schema: gqlProxyUpstreamSchema,+ }}}+ record := &analytics.AnalyticsRecord{}++ recordGraphDetails(record, req, resp, spec)++ require.True(t, record.GraphQLStats.IsGraphQL)+ require.False(t, record.GraphQLStats.HasErrors)+ require.ElementsMatch(t, []string{"hello", "httpMethod"}, record.GraphQLStats.RootFields)+ restoredBody, err := io.ReadAll(resp.Body)+ require.NoError(t, err)+ require.JSONEq(t, responseBody, string(restoredBody))+}