poalrom/sharepaste
Languages
Custom repository benchmark.
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 15 / 30 | $3.53 | $5/$25 | 11m10s | ||
2 | Mini-SWE-agent | 11 / 30 | $1.97 | $5/$30 | 7m47s |
Key Takeaways
- Claude Opus 5 with Mini-SWE-agent scores 50% versus GPT-5.6 Sol with Mini-SWE-agent at 36.67%.
- GPT-5.6 Sol with Mini-SWE-agent costs $1.97 per test and takes 466.55 seconds, compared with $3.53 and 670.47 seconds for Claude Opus 5.
- With 30 tasks, these are directional benchmark results rather than a smoke test.
Model Comparison
Accuracy
50.00%
Claude Opus 5
36.67%
GPT-5.6 Sol
Task outcomes
30 tasks
Cost / test
$3.53
Claude Opus 5
$1.97
GPT-5.6 Sol
Cost distribution
Latency
11m 10s
Claude Opus 5
7m 47s
GPT-5.6 Sol
Latency distribution
Cost Analysis
Average Token Use / Test
Cost is the clearest tradeoff in this comparison. Claude Opus 5 leads at 50.00% for $3.53 per test. GPT-5.6 Sol is the lower-cost option at 36.67% for $1.97 per test.
Latency Analysis
Average Response Time / Test
Latency separates several models with similarly strong scores. Claude Opus 5 leads at 50.00%, while GPT-5.6 Sol is fastest at 7m 47s with 36.67% accuracy.
Tasks with failures
| Models | |||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5 | |||||||||||||||||||
| GPT-5.6 Sol |
Task detail
ea00af0Issue statement
The relay needs a single, reusable entry-validation contract. It must calculate the decoded byte size of padded base64 ciphertext, reject incomplete base64 groups and empty ciphertext with HTTP 400, and reject ciphertext larger than the configured byte cap with HTTP 413. Rejection messages must remain suitable for display by clients (malformed base64, empty ciphertext, and ciphertext exceeds maxEntryBytes). Entry path identifiers must be parsed as positive integers, accepting JavaScript numeric forms such as exponent notation while rejecting empty, zero, negative, fractional, non-numeric, and non-finite values with HTTP 400 and bad id. The exposed request-body limit must also be large enough to admit the base64 representation of a maximum-sized entry.
View Hidden Tests
diff --git a/server/tests/unit/valsmith-entry-rules.test.ts b/server/tests/unit/valsmith-entry-rules.test.tsnew file mode 100644index 0000000..8f6c533--- /dev/null+++ b/server/tests/unit/valsmith-entry-rules.test.ts@@ -0,0 +1,43 @@+import { describe, expect, it } from "vitest";+import { entryRules } from "../../src/server/refusal.js";++const verdict = (act: () => number) => {+ try {+ return { value: act() };+ } catch (error) {+ const refusal = error as { statusCode?: number; message?: string };+ return { status: refusal.statusCode, reason: refusal.message };+ }+};++describe("entry validation rules", () => {+ it("computes decoded base64 byte sizes including padding", () => {+ const rules = entryRules({ maxEntryBytes: 16 });+ expect(rules.sizeOf("AAAA")).toBe(3);+ expect(rules.sizeOf("AAA=")).toBe(2);+ expect(rules.sizeOf("AA==")).toBe(1);+ });++ it("rejects malformed, empty, and oversized ciphertext with stable client errors", () => {+ const rules = entryRules({ maxEntryBytes: 3 });+ expect(verdict(() => rules.sizeOf("AAA"))).toEqual({ status: 400, reason: "malformed base64" });+ expect(verdict(() => rules.sizeOf(""))).toEqual({ status: 400, reason: "empty ciphertext" });+ expect(verdict(() => rules.sizeOf("AAAAAA=="))).toEqual({+ status: 413,+ reason: "ciphertext exceeds maxEntryBytes",+ });+ });++ it("accepts only positive integer entry identifiers", () => {+ const rules = entryRules({ maxEntryBytes: 16 });+ expect(rules.idFrom("1e2")).toBe(100);+ for (const raw of ["", "0", "-1", "1.5", "abc", "1e400"]) {+ expect(verdict(() => rules.idFrom(raw))).toEqual({ status: 400, reason: "bad id" });+ }+ });++ it("sets the request body limit above base64 expansion at the entry cap", () => {+ const rules = entryRules({ maxEntryBytes: 64 * 1024 });+ expect(rules.bodyLimit).toBeGreaterThan(Math.ceil((rules.maxEntryBytes / 3) * 4));+ });+});