Public

conjfrnk/memovox

Updated: 8/13/2026

Languages

Python97.9%JavaScript2%Makefile0.1%Dockerfile<0.1%
2 Models30 Tasks

A local-first multimodal video-to-knowledge engine.

Harness

1

Mini-SWE-agent
27 / 30

$2.81

11m05s

2

Mini-SWE-agent
25 / 30

$0.06

4m17s

Key Takeaways

  • Claude Opus 5 with Mini-SWE-agent scores 90%, compared with 83.33% for GPT-5.6 Luna with Mini-SWE-agent.
  • GPT-5.6 Luna with Mini-SWE-agent costs $0.06 per test and takes 257.46 seconds, versus $2.81 and 665.35 seconds for Claude Opus 5.

Model Comparison

Accuracy

90.00%

Claude Opus 5

83.33%

GPT-5.6 Luna

Task outcomes

30 tasks

Both
Claude Opus 5 only
GPT-5.6 Luna only
Neither
Not attempted

Cost / test

$2.81

Claude Opus 5

$0.06

GPT-5.6 Luna

Cost distribution

$0.00$3.40$6.80

Latency

11m 5s

Claude Opus 5

4m 17s

GPT-5.6 Luna

Latency distribution

0s13m 23s26m 47s

Cost Analysis

Cost / Test vs. Accuracy
ACCURACYCOST

Average Token Use / Test

Token Usage
InputOutputReasoningCache readCache write
Claude Opus 5
2.6M
GPT-5.6 Luna
992K

Cost is the clearest tradeoff in this comparison. Claude Opus 5 leads at 90.00% for $2.81 per test. GPT-5.6 Luna is the lower-cost option at 83.33% for $0.06 per test.

Latency Analysis

Latency vs. Accuracy
ACCURACYLATENCY

Average Response Time / Test

Response Time
Claude Opus 5
11m 5s
GPT-5.6 Luna
4m 17s

Latency separates several models with similarly strong scores. Claude Opus 5 leads at 90.00%, while GPT-5.6 Luna is fastest at 4m 17s with 83.33% accuracy.

Tasks with failures

Models
Claude Opus 5
GPT-5.6 Luna

Task detail

0287abe

Issue statement

Harden two answer-validation paths against unusually large or international inputs. The relevance topicality gate must place a deterministic, conservative upper bound on the number of per-token document-frequency lookups it performs, so a query containing hundreds or thousands of distinct out-of-corpus terms cannot trigger one database probe per term. Normal-sized queries and title-token matches must retain their behavior. Also, citation validation must recognize a new sentence after an abbreviation when the following capitalized word begins with a non-ASCII uppercase/titlecase letter (for example accented Latin, Cyrillic, or Greek), while continuing to treat initials and titles such as U.S. and Mr. correctly.

View Hidden Tests
diff --git a/tests/test_valsmith_round11.py b/tests/test_valsmith_round11.pynew file mode 100644index 0000000..f7ff175--- /dev/null+++ b/tests/test_valsmith_round11.py@@ -0,0 +1,50 @@+"""Task-local regression tests for answer-gate hardening."""++from memovox.augur import answer+from memovox.augur.types import Citation+++def _citation() -> Citation:+    return Citation(+        index=1,+        video_id="video",+        moment_id="video#m1",+        t_start_s=0,+        t_end_s=1,+        source_text="proteins decline",+    )+++def test_citation_gate_recognizes_unicode_sentence_starts():+    citation = [_citation()]+    unsafe = (+        "Napoleon invented ribosomes etc. Élément cites proteins [1]",+        "Napoleon made them etc. Авторы note a decline [1]",+        "Napoleon made them etc. States note a decline [1]",+    )+    assert all(not answer._llm_citations_valid(text, citation) for text in unsafe)++    safe = (+        "The U.S. economy grew [1].",+        "Mr. Jones disagreed [1].",+        "Les ribosomes synthétisent des protéines [1].",+    )+    assert all(answer._llm_citations_valid(text, citation) for text in safe)+++def test_relevance_gate_caps_document_frequency_probes():+    class CountingStore:+        def __init__(self):+            self.probes = []++        def stats(self):+            return {"moments": 100_000}++        def doc_freq(self, token):+            self.probes.append(token)+            return 0++    store = CountingStore()+    query = " ".join(f"unmatched{i}" for i in range(500))+    assert answer._relevance_coverage(store, query, [_citation()]) == 0.0+    assert len(store.probes) <= answer._REL_TOPICALITY_MAX <= 1000