conjfrnk/memovox
Languages
A local-first multimodal video-to-knowledge engine.
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 | 27 / 30 | $2.81 | $5/$25 | 11m05s | |||
2 | 25 / 30 | $0.06 | $0.2/$1.2 | 4m17s |
Key Takeaways
- Claude Opus 5 with Mini-SWE-agent scores 90%, compared with 83.33% for GPT-5.6 Luna with Mini-SWE-agent.
- GPT-5.6 Luna with Mini-SWE-agent costs $0.06 per test and takes 257.46 seconds, versus $2.81 and 665.35 seconds for Claude Opus 5.
Model Comparison
Accuracy
90.00%
Claude Opus 5
83.33%
GPT-5.6 Luna
Task outcomes
30 tasks
Cost / test
$2.81
Claude Opus 5
$0.06
GPT-5.6 Luna
Cost distribution
Latency
11m 5s
Claude Opus 5
4m 17s
GPT-5.6 Luna
Latency distribution
Cost Analysis
Average Token Use / Test
Cost is the clearest tradeoff in this comparison. Claude Opus 5 leads at 90.00% for $2.81 per test. GPT-5.6 Luna is the lower-cost option at 83.33% for $0.06 per test.
Latency Analysis
Average Response Time / Test
Latency separates several models with similarly strong scores. Claude Opus 5 leads at 90.00%, while GPT-5.6 Luna is fastest at 4m 17s with 83.33% accuracy.
Tasks with failures
| Models | |||||
|---|---|---|---|---|---|
| Claude Opus 5 | |||||
| GPT-5.6 Luna |
Task detail
0287abeIssue statement
Harden two answer-validation paths against unusually large or international inputs. The relevance topicality gate must place a deterministic, conservative upper bound on the number of per-token document-frequency lookups it performs, so a query containing hundreds or thousands of distinct out-of-corpus terms cannot trigger one database probe per term. Normal-sized queries and title-token matches must retain their behavior. Also, citation validation must recognize a new sentence after an abbreviation when the following capitalized word begins with a non-ASCII uppercase/titlecase letter (for example accented Latin, Cyrillic, or Greek), while continuing to treat initials and titles such as U.S. and Mr. correctly.
View Hidden Tests
diff --git a/tests/test_valsmith_round11.py b/tests/test_valsmith_round11.pynew file mode 100644index 0000000..f7ff175--- /dev/null+++ b/tests/test_valsmith_round11.py@@ -0,0 +1,50 @@+"""Task-local regression tests for answer-gate hardening."""++from memovox.augur import answer+from memovox.augur.types import Citation+++def _citation() -> Citation:+ return Citation(+ index=1,+ video_id="video",+ moment_id="video#m1",+ t_start_s=0,+ t_end_s=1,+ source_text="proteins decline",+ )+++def test_citation_gate_recognizes_unicode_sentence_starts():+ citation = [_citation()]+ unsafe = (+ "Napoleon invented ribosomes etc. Élément cites proteins [1]",+ "Napoleon made them etc. Авторы note a decline [1]",+ "Napoleon made them etc. States note a decline [1]",+ )+ assert all(not answer._llm_citations_valid(text, citation) for text in unsafe)++ safe = (+ "The U.S. economy grew [1].",+ "Mr. Jones disagreed [1].",+ "Les ribosomes synthétisent des protéines [1].",+ )+ assert all(answer._llm_citations_valid(text, citation) for text in safe)+++def test_relevance_gate_caps_document_frequency_probes():+ class CountingStore:+ def __init__(self):+ self.probes = []++ def stats(self):+ return {"moments": 100_000}++ def doc_freq(self, token):+ self.probes.append(token)+ return 0++ store = CountingStore()+ query = " ".join(f"unmatched{i}" for i in range(500))+ assert answer._relevance_coverage(store, query, [_citation()]) == 0.0+ assert len(store.probes) <= answer._REL_TOPICALITY_MAX <= 1000