Compare the same open-weight model across inference providers on accuracy, cost, and speed, measured on agentic finance, coding, and legal tasks.
Overall accuracy vs. cost
First-party APIMedian tokens/sec · 50, 200, 400
Benchmark overview
Open-weight models now carry most of the traffic on OpenRouter, which routes each request to one of many inference providers: more than half of all tokens since April, and over 62% since July. Every provider runs the same weights on its own hardware, caching, and tool-call parsing, and those choices change results. Model creators are responding: Moonshot publishes a verifier for third-party Kimi endpoints, and MiniMax published fixes after finding implementations that hurt its model in agent workflows.
The Vals Inference Provider Index runs every provider on the same multi-step agent tasks in finance, coding, and legal, and compares each one, task by task, with the model creator’s own API on accuracy, cost per task, and speed. The same model cost up to 2.3× more depending on the provider, and for all three models the creator’s own API was neither the cheapest nor the fastest option.
Provider results also change over time. Providers ship new inference engines, kernels, quantization schemes, and speculative decoders to cut cost and latency, usually behind the same model name you already call. OpenRouter has reported quality regressions as providers adjusted their stacks, and early provider comparisons for gpt-oss-120b found double-digit accuracy gaps that closed only after vendors fixed their deployments. We rerun the index every two weeks on the same tasks and settings, with a fresh run of each model creator’s API as the baseline, to check whether each gap lasts.
Explore results
Summary
Provider
AccuracyScore ±1 SE
Cost / taskUSD
First tokenMedian · P95
First answerMedian · P95
GenerationMedian tokens/sec
Fireworks
73.0% ± 3.0 pp+2.7 pp vs. first party (within error)
$0.3634% less
2.15s · 8.85s
9.94s · 134.26s
1030.4× first party
DeepSeekFirst party
70.3% ± 3.1 pp
$0.55
1.49s · 4.96s
4.30s · 61.38s
286
Baseten
68.9% ± 3.3 pp−1.5 pp vs. first party (within error)
$0.596% more
1.81s · 11.35s
6.68s · 54.03s
3161.1× first party
Together AI
66.4% ± 3.3 pp−3.9 pp vs. first party (within error)
$0.4814% less
0.43s · 5.93s
5.04s · 42.60s
3981.4× first party
Accuracy: Every provider’s overall accuracy was within error of the creator’s API. The gaps showed up on single benchmarks: Kimi K3 scored 60.6% on Legal through Baseten, against 75.8% on Moonshot’s API, and 87.7% on Coding through Fireworks, against 80.1%.
Cost: Kimi K3 cost $12.65 per task on Baseten, more than twice the $5.43–$5.97 elsewhere, while writing about the same amount. Baseten also served less of Kimi K3’s input from cache: 72%, against 94% on Together AI and Fireworks, and 70% against 98% on Coding. MiniMax M3 cost about half as much on Together AI ($1.01) and Fireworks ($1.07) as on MiniMax’s own API ($2.18).
Speed: Kimi K3 generated a median 229 tokens/sec on Baseten, five times the 45 on Moonshot’s API. Moonshot’s API was also the slowest to respond: a median 7.2s to the first token and 12.6s to the first answer, against 1.8s and 5.5s on Together AI.
Tool calls: Providers had similar tool-call failure rates: for each model and benchmark, they were within about 2 percentage points of each other. Failure rates depended more on the benchmark: roughly 1–6% of calls failed on Finance and Coding, and 6–12% on Legal.
Breakdown
MedianMiddle 50%P5–P95
ProviderSeconds · log scaleMedianP95
DeepSeek
Median1.49 sP954.96 s
Together AI
Median0.43 sP955.93 s
Fireworks
Median2.15 sP958.85 s
Baseten
Median1.81 sP9511.35 s
0.1110100
Methodology
Tasks and settings
Each model–provider comparison covers 165 tasks, a subset of three Vals Index benchmarks. We chose these tasks for how likely they are to surface provider regressions, and sized the set so the error bars stay tight enough to compare providers.
Every provider uses the same agent and task definitions. Sampling and reasoning settings are aligned within each model, with output token budgets set to each provider’s recommended limits.
During runtime, agents sent requests through our gateway, built on model-library, our open-source library for accessing model providers. The gateway records request timings, token usage, and costs. The benchmark ran on Valkyrie, our open-source harness for running agentic benchmarks.
Models and providers
Each model ran on all of its providers at about the same time.
Model
First-party API
Third-party APIs
DeepSeek V4.1 Flash
DeepSeek
Together AI, Fireworks, Baseten
Kimi K3
Moonshot AI
Together AI, Fireworks, Baseten
MiniMax M3
MiniMax
Together AI, Fireworks
MiniMax M3 was unavailable through Baseten’s Model APIs at the time of evaluation.
Accuracy and uncertainty
Finance accuracy averages nine category scores. Coding and Legal accuracy average their task scores. Overall accuracy gives each benchmark equal weight within each model–provider combination. Each subset uses the same accuracy calculation as its full benchmark.
Cost per task averages recorded successful model-request costs over tasks with complete cost records. Overall cost is the equal average of the three benchmark cost averages. Prompt caching was enabled on every provider, either by default or explicitly, and no model changes occurred between runs. Cache hit rate is the share of input tokens a provider served from its prompt cache: cached input tokens divided by cached plus uncached input tokens, summed over every request in the run.
Timing definitions
Other sites may measure metrics with the same names differently, so check these definitions before comparing numbers across sites.
Metric
Definition
First token
Time from the model client’s response timer starting to the first non-empty streamed chunk of reasoning, answer text, or tool-call arguments.
First answer
Time from the same timer starting to the first chunk of answer text or tool-call arguments, excluding reasoning.
Generation speed
Output tokens plus reasoning tokens, divided by the seconds between the first and last streamed chunks. Calculated per request, in tokens per second.
Reasoning
Total duration of recorded reasoning segments, with overlapping periods counted once.
Total time
Duration of a successful model request, including the initial wait and response generation. Earlier failed attempts and retry waits are excluded.
Task duration
Sum of successful model-request durations divided by the number of tasks. This is accumulated model time per task; it excludes tool execution and retry waits.
Tool-call failures
Tool-call failure rate is failed calls divided by recorded attempts, reported separately for Finance, Coding, and Legal. A higher failure rate on one provider can indicate that its serving stack, for example tool-call parsing, is degrading the model’s tool use.
Updates
We will add providers based on demand and replace models as newer ones are released or older ones are deprecated. The task set may change as the Vals Index does. Whenever the tasks or settings change, we rerun every model so results stay comparable.