Public

BerriAI/litellm

Updated: 8/13/2026

Languages

Python84.5%TypeScript12.6%HTML1.5%Rust0.6%HCL0.2%Go0.2%Other0.4%

2 Models30 Tasks

The fastest, litest AI Gateway. Rust core with Python SDK. Call 100+ LLM APIs in OpenAI (or native) format with cost tracking, guardrails, load balancing, and logging [Bedrock, Azure, OpenAI, Anthropic, OpenAI, VertexAI, vLLM, Nvidia NIM]

Add models

		Harness			Input / Output Cost
1	Gemini 3.6 Flash	Mini-SWE-agent	21 / 30	$1.95	$1.5/$7.5	12m55s
2	GLM 5.2	Mini-SWE-agent	19 / 30	$0.73	$1.4/$4.4	11m29s

Key Takeaways

Gemini 3.6 Flash with Mini-SWE-agent scores 70%, versus 63.33% for GLM 5.2 (Fireworks) with Mini-SWE-agent.
GLM 5.2 (Fireworks) with Mini-SWE-agent costs $0.73 per test and takes 689.22 seconds, compared with $1.95 and 775.49 seconds for Gemini 3.6 Flash with Mini-SWE-agent.

Model Comparison

Accuracy

70.00%

Gemini 3.6 Flash

63.33%

GLM 5.2

Task outcomes

30 tasks

Both: 16 tasks

Gemini 3.6 Flash only: 5 tasks

GLM 5.2 only: 3 tasks

Neither: 6 tasks

Both

Gemini 3.6 Flash only

GLM 5.2 only

Neither

Not attempted

Cost / test

$1.95

Gemini 3.6 Flash

$0.73

GLM 5.2

Cost distribution

Latency

12m 55s

Gemini 3.6 Flash

11m 29s

GLM 5.2

Latency distribution

Cost Analysis

Cost / Test vs. Accuracy

ACCURACYCOST

Average Token Use / Test

Token Usage

InputOutputReasoningCache readCache write

No token usage data available.

Cost is the clearest tradeoff in this comparison. Gemini 3.6 Flash leads at 70.00% for $1.95 per test. GLM 5.2 is the lower-cost option at 63.33% for $0.73 per test.

Latency Analysis

Latency vs. Accuracy

ACCURACYLATENCY

Average Response Time / Test

Response Time

Gemini 3.6 Flash

12m 55s

GLM 5.2

11m 29s

Latency separates several models with similarly strong scores. Gemini 3.6 Flash leads at 70.00%, while GLM 5.2 is fastest at 11m 29s with 63.33% accuracy.

Tasks with failures

Models
Gemini 3.6 Flash
GLM 5.2

Task detail

3d76dfc

Issue statement

Expose a keepalive timing helper for streaming error handling. Given elapsed time and the configured SSE ping interval, it must report whether a keepalive ping has already fired and therefore flushed response headers. The boundary is inclusive: elapsed time equal to the interval counts as fired. Numeric string intervals must behave like numeric values, while disabled, non-positive, non-finite, or otherwise invalid intervals must report that no ping has fired.

View Hidden Tests

diff --git a/tests/test_litellm/proxy/common_utils/test_sse_keepalive_fired.py b/tests/test_litellm/proxy/common_utils/test_sse_keepalive_fired.pynew file mode 100644index 0000000000..5681647fc2--- /dev/null+++ b/tests/test_litellm/proxy/common_utils/test_sse_keepalive_fired.py@@ -0,0 +1,18 @@+import math++from litellm.proxy.common_utils import sse_keepalive+++def test_keepalive_ping_fired_respects_boundary_and_disabled_intervals():+    helper = getattr(sse_keepalive, "keepalive_ping_has_fired", None)++    assert callable(helper)+    assert helper(elapsed_seconds=9.99, ping_interval_seconds=10) is False+    assert helper(elapsed_seconds=10, ping_interval_seconds=10) is True+    assert helper(elapsed_seconds=10.01, ping_interval_seconds="10") is True+    assert helper(elapsed_seconds=100, ping_interval_seconds=None) is False+    assert helper(elapsed_seconds=100, ping_interval_seconds=0) is False+    assert helper(elapsed_seconds=100, ping_interval_seconds=-1) is False+    assert helper(elapsed_seconds=100, ping_interval_seconds="not-a-number") is False+    assert helper(elapsed_seconds=100, ping_interval_seconds=math.inf) is False+    assert helper(elapsed_seconds=100, ping_interval_seconds=math.nan) is False