Public

NVIDIA/nccl

Updated: 8/13/2026

Languages

C++53.6%Cuda18%C13.6%Python9%Cython2.3%Rust1.6%Other1.9%

3 Models30 Tasks

Optimized primitives for collective multi-GPU communication

		Harness			Input / Output Cost
1	Claude Opus 5	Mini-SWE-agent	14 / 30	$2.96	$5/$25	9m56s
2	GPT-5.6 Sol	Mini-SWE-agent	13 / 30	$2.35	$5/$30	9m13s
3	Muse Spark 1.2	Mini-SWE-agent	11 / 30	$0.67	$1.25/$4.25	6m07s

Key Takeaways

GPT-5.6 Sol with Mini-SWE-agent resolves 13 of 30 tasks, one behind Claude Opus 5, at $2.35 per test and 552.64 seconds.
Muse Spark 1.2 with Mini-SWE-agent resolves 11 of 30 tasks, with the lowest cost at $0.67 per test and latency of 367.22 seconds.
These 30-task results are directional comparisons across the supplied Mini-SWE-agent runs.

Model Comparison

Accuracy

46.67%

Claude Opus 5

43.33%

GPT-5.6 Sol

Task outcomes

30 tasks

Both: 8 tasks

Claude Opus 5 only: 6 tasks

GPT-5.6 Sol only: 5 tasks

Neither: 11 tasks

Both

Claude Opus 5 only

GPT-5.6 Sol only

Neither

Not attempted

Cost / test

$2.96

Claude Opus 5

$2.35

GPT-5.6 Sol

Cost distribution

Latency

9m 56s

Claude Opus 5

9m 13s

GPT-5.6 Sol

Latency distribution

Cost Analysis

Cost / Test vs. Accuracy

ACCURACYCOST

Average Token Use / Test

Token Usage

InputOutputReasoningCache readCache write

Muse Spark 1.2

2.9M

Claude Opus 5

2.7M

GPT-5.6 Sol

2.3M

Cost is the clearest tradeoff in this comparison. Claude Opus 5 leads at 46.67% for $2.96 per test. GPT-5.6 Sol is the lower-cost option at 43.33% for $2.35 per test.

Latency Analysis

Latency vs. Accuracy

ACCURACYLATENCY

Average Response Time / Test

Response Time

Claude Opus 5

9m 56s

GPT-5.6 Sol

9m 13s

Muse Spark 1.2

6m 7s

Latency separates several models with similarly strong scores. Claude Opus 5 leads at 46.67%, while Muse Spark 1.2 is fastest at 6m 7s with 36.67% accuracy.

Tasks with failures

Models
Claude Opus 5
GPT-5.6 Sol
Muse Spark 1.2

Task detail

b91894b

Issue statement

Add a pure-Python ctypes interface for the NCCL Expert Parallel (EP) API under contrib/nccl_ep/python/nccl_ep. The interface must expose low-latency (value 0) and high-throughput (value 1) algorithm constants, tensor tags numbered consecutively from NONE (0) through TOKENS_PER_EXPERTS (7), and ABI-compatible ctypes structures for EP tensors and group configuration. The tensor descriptor fields, in order, are version, ndim, sizes, strides, datatype, data, tag, and flags; the group configuration fields, in order, are version, algorithm, num_experts, max_tokens_per_rank, token_size_bytes, rdma_buffer_size, num_qp_per_rank, and num_channels. Also provide ctypes allocator/free callback types and an NCCLLibrary function registry covering group creation/destruction, handle creation/destruction, dispatch, combine, received-token-count lookup, and completion. Group creation must include the allocator and free callbacks as its final two argument types. Importing these type definitions and inspecting the registry must not require a CUDA device or loading an NCCL shared library.

View Hidden Tests

diff --git a/tests/valsmith/test_nccl_ep_python_api.py b/tests/valsmith/test_nccl_ep_python_api.pynew file mode 100644index 0000000..914681b--- /dev/null+++ b/tests/valsmith/test_nccl_ep_python_api.py@@ -0,0 +1,60 @@+import ctypes++from nccl_ep import nccl_wrapper as ep+++def test_nccl_ep_ctypes_public_contract():+    assert ep.ncclEpAlgorithm_t.NCCL_EP_ALGO_LOW_LATENCY == 0+    assert ep.ncclEpAlgorithm_t.NCCL_EP_ALGO_HIGH_THROUGHPUT == 1+    assert [+        ep.ncclEpTensorTag_t.NCCL_EP_TENSOR_TAG_NONE,+        ep.ncclEpTensorTag_t.NCCL_EP_TENSOR_TAG_TOKENS,+        ep.ncclEpTensorTag_t.NCCL_EP_TENSOR_TAG_TOPK_IDX,+        ep.ncclEpTensorTag_t.NCCL_EP_TENSOR_TAG_TOPK_WEIGHTS,+        ep.ncclEpTensorTag_t.NCCL_EP_TENSOR_TAG_SCALES,+        ep.ncclEpTensorTag_t.NCCL_EP_TENSOR_TAG_RECV_EXPERT_COUNTER_DEVICE,+        ep.ncclEpTensorTag_t.NCCL_EP_TENSOR_TAG_RECV_EXPERT_COUNTER_HOST,+        ep.ncclEpTensorTag_t.NCCL_EP_TENSOR_TAG_TOKENS_PER_EXPERTS,+    ] == list(range(8))++    assert ep.ncclNDTensor_t._fields_ == [+        ("version", ctypes.c_uint),+        ("ndim", ctypes.c_uint),+        ("sizes", ctypes.POINTER(ctypes.c_uint)),+        ("strides", ctypes.POINTER(ctypes.c_uint)),+        ("datatype", ctypes.c_int),+        ("data", ctypes.c_void_p),+        ("tag", ctypes.c_uint),+        ("flags", ctypes.c_int),+    ]+    assert ep.ncclEpGroupConfig_t._fields_ == [+        ("version", ctypes.c_uint),+        ("algorithm", ctypes.c_int),+        ("num_experts", ctypes.c_uint),+        ("max_tokens_per_rank", ctypes.c_uint),+        ("token_size_bytes", ctypes.c_uint),+        ("rdma_buffer_size", ctypes.c_ulong),+        ("num_qp_per_rank", ctypes.c_uint),+        ("num_channels", ctypes.c_uint),+    ]+++def test_nccl_ep_function_table_exposes_required_operations():+    functions = {item.name: item for item in ep.NCCLLibrary.exported_functions}+    expected = {+        "ncclEpCreateGroup",+        "ncclEpGroupDestroy",+        "ncclEpCreateHandle",+        "ncclEpHandleDestroy",+        "ncclEpDispatch",+        "ncclEpCombine",+        "ncclEpHandleGetNumRecvTokens",+        "ncclEpComplete",+    }+    assert set(ep.NCCLLibrary.ep_function_names) == expected+    assert expected <= functions.keys()+    assert functions["ncclEpCreateGroup"].restype is ctypes.c_int+    assert functions["ncclEpCreateGroup"].argtypes[-2:] == [+        ep.ncclEpAllocFn_t,+        ep.ncclEpFreeFn_t,+    ]