The collective-protocol choice that cost us 4.4× on prefill
Field note · measured on an 8× GB10 cluster, large sparse-MoE model
Decode was fine. Bandwidth tests were fine. The fabric was at line rate. And yet prefill — the phase that chews through your prompt before the first token comes back — was crawling at 699 tok/s on a cluster whose raw math said it should be far faster. Time-to-first-token on a 10k-token prompt: nearly 15 seconds. Users notice 15 seconds.
Everything the usual playbook says to check came up clean. That's what makes this trap expensive: it lives in a layer most teams treat as a black box — the collective-communication library that moves activations between nodes during tensor-parallel inference.
The mental model that finds it
Collective libraries like NCCL don't have one algorithm for moving data — they have several protocols, chosen dynamically. Each suits a different kind of traffic, and prefill traffic looks nothing like decode traffic.
Here's the trap: the library's automatic choice can be wrong for your message sizes on this fabric — and when it picks wrong for prefill-sized transfers, you pay a tax on every single prefill operation. We measured that tax at 4.4× on the all-reduce path. Nothing errors. Nothing warns. The tokens just come back slower, and every standard dashboard tells you the network is healthy — because it is. The wire is fine; the strategy is wrong.
The signature: decode throughput normal, fabric benchmarks at line rate, but prefill throughput and TTFT far below what the bandwidth math predicts — and the gap grows with prompt length. If that's your cluster, stop debugging the network and start looking at the choices the communication layer makes on its own.
What one line bought
- Prefill: 699 → 1,520 tok/s (2.2×)
- Time-to-first-token at 10k tokens: 14.6 s → 6.7 s
- Decode: unchanged — which is itself the proof the diagnosis was right; this was a prefill-phase disease.
The fix was one configuration line. But which line, with which value, depends on your message-size profile, your fabric, and your framework version — and forcing the wrong protocol makes things worse, which is exactly how most attempts at this setting end. Measuring your actual collective message sizes first is the difference between the 2.2× and a regression.
Prefill slower than the math says it should be?
Book a one-day diagnostic with Anantus. We measure your actual collective sizes and protocol behavior on real GB10 hardware, confirm whether you're paying this tax, and either fix it or tell you exactly how — with the measured before/after TTFT to prove it.
Book a 1-day diagnostic →Free to read and share. Numbers are from hardware we've measured directly; your fabric and framework will differ — measure your own.