Why tensor parallelism falls apart past a couple of DGX Spark nodes
Field note · measured across 1–8 GB10 nodes
You cabled two Sparks together, ran tensor-parallel, and it worked. You added two more and engine init started failing — or it launched but throughput went down. This is one of the most common multi-node walls on GB10 — TP=2 works, TP=4 fails or slows down, some stacks don't recognize the extra nodes at all — and it's widely reported with no clean answer. Here's the shape of the problem.
Tensor parallelism is the most bandwidth-hungry way to split a model
TP shards every layer across every node and synchronizes twice per layer, per token with all-reduce collectives. That's fine inside one box with terabytes/sec of internal bandwidth. Across a 100 Gbps switch it means the network is on the critical path of every single token — and the cost doesn't stay flat as you add nodes, it grows. Past a certain node count the collective time dominates and each additional Spark makes single-stream decode slower, not faster. We've measured exactly where that crossover sits on this fabric.
The mental model: more TP nodes = more synchronization per token = more network on the critical path. TP scales capacity (fit a bigger model) but actively fights single-stream speed past a small node count. If you added nodes and got slower, that's not a bug — that's TP behaving exactly as designed on a modest interconnect.
Pipeline parallelism is often the right answer — and nobody tells you
Pipeline parallelism splits the model by layers across nodes and passes activations forward once, instead of synchronizing every layer. It trades single-request latency for far less network chatter, and on batch workloads it can beat TP by a wide margin on this hardware — a result even the professional reviews that stopped at two nodes only hinted at. The catch: PP has its own traps (bubble scheduling, where the barrier actually lives, which framework paths are wired for GB10 at all), and mixing TP-within-node and PP-across-node is a third regime with its own tuning.
Why "just add nodes" is the wrong instinct
- The right node count is set by the model, not your ambition. A model that fits in two boxes gains almost nothing on four — you've added synchronization cost for capacity you didn't need.
- Which parallelism, at which boundary, is the whole game. TP inside a node where bandwidth is cheap; PP across nodes where it's expensive — the exact split depends on your model's shape, layer count, and your switch.
- The framework may simply not support your topology. Several multi-node paths silently don't recognize GB10 as a valid device; that's a known wall, not your misconfiguration.
Getting a model to serve well across four or eight Sparks is a topology-design problem, not a "turn TP up" problem — and the design that wins is specific to your model and your fabric. Guess it wrong and you'll spend a week watching throughput go the wrong direction.
Stuck scaling past two Sparks?
This is the exact wall we specialize in. A diagnostic reproduces your multi-node setup on our 8-node lab, finds the topology that actually serves your model, and hands you the measured before/after — or it's free.
Book a 1-day diagnostic →Free to read and share. Numbers are from hardware we've measured directly; your versions and topology will differ — measure your own.