GPUDirect RDMA doesn't work on GB10 — the cost, and the way around it
Field note · measured on an 8× GB10 cluster
You try to bring up multi-node inference, and either NCCL hangs silently at init with no error, or it runs but moves data at a small fraction of what your 100 Gbps link should deliver. You check the cables, the MTU, the switch — all fine. It's one of the most consequential and least-understood problems on GB10 multi-node, and here's why it's so slippery.
The zero-copy path you expect doesn't exist here
On datacenter GPUs, GPUDirect RDMA lets the network card read and write GPU memory directly — no CPU, no copy. On GB10 the CPU and GPU share one unified memory pool and there's no discrete GPU-attached HBM for the NIC to target that way, so GPUDirect RDMA is architecturally unavailable. NVIDIA has confirmed this. The consequence: multi-node collectives have to stage through host memory, which is slower and — worse — some default NCCL configurations assume the zero-copy path exists and silently hang when it isn't there.
The signature: a multi-node run that hangs at init with no traceback, or completes but delivers well under half your link's bandwidth in collectives — while single-node runs and raw point-to-point bandwidth tests both look fine. That gap between "the wire is fast" and "the collectives are slow" is the fingerprint of the missing zero-copy path.
It's not fatal — it's a configuration problem with a real answer
The link genuinely can move data well; the collective library just has to be told to use a path that exists on this architecture instead of the one it assumes. There's a set of environment settings that gets NCCL onto a working staged path and recovers a large share of the bandwidth — we've measured the before and after. But the defaults are wrong for GB10, several of the settings interact, and the wrong combination trades a silent hang for slow-but-working (or a different silent hang). This is exactly the kind of thing that reads as "GB10 multi-node is broken" when it's actually "GB10 multi-node needs to be told the truth about its own memory."
What this means for your cluster
- Don't chase the network hardware. Raw bandwidth tests passing while collectives crawl points at the staging path, not the wire.
- The default NCCL config is not safe to assume on GB10. What works on an H100 pod can hang here.
- The recoverable bandwidth is substantial — enough to move a multi-node model from unusable to serving — once the collectives take a path the architecture actually supports.
NCCL hanging or crawling on your multi-node bring-up?
This is the GB10 wall almost nobody has documented. We reproduce your multi-node setup on real hardware, get the collectives onto a working path, and show you the measured bandwidth recovery — or it's free.
Book a 1-day diagnostic →Free to read and share. Numbers are from hardware we've measured directly; your versions and topology will differ — measure your own.