Your GB10 RDMA isn't broken at 94 Gbps on a 100G link — that's the ceiling
Field note · measured on an 8× GB10 cluster over a 100 Gbps fabric
If you've cabled two Grace-Blackwell boxes together over a 100 Gbps switch, and watched a single flow top out around 93–94 instead of a clean 100 — you probably filed a bug or started tearing your stack apart. Don't. That number is not a regression. It's the fabric telling you the truth, and the real problem (if there is one) is somewhere else entirely.
Your ceiling is your networking gear, not the GB10
This is the mental model that saves you a week: the RDMA ceiling is set by your switch and cables, not by the Grace-Blackwell silicon. On a 100 Gbps switch with 100 Gbps DAC cables, 100 is the wall — and you never reach the full wall on RDMA anyway. Headers, line encoding, and protocol overhead cost you roughly 6–7%, so a healthy one-way flow on a 100G link lands around 93–94 Gbps. That's line rate, in practice. Someone on 200G gear would see roughly double; someone on 400G, double again. The GB10 is not what's holding you back.
Two corollaries that trip people up:
- The rails do not aggregate into one fat pipe for a single stream. Expecting two rails to sum is the most common wrong assumption we see — a single flow rides a single rail.
- "100 Gbps" on the NIC is a link rating, not a single-flow guarantee. ~94 one-way on a 100G link is the win condition, not a shortfall.
Rule of thumb: a single one-way flow at ~94 on a 100G fabric with sane latency is healthy — that's your gear at line rate. If you want more, the fix is faster switches and cables, not debugging the GB10.
First, find out if it's actually a problem at all
Before you touch anything, measure a single one-way flow on one rail with a standard RDMA perftest and compare it to your link's line rate. On a 100G fabric, a healthy result sits in the low-to-mid 90s Gbps. If that's what you see, stop — your fabric is fine and whatever's slow lives upstream in the collective library or the framework, not on the wire. Chasing the network here is a week you'll never get back.
The interesting case is when a single flow comes in far below line rate — say, down around 13 Gbps on a 100G link. That's not the GB10 and it's not "just overhead." It's almost always one of two specific, fixable misconfigurations:
- An MTU clamp somewhere in the path. Your NICs can be set perfectly and it won't matter — the smallest link in the chain wins, and a single wrong default several hops away silently drops you to small frames. The trap is that what you configured and what the fabric actually negotiated are two different values, and only one of them is the one that counts. Most people never check the one that counts.
- A rail assumption. The rails do not sum into one pipe for a single stream, so "I have two rails, I should get double" quietly sends you down the wrong path — sometimes literally onto the wrong rail.
Which one it is, and exactly where in the path it's being clamped, is what turns a week of bisecting into a ten-minute fix — once you know which single value to read and what to set it to. That part is specific to your switch model, firmware, and cabling, and getting it wrong just moves the clamp somewhere else.
The short version: ~94 on a 100G link = healthy, look elsewhere. Down near 13 = an MTU or rail misconfig, not your hardware. Which one, and where, is the whole ballgame.
Still stuck below line rate?
If this didn't fix it, book a one-day diagnostic with Anantus. We reproduce your exact multi-node setup on real GB10 hardware, find where the clamp actually is, and either fix it or tell you precisely how — with the measured before/after to prove it.
Book a 1-day diagnostic →Free to read and share. Numbers are from hardware we've measured directly; your fabric will differ with firmware, cabling, and switch model — measure your own.