Fabric & multi-node
Where "distributed" quietly becomes "slow."
A single RDMA flow runs at ~13 Gbps on a 100G link.
Not your NICs and not the GB10 — a mismatch somewhere in the path that can cost 85% of your bandwidth while every dashboard stays green.
"I have two rails, so I should see double the bandwidth."
A topology assumption that sends planning (and cable budgets) down the wrong path. Worth knowing before you buy more hardware.
Prefill is several times slower than the math says it should be.
A silent strategy choice inside the communication layer — a
multi-× tax on every long prompt, with every network test passing.
Full note →Multi-node launch "hangs loading weights" forever.
Often not a loading problem at all — the nodes are stuck
before loading starts. Debugging the loader here wastes the day.
Full note →A healthy 100G link "only" reaches ~94 Gbps.
That's line rate after protocol overhead.
Stop debugging — the slowness you're chasing lives upstream.
Full note →
Model loading
Minutes-long stalls that look like "big model, be patient." They aren't.
Weights crawl to the GPU at hundreds of MB/s instead of gigabytes.
Not your disk and not your model — the copy path itself. A one-line-class fix once identified: we took a load from 7.5 minutes to 35 seconds.
One CPU core pinned at 100%, nineteen idle, GPUs idle, load takes ~8 minutes.
The loader itself is the wall. Storage upgrades and resharding won't help.
Random ranks stall at engine init; every launch a different node is "the slow one."
One rank drags every node down — the whole cluster waits for its worst member, and restarts only move the problem around.
Eight nodes loading from one NFS export takes forever.
A distribution-pattern problem, not a storage-speed one. A faster NFS server won't fix it.
After big reads, "used" RAM looks maxed and you're afraid to load.
Much of that memory isn't really spoken for. Knowing what's recoverable can free tens of GB you didn't know you had.
The full triage: three loading hangs that look identical.
Unified memory
CPU and GPU share one pool. Every intuition from discrete GPUs is wrong here.
Hard-killing the server leaks ~95 GB per node that no process owns.
Memory held below the process level — invisible to the usual tools. Recoverable without a reboot, if you know the sequence.
A 112 GB leak on the master, and a reboot would drop the whole cluster.
There is a no-reboot reclaim path. Knowing it exists is most of the fix.
Loaded a second model because MemAvailable said it would fit. Node OOM-rebooted.
The standard free-memory number is wrong on unified memory. Using it as a go/no-go gate costs you a node.
Raised the GPU memory fraction for more KV cache; a downstream component wedged silently.
On unified memory that fraction isn't taken from where you think — and the failure is silent.
What the silicon actually does
Spec sheets and B200 blog posts do not describe this chip.
FP4 kernels that scream on B200 produce fluent-looking garbage here.
A kernel-dispatch problem on this chip, not your model or your prompt — every stage reports success.
Full note →You're getting ~10% of the marketing TFLOP number and assume misconfiguration.
The marketing figure isn't a target for your workload. Tuning toward an impossible number wastes weeks — know the real ceiling first.
Decode throughput is capped no matter what you tune.
There's a hard physical floor. Near it, no setting helps; far from it, something specific is in the way. We tell you which in a day.
One node halves the whole cluster's throughput; benchmarks won't reproduce.
One misbehaving node poisons every collective — and it can survive reboots. Standard benchmarks won't catch it.
Speculative decoding & CUDA graphs
The biggest wins on this platform, and the easiest to measure wrong.
Turned on speculative decoding; nothing got faster.
Two good features silently cancelling each other. The win is recoverable — we measured
+58%.
Full note →More draft tokens should mean more speedup. It got slower.
Draft depth is not the lever most people think it is. Past a point, more drafting is pure cost.
Spec-decode benchmarks great, but users say it feels the same.
Your benchmark may be counting the wrong thing.
Full note →Config A beat config B on Monday; B beats A on Wednesday.
Restart-to-restart variance is bigger than most wins. The restart is the unit of replication — same-instance A/Bs are confounded.
Great single-stream speed, terrible under concurrent load (or vice versa).
These pull in opposite directions. One configuration can't be optimal for both.
The draft model won't load next to a quantized target.
A loader limitation with mixed-precision pairs — fixable, but not with a setting.
Silent wrong-output traps
The worst class: everything green, output quietly wrong.
A multimodal model describes images as word salad after sharding.
Not every part of a model tolerates being split — and the tooling won't warn you.
A coding agent starts strong, then rambles and gives up mid-task.
Often not the model's fault — a default nobody checks is quietly in effect.
Vision requests fail on a vision model.
A silent default, not a broken model — and no error tells you.
Temperature 0 twice, two different answers — is it broken?
Not necessarily broken — but telling harmless variation from real corruption takes a method.
A fix that worked on model A corrupts model B with the "same" architecture.
Recipes don't transfer between checkpoints, even within a family — each one has to be measured.
Ops & monitoring lies
On this stack, several standard observability tools return confident wrong answers.
pgrep says the server is gone. It's serving traffic.
Standard process tools give confident wrong answers on this stack — and automation that trusts them kills healthy servers.
The health endpoint returns dead while the server is mid-stream.
Under load, the usual liveness signal isn't reliable on this stack — supervisors that trust it restart working servers.
Your cleanup script kills itself; your grep finds itself.
A classic automation trap that takes down your own tooling. Easy once you've seen it; baffling until then.
Your "air-gapped" inference stack is phoning home benchmark fingerprints.
Blocking hostnames isn't isolation. A real air gap is proven
at the wire.
Full note →Env vars set on the master mysteriously don't apply on worker ranks.
Multi-node launchers don't behave the way their docs imply — settings silently go missing between nodes.
A tool's integrations vanish every time it starts.
A startup behaviour silently undoes your changes. You'll fix it again every day until you find it.
A node is down and it's headless in a rack.
Remote recovery is possible without a site visit — and the obvious "it's back up" signal is the wrong one to trust.
Free performance most setups leave on the table
Not bugs — levers. Each of these was worth more than a hardware upgrade.
Stuck at a 32k context window on a box that can hold half a million tokens.
The memory math allows far more — 512k contexts fit on this hardware, validated deep into the window.
Decode tuned to death, still slow.
The single biggest win we ever measured came from a limit almost nobody touches — not the knobs everyone argues about.
Agent workloads re-paying full prefill on every call.
There's a way to stop paying for the same prompt twice — up to 67× faster prefill on agent workloads.
Out-of-box config on a single node.
Baseline 25.5 tok/s → 45.2 tok/s on the same silicon, same model, config only (+77%). That gap is what a diagnostic finds.