Your server says 40.7 tok/s. Your users get 27.8. Neither number is lying.

Field note · measured on an 8× GB10 cluster, ~1T-parameter MoE with speculative decoding

Here are three throughput numbers we pulled from the same server, in the same time window, running the same model:

• The server log's generation-throughput line peaked at 40.7 tok/s.
• The accepted-throughput counter in the same window read 29.4 tok/s.
• A strict client-side benchmark — 15 runs, 95% confidence interval, counting only tokens actually delivered — said 27.8 tok/s.

None of these numbers is fake. They measure three different things, and if you quote the wrong one you will make a wrong decision — we know, because one of these nearly made us ship the wrong configuration, and a cousin of it once made us condemn an optimization that was actually working.

Why the gap exists

With speculative decoding, a small draft model proposes several tokens and the big model verifies them in one pass. Some proposals get accepted; the rest are thrown away. The server's internal throughput counter is proud of everything it generated — including the tokens that went in the trash. The accepted counter is closer to reality, but it's still an instantaneous window, not what a full request experiences. Only the client-side number — real requests, real tokens received, enough repetitions to see the variance — is the number your users live in.

The same disease has more strains than most teams realize:

• Stream-chunk counting. If your benchmark counts streamed chunks, on a spec-decode server you're measuring steps per second, not tokens per second. This one made a working optimization look net-neutral for weeks — the fix was fine; the ruler was broken.
• Run-to-run variance. The same config on this hardware varies 15–19% between runs. A median-of-3 is not a measurement; it's a coin flip with extra steps. One config measured 30.2 once and 22.2 another time. Its true value was 25.4 ± 1.9.
• Restart-to-restart variance. Two server instances of the identical config produced non-overlapping confidence intervals. If your A/B ran each arm in a single instance, your A/B is confounded — full stop.
• Aggregate vs. single-stream. "1,400 tok/s" and "44 tok/s per user" can be the same server. Vendor slides love the first number. Your users only ever feel the second.
• Prefill vs. decode. Prompt ingestion runs an order of magnitude hotter than generation. A "139 tok/s" that was secretly a prefill measurement will make your decode look broken when it's fine.

Building a benchmark harness that gets this right — negative controls, restart-aware replication, per-phase separation, honest confidence intervals — is genuinely fiddly, and the failure mode is silent: a broken ruler still produces confident-looking numbers. Which numbers to trust in your specific stack's logs, and how to structure an A/B on hardware with this much variance, is exactly the kind of thing that takes a few painful months to learn and a day to be shown.

Free to read and share. Numbers are from hardware we've measured directly; your stack will differ — measure your own, and count only what was delivered.