Kimi K2-class (~1 trillion parameters) on 8× DGX Spark
8× GB10 · 100 Gbps switch · 4-bit · 256k context · per the methodology
A frontier-scale mixture-of-experts model at roughly a trillion parameters is widely assumed to be impractical on desktop-class hardware. Measured, it isn't: a ~1-trillion-parameter Kimi K2-class MoE serves at 26–28 tok/s delivered on code workloads (22–25 on prose), single stream, on eight GB10 nodes connected by an ordinary 100 Gbps switch — not exotic fabric. Native 256k context, and a fast cold start.
| metric | result | notes |
|---|---|---|
| decode · code | 26–28 tok/s | delivered tokens, n=15, 95% CI |
| decode · prose | 22–25 tok/s | same protocol |
| context | 256k native | tuned KV cache |
| cold start → serving | ~2.5 min | was 7.5+ min before load-path fixes |
For scale
The best-known public 8-Spark result is a 397B unquantized model at 24 tok/s on 200G-class networking. This entry is a model roughly 2.5× that size, running faster, on half the network bandwidth — quantized to 4-bit — on this silicon, precisely the part that goes wrong by default.
What stood between out-of-box and this number
- It would not load usably at first — minutes-long stalls with three different root causes that look identical (the loading triage).
- FP4 on sm_121 produces garbage with the default kernel dispatch — fluent-looking nonsense, no errors. Getting correct FP4 output on this chip is its own discipline.
- Speculative decoding had to be tuned for the workload — the "obvious" configuration was measurably slower.
- The server log claimed 40.7 tok/s while clients received 27.8 — which number to believe is half the work.
Running a frontier-class MoE on Spark hardware?
This entry is the end state of extensive measured tuning. The diagnostic gets your cluster the same treatment: we reproduce your config on our 8-node lab, find what stands between you and numbers like these, and put it in writing — or it's free.
Book a 1-day diagnostic →Free to read and share. Numbers are from hardware we've measured directly; your engine versions and configs will differ — measure your own, and count only delivered tokens.