Does Spark prefill really fall apart at long context? The measured answer
Field note · measured prefill across context lengths on GB10
Someone on Hacker News asks it every few weeks: "does inference screech to a halt at 100k context on a Spark?" The honest answer is that prefill — processing the prompt before the first token — degrades hard as context grows, and almost nobody has published the curve. Rough shape from real measurement: strong throughput at a few thousand tokens, falling steadily, and by the time you're deep past a few hundred thousand tokens of context, prompt ingestion is slow enough to feel broken.
Why it degrades — and which part is physics vs. fixable
Attention cost grows with context length, so some slowdown is real and unavoidable — that part is physics. But a large fraction of what people experience as "long context is unusable" on GB10 is not the attention math; it's the same fixable issues that bite at every context length, just amplified: the wrong collective protocol on a bandwidth-bound path, a KV cache in the wrong precision eating memory bandwidth, batching limits set for short prompts. The physics sets a floor; the configuration decides whether you're near the floor or far above it.
The distinction that matters: long-context prefill (prompt ingestion) and long-context decode (token generation) degrade for different reasons and have different fixes. Lumping them together as "long context is slow" guarantees you tune the wrong one. Measure them separately or you're guessing.
What keeps a long-context deployment usable
- KV-cache precision is a first-order lever at long context. The format that's fine at 8k can dominate your memory bandwidth at 256k — and the right choice frees enormous context headroom. We've validated large context windows deep into the range where naive configs fall over.
- Prefix caching changes the economics entirely for repeated-prefix workloads (agents, RAG with fixed system prompts) — we've measured order-of-magnitude prefill savings when the same prefix recurs.
- The batching knobs that matter for long prompts aren't the ones people tune. The default limits are set for short contexts.
Whether your long-context slowdown is the unavoidable attention floor or a fixable configuration problem is exactly the thing worth measuring before you conclude the hardware can't do it — because usually, a large chunk of it is fixable.
Long-context deployment feeling unusable?
We measure your prefill and decode curves separately on real GB10 hardware, separate the physics from the fixable, and hand you the config that keeps long context usable — with the numbers to prove it, or it's free.
Book a 1-day diagnostic →Free to read and share. Numbers are from hardware we've measured directly; your versions and topology will differ — measure your own.