A chip rated for a petaFLOP can still feel slow when it is answering one person. On a desk-side NVIDIA DGX Spark, that is not a paradox. NVIDIA’s product page rates the box’s FP4 Tensor Cores at up to 1 petaFLOP. At batch size 1, generating the next token is mostly a memory errand: the model spends its time reading weights, not burning through math. The peak barely gets a vote.
NVIDIA timed that constraint with Inclusion AI’s Ling-3.0-flash-fp4, a mixture-of-experts model with 124 billion parameters and 5.1 billion active on each token. The machine was the 128 GB DGX Spark, a desktop-size Grace Blackwell system. NVIDIA rates that configuration for inference up to 200 billion parameters. Fine-tuning on Spark is rated only to 70 billion, so this is a serving result, not proof you can train the model there.
For anyone buying local AI for business (a private assistant, an internal agent, a sales desk that should not ship transcripts to a cloud API), the useful question is how many bytes one reply forces across memory, and how many tokens come back per trip. The FLOP count on the box is the wrong scoreboard.
Why one user waits on weights
Mixture-of-experts is the architectural bargain. A router sends each token to a slice of specialist layers instead of waking the whole network. On Ling’s MoE layers, that is 8 of 512 routed experts, plus one shared expert that runs every time. About 4.1 percent of all parameters are active per token, which is simply 5.1 billion divided by 124 billion. That is not the same figure as the router’s 1-in-64 sparsity. The shared expert, the dense layers, and attention sit outside the 8-of-512 ratio.
Sparsity is why a 124 billion-parameter checkpoint fits in 128 GB. It does not make single-user decoding compute-bound. In NVIDIA’s technical write-up, profiling of this FP4 checkpoint found decode limited by weight reads. Two operations dominated: MoE weight reads, and vocabulary-projection reads (the language-model head that scores the next token).
That head is a matrix from 2, 560 hidden units out to a vocabulary of 157, 184. The checkpoint stores it in BF16 and lists it among layers ordinary loading will not quantize. Every new token reads it again. The official baseline was the W4A8 cookbook path: FlashInfer MXFP4 kernels for the experts, CUTLASS for dense matrix multiplies, and an FP32 head path. NVIDIA does not define W4A8. In common usage it means 4-bit weights and 8-bit activations.
On a synthetic run (32, 000 tokens in, 2, 000 out), NVIDIA reports 27.71 milliseconds per output token, or 36.09 tokens per second. That is decode time, prefill excluded. A 32, 000-token prompt is not coming back in tens of milliseconds. The figures are point measurements. The write-up does not publish a run count or an interval.
Three fixes, two different jobs
NVIDIA stacked three changes. The first two make each decode step cheaper. The third tries to take fewer steps. They are not three flavors of the same trick, which is why a flattering synthetic number and a short-answer number can share a lab notebook.
Kernels built for the shape you actually have
The first swap replaces FlashInfer’s MXFP4 MoE backend with Humming, an independent kernel library. Humming uses NVRTC to compile at runtime for the real matrix shapes, data types, and GPU, including Ling’s narrow expert layer, an intermediate size of 768. A kernel compiles in seconds. The FlashInfer CUTLASS path compiles 97 targets and takes on the order of tens of minutes. That gap is a first-start cost, not a steady-state speedup (the difference between a coffee and a meeting before the server is even answering).
On the same long synthetic decode, NVIDIA’s table shows 25.35 milliseconds and 39.45 tokens per second, a 9.3 percent gain. Careful engineering. Not a new product category.
Stop hauling the vocabulary in full precision
The second change quantizes only the language-model head to FP8 when that layer is built, leaving the rest of the checkpoint’s FP8 and MXFP4 setup alone. BF16 to FP8 halves the bytes per value, so that layer’s reads drop by half. The switch is the environment flag SGLANG_ENABLE_FP8_LM_HEAD=1. NVIDIA says it cannot be combined with the baseline –enable-fp32-lm-head option.
Stacked on Humming, the synthetic decode reached 22.76 milliseconds and 43.94 tokens per second: 21.7 percent over the original baseline, and 11.4 percent over Humming alone. The budget-meeting version is simple. They did not buy a faster chip. They stopped rereading one fat matrix at twice the bytes per value.
Draft three tokens, verify four
Ling was trained with multi-token prediction. NEXTN, a built-in draft layer, proposes three tokens. The target model then scores four positions in one pass: those three drafts, plus a bonus token (the token it would emit next if the drafts hold). Accept length is how many of those tokens you keep per round, bonus included. There is no separate draft model to host, which matters on a box already holding a 124 billion-parameter checkpoint.
Verification is not free. NVIDIA measured it at about 2.26 times a normal decode step. At batch size 1 the extra cost was expert activation, not attention. One token wakes eight routed experts. Four positions can wake up to 32, and that “up to” is a worst case. It assumes the four positions pick disjoint experts. Overlap costs less. The shared expert still runs on top of that count.
Think of a junior associate drafting three sentences so a partner can sign them in one sitting. If the partner keeps the draft, you saved rounds. If the partner rejects it, you added a meeting.
The number that should not set your SLA
NVIDIA’s summary of the synthetic workload says that once NEXTN joined the other two changes, accept length hit 3.90 tokens per round and throughput rose 185.1 percent. It does not print a tokens-per-second cell next to that percentage, and it does not spell out the denominator. The natural reading, matching how the earlier gains are stated, is versus the original 36.09 tokens-per-second baseline. On that reading, 185.1 percent is a near-tripling, about 103 tokens per second. If the 185.1 percent were instead versus the Humming-plus-FP8 step, the implied rate would be higher still, about 125 tokens per second. Either way, 103 is arithmetic on a headline, not a second published measurement.
The inputs do not reconstruct either reading. Keeping 3.90 tokens while paying 2.26 times a normal step is a speedup of about 1.7 times on the step you are comparing against. Start from the 43.94 tokens per second already measured with Humming and the FP8 head, and you land near 76 tokens per second, about 110 percent over the original baseline, not 185. A misread of a 1.85× ratio as “+185.1 percent” does not close the gap either. The published percentage and the accept-length math are telling different stories. Do not let 185 percent into a service-level agreement until a stacked table reconciles them.
The real-workload headline has a similar split, and this one does have a clean explanation. On the full GSM8K set, NVIDIA’s summary says the three changes together raised decode throughput 42.5 percent and cut mean latency 24.5 percent. A 42.5 percent throughput gain would cut time per generated token by about 30 percent, because 1 divided by 1.425 is about 0.70. Mean latency is the whole reply, prompt included. Reading the prompt does not get the same discount as generating the answer, and grade-school word problems are short, so prefill is a larger share of the clock. The 24.5 percent is end to end. The 42.5 percent is decode only. Quote the one that matches the wait your users actually feel.
That distinction belongs in the runbook. AI for sales, support macros, and a lot of AI automation are short turns: a qualification note, a CRM update, a tool call, a paragraph. AI agents that hop between tools live on brief, high-stakes snippets, not on 2, 000 tokens of agreeable prose. Size a desk-side deployment off the GSM8K-class gain, and only after you have decided what the 185 percent figure even is. Size it off the synthetic accept length and you will overbuy the story and underbuy the latency.
A faster step can still be a slower reply
There is a quieter failure mode, and the arithmetic is the same. NVIDIA says the first two changes cut the cost of each step, while speculative decoding cuts the number of steps, and that an integration mistake can make the combined setup slower than the unoptimized baseline. You still pay about 2.26 times a normal decode and keep one token. That is just a more expensive way to generate. Confirm at startup that the FP8 head is actually on, that it is not fighting the FP32 head flag, and that speculative decoding is keeping more than one token per round.
Do not copy the multi-GPU recipe onto this result. Inclusion AI’s model card points at a fork of SGLang, branch ling_v3_support_mxfp4, with FlashInfer pinned at 0.6.16.post1, and –speculative-algorithm NEXTN because the model was trained that way. The published example is two Blackwell GPUs, tensor parallel 2, a 262, 144-token context, and up to 64 running requests. That is a different regime from one Spark and one user. Paste one config into the other and you are no longer running the benchmark you think you are.
Why 124 billion parameters fit on a desk
The speed test is easier to trust once you see why the model fits the box at all. Ling-3.0-flash keeps most of its attention in a fixed-size state instead of a key-value cache that grows with every token. Of 42 layers, 35 are Kimi Delta Attention, from Moonshot AI’s Kimi Linear work. Only 7 do full attention, and they use a compressed cache. Sparse routing plus that mostly fixed-size cache is why 124 billion parameters fit, and why a long context does not explode memory the way a conventional cache would.
Native context is 256, 000 tokens. NVIDIA says the window can extend to 1 million, and does not say how. One wording caveat, from the model card versus NVIDIA’s write-up: NVIDIA calls those 7 layers multi-head latent attention. Inclusion AI calls them Gated MLA. The layer count matches. The name does not.
The checkpoint NVIDIA ran is FP4. On Inclusion AI’s own quantization table, that compression is mostly polite and occasionally not:
- GPQA-diamond: 84.97 in BF16, 83.68 in FP4.
- IFBench: 74.49 to 73.87.
- ArcPrize: 68.75 to 68.06.
- SciCode: 41.24 to 38.43, a drop of 2.81 points, the clear weak spot.
FP4 stays within 1.3 points of BF16 on three of those four tests. The card describes FP8 as blockwise, and INT4 and FP4 as groupwise on routed-expert weights. None of that table was remeasured in the Spark speed run. A math quiz that barely moves does not tell you whether a coding agent will feel the SciCode gap. If the AI-for-business case is math-ish question-and-answer, the quantization story looks calm. If it is software agents, look harder before you standardize on the FP4 file that happens to fit one Spark.
Spark itself has a ceiling this test does not touch. NVIDIA’s product page says ConnectX can link up to four machines, rated at 512 GB and up to 700 billion parameters. Nothing here measures that cluster, a higher batch, or concurrent users. One fast private session is not a contact center.
Questions worth asking before you quote the speedup
-
Why doesn’t a petaFLOP desktop answer faster out of the box?
At batch size 1, NVIDIA found this decode memory-bound. MoE weight reads and vocabulary-projection reads set the pace, not the FP4 Tensor Core peak, which NVIDIA rates at up to 1 petaFLOP.
-
Which change actually mattered?
Humming alone added 9.3 percent, to 39.45 tokens per second. The FP8 head took the synthetic decode to 43.94, 21.7 percent over baseline. NEXTN is what NVIDIA’s summary credits for a 185.1 percent jump (about 103 tokens per second if that is versus the 36.09 baseline), but 3.90 accepted tokens at 2.26 times the step cost implies closer to 76, so the headline and the inputs still disagree.
-
What should a short business turn be sized against?
The GSM8K-class result: 42.5 percent more decode throughput, and 24.5 percent less mean latency once the prompt is in the clock. Not 185 percent, and not until that figure is reconciled with the accept length. Sales notes, tool calls, and most agent steps look like short answers, not a 2, 000-token synthetic monologue.
-
Did quality collapse to get that speed?
On the GSM8K comparison, NVIDIA says the accuracy change was not statistically detectable, and does not show the test, interval, or run count behind that phrase. On Inclusion AI’s quantization table, FP4 stays within 1.3 points of BF16 on three of four scores and is 2.81 points lower on SciCode.
-
Can the three optimizations be switched on and forgotten?
No. The FP8 head flag cannot be combined with the baseline FP32 head option, and a bad pairing with speculative decoding can run slower than doing nothing, because you still pay about 2.26 times a normal step. Check that each optimization is live, and that tokens kept per round are well above one.
-
Is a 124 billion-parameter model now a data-center replacement?
Not on this evidence. The result is one user, batch size 1, inside Spark’s 200 billion inference rating and outside its 70 billion fine-tune rating. No concurrency, power, or cost-per-token figure comes with it, and a cloud API’s tokens per second are a different serving setup entirely.
The durable lesson is almost dull, which is how you know it will survive the next kernel release. For single-user AI on a desk, bandwidth and accept length beat the number on the FLOPs slide. Fewer bytes per step, more tokens kept per verification round, and a workload that resembles the replies you actually send. Everything else is a demo with good lighting.