Sparse attention has a tell. The last multiply looks cheap, and then the piece that picks which tokens to read quietly scores every past token you hoped to skip.
Zyphra Hierarchical Attention, ZHA, tries to make that search sparse. The full key-value cache stays in memory. A tree of chunk summaries only decides which cached keys a new query is allowed to read. Optimized kernels have not been released. The large speed numbers are a roofline. They model what a chip might do, not what shipping code has measured.
That matters if an agent rereads a contract, a codebase, or a year of customer mail on every turn. A system that only sparsifies the final step still charges you for looking at everything. Zyphra points to work where, near a million tokens, indexing consumes 60 to 90 percent of layer time. Fixed-state shortcuts such as state-space models and sliding windows, in Zyphra’s account, saturate as context grows. Fine-grained designs such as DeepSeek Sparse Attention cut the attention work and still leave an indexer that scores the past token by token.
The paper, Subquadratic Attention with a Hierarchical Tree-Structured Indexer (arXiv:2610.42112), aims at subquadratic latency. Zyphra sets that against chunk-sparse methods that, it says, only buy a constant-factor trim on a scan that still grows with the whole context. That is an aim, not a result anyone has timed.
How the tree decides what to read
Think of the cache as a filing cabinet you do not reshuffle on every question. A small set of learned reference queries compresses each chunk of keys into a summary. The summary is a guess of how much attention a future query would put on that chunk. The query does not score the cabinet. It starts near the top and walks down only the branches that look promising.
Top-p routing keeps the budget adaptive. At each level the router keeps the smallest set of nodes that together hold a fraction p of the estimated attention, then descends until it reaches raw keys. Too low, and it drops branches the model needed. Too high, and it expands enough of the tree to buy the quadratic cost back.
Zyphra’s write-up says the first token is always kept as an attention sink, and the explainer also keeps exact attention on the current chunk. MiniMax Sparse Attention always keeps the local block that contains the query. Related habit, different switch.
On Qwen3.6-35B-A3B, Zyphra showcased two trees, both with 16 reference queries and p = 0.95. The deep tree uses chunk size 4 and six levels. The wide tree uses chunk size 256 and a single level, so its router scores chunk summaries across the context rather than walking a deep hierarchy. Six levels of four cover 4, 096 tokens under one top node. A 262, 144-token context then holds 64 such nodes. If the query still scores every top node, the sparse walk starts only after that scan. “The search itself is sparse” is not a claim about a single tree over the whole context.
The paper’s speedup case study uses the deep geometry, with depths from 2 to 6, chunk size 4, and 16 reference queries. The wide tree belongs to Zyphra’s reported comparison. That case study does not restate it.
The conversion, as Zyphra describes it, leaves the pretrained model frozen and trains only the reference queries. The objective is a KL loss that matches dense attention mass per chunk at every tree level. They report a warmup of about 4 billion tokens at 32K context, 3.98 billion tokens over 1, 900 steps on the Qwen and Nemotron runs, and 81, 920 new parameters on Qwen3.6-35B-A3B, about 0.00024 percent of that model. A small knob, if the recipe holds up outside their lab.
The numbers, with the labels still on
Every head-to-head score below is self-reported by Zyphra. No independent replication sits next to the table. Read them as one lab’s chosen settings, not as a law of sparse attention.
Accuracy, in percent. Dense / ZHA deep / ZHA wide / MiniMax Sparse Attention (MSA):
- AIME 2026: 93.1 / 91.2 / 90.8 / 91.7
- SWE-bench Verified: 67.4 / 67.4 / 68.4 / 63.6
- Terminal-Bench 2.1: 49.9 / 46.7 / 53.3 / 46.1
- MMLU-Pro: 84.9 / 85.3 / 85.2 / 85.3
- GPQA Diamond: 84.8 / 85.0 / 85.0 / 83.1
- RULER-13 at 256K: 96.0 / 93.3 / 96.2 / 77.1
- LongBench v2: 58.0 / 63.4 / 62.4 / 49.3
The paper itself records the dense AIME score of 93.1 percent. The rest of the split is Zyphra’s. On that split, short knowledge tasks stay close to dense. Coding does not tell one story. The wide tree is a point ahead on SWE-bench Verified, 68.4 versus 67.4, and clearly ahead on Terminal-Bench 2.1, 53.3 versus 49.9. The deep tree, the sparsity story, is flat on SWE-bench, down 3.2 points on Terminal-Bench, and down 1.9 on AIME 2026. It is not free. LongBench v2 is the gain in Zyphra’s table: 63.4 percent for the deep tree against 58.0 percent dense.
Sparsity is a different table, and it is decode-only. Share of KV keys read during decoding, in percent. ZHA deep / ZHA wide / MSA:
- AIME 2026: 5.7 / 47 / 8.7
- MMLU-Pro: 16 / 81 / 85
- GPQA Diamond: 9 / 60 / 21
- RULER-13 (256K): 2.8 / 42 / 0.76
- LongBench v2: 1.1 / 21 / 1
Zyphra did not publish those shares for SWE-bench Verified or Terminal-Bench 2.1, so the coding results have no key-share beside them. Decode-only sparsity also does not answer prefill. If an agent restuffs a long history at the start of every turn, prefill is the bill.
Where shares were published, the deep tree is the one that actually skips. On LongBench v2 it reads about 1.1 percent of keys and, in Zyphra’s numbers, beats dense by 5.4 points. The wide tree often reads a large minority of the cache and sometimes scores higher. MSA can be stingier, 0.76 percent of keys on RULER-13 at 256K, and still land at 77.1 percent, while both ZHA trees stay above 93. You can read fewer keys and still miss the ones that matter.
Zyphra also says the MSA indexer in this comparison was trained on 40 billion tokens, ten times ZHA’s budget, and that the deep tree beats MSA by 14.1 points on LongBench v2 at a similar key share, 1.1 percent versus 1 percent. MSA’s own paper does not back up that table. Treat the gap as a claim from one side of the match.
Across 28 tree geometries, Zyphra fit a small model of the relative RULER penalty from geometry and sparsity. The check holds out sequence length, not the model and not the task, and it still needs the dense score at the held-out length. For the 16-reference setting, the paper’s pooled leave-one-length-out error is 1.44 points across 64K, 128K, and 256K. Smooth enough to predict is not flat, and the fit does not say every tree works.
Two ways it falls over
“Close to dense” is a setting, not a property of the method. The paper reports that the deep geometry, chunk size 4, six levels, 16 reference queries, at p = 0.7 scores 44.0 percent on AIME, against 93.1 percent for dense. That is a 49.1-point collapse. Drop the threshold and the tree can throw away branches the model actually needed. The showcased runs use p = 0.95. Treat that as a chosen operating point, not a default that travels.
Architecture fit is the other miss. Zyphra reports that on Nemotron-3.5-Lightning-30B-A3B, which it describes as a model without positional embeddings, deep-tree RULER-13 falls 25.3 points below that model’s own dense score, and the wide tree falls 10.2. Public listings give both Qwen3.6-35B-A3B and this Nemotron model a 262, 144-token window, so a retrieval miss at 256K is not explained by the context failing to fit. If the drops hold up outside Zyphra’s lab, they are a compatibility result, not “Nemotron is simply the weaker model.” The cause, including the no-positional-embedding explanation, is not established.
Speed you cannot buy yet
No optimized ZHA kernels were released. The paper’s roofline is more careful than a raw multiply count. It adds serial passes for reference scoring, node selection, and exact attention on the tokens that survive, plus launch overhead and a term for limited GPU occupancy. Pruning can leave higher tree levels with too little work to fill the chip. Streaming multiprocessors then sit idle. Subquadratic arithmetic does not automatically become subquadratic latency. The paper is explicit that practical speed still depends on efficient GPU execution.
Zyphra’s write-up models an H100 in BF16. The selection counts behind that speed model were fit on ZHA generations for AIME 2026, with response lengths of 16K to 64K. A lighter projection uses one reference query, chunk size 8, p = 0.75, a single transformer layer, and batch size 1. That is not the quality tree, and it is not a production batch.
Inside Zyphra’s model, the crossover against dense attention sits near 48K tokens, and against MSA near 0.48 million. At 256K, Pareto-frontier settings are modeled at 1.93x to 2.28x over dense, with RULER-13 still at 92.7 to 96.4 percent. At 1 billion tokens, the same style of projection quotes 373x to 417x over dense and 52x to 58x over MSA. Those billion-token figures step past both the 16K to 64K fit and the 262, 144-token windows these models are published with.
The deepest geometry at p = 0.7 also breaks the usual pattern in the paper’s speed fit: scaling exponents rise again toward the leaves. The paper says to read that next to the accuracy collapse, not as healthy scaling.
MSA is a real comparator, and the evidence is a different species. Its paper, arXiv:2606.13392, describes an indexer on grouped-query attention, trained with a KL loss against a frozen main branch, plus a warmup that starts in full attention and later scores only the selected positions. Deployed selection uses a top-k of 16 blocks of 128. It also describes actual kernels, with a top-k benchmark on an H800. A cheap flat index plus a fixed block budget can already open a growing compute gap when the kept budget is much smaller than the context. That is not the same claim as “the search itself is sparse.” Neither side has published an independent head-to-head at million-token scale. Comparing a roofline to someone else’s kernels is how slide decks get ahead of hardware.
If you pay for long-context agents
Sparse attention already exists. The useful question is whether the index stops dominating the layer. Anyone whose agent rereads a whole history on every turn is buying indexer time. Contracts, code, and a year of customer mail have the same shape. Do not reprice a product around a 400x slide.
A pilot worth running has to produce three things the write-up does not: kernels that survive launch overhead and half-empty GPUs, a threshold sweep that does not fall off the AIME cliff, and a second architecture that is not Qwen. Until then, ZHA is a serious research bet on making the search sparse, reported by the people who built it, with one documented way to make quality fall over.
Questions worth asking before you reprice a product
- Does ZHA use less memory?
No. It sparsifies reads, not storage. The full key-value cache remains, so you still pay to keep it and only hope to pay less to search it.
- Do the sparsity numbers cover prefill, or only decode?
The published key shares are the fraction of KV keys read during decoding. They do not answer prefill, which is the bill when an agent restuffs a long history at the start of a turn.
- How close does quality stay to dense attention?
Only at the settings Zyphra showcased, and only on their Qwen numbers: often near dense, ahead on LongBench v2, and not free for the deep tree on AIME or Terminal-Bench. The paper’s own deep tree at p = 0.7 scores 44.0 percent on AIME against 93.1 percent for dense.
- Are the hundred-fold speedups measured?
No. They are roofline projections, optimized kernels were not released, and the selection patterns were fit on AIME responses of 16K to 64K tokens. Figures out at a billion tokens leave that fit and the models’ published 256K-class windows.
- Will this drop into any model we already serve?
Not on the evidence in hand. Zyphra reports a RULER miss on Nemotron-3.5-Lightning-30B-A3B of 25.3 points for the deep tree and 10.2 for the wide tree versus that model’s own dense score, and the cause is not established.
- What should a pilot measure before anyone changes a price?
Kernel occupancy on real hardware, a sweep of p, prefill and decode separately, and at least one architecture that is not Qwen. A projection is not a quote.