Faster matrix math did not keep the matrix units busy. When Zyphra traced AMD’s AITER attention library on an Instinct MI355X, the FP8 path left the matrix pipe idle for 31.7% of cycles. The wait sat in the softmax, the step that turns scores into probabilities. It was not a shortage of peak matrix throughput.
That trace is why ZYFA exists. Zyphra FlashAttention on AMD is an open set of FP8 and BF16 kernels for causal prefill, the forward pass that scores a long prompt before a model emits a token. Zyphra presents it as a drop-in flash_attn_func and points to an Apache 2.0 release at https://github.com/Zyphra/zyfa. Every throughput, clock, and error figure below is Zyphra’s. None of it has been independently confirmed.
A narrow kernel, not a new serving stack
Read the support note before the speedups. Zyphra says ZYFA runs on the MI355X only, which it labels CDNA4 (gfx950), and names a stack of ROCm 7.2.3 and PyTorch 2.12. Pin a cluster to that pair only after checking the release you actually clone. The kernels cover causal forward attention in FP8 e4m3, an 8-bit float whose short exponent sends small values to subnormal or zero, and in BF16. Supported shapes are head dimension 128, and the MLA shape Zyphra labels 192/128: query/key head dimension 192, value head dimension 128. MLA, multi-head latent attention, is the compressed form DeepSeek-style models use. AMD has already wired an AITER MLA path into vLLM for those models, so the shape is a production inference target. ZYFA’s result on it is still a kernel result, not a vLLM benchmark.
Decode is out. Paged KV cache is out. Other GPUs are out. Zyphra says the kernels now sit under the attention path of its Optimization Engine, ZOE, which it uses to run Zyphra Cloud. That is a vendor production claim. They published no time to first token, tokens per second, or cost per request with it. A team already serving long prompts on MI355X iron has an experiment. Everyone else should not put this on a procurement slide.
The hardware fact that does matter for long prompts is memory. AMD lists the MI355X, an OAM accelerator in the MI350 series, at 288GB. In a May 2025 Performance Labs calculation (MI350-021), AMD compares an eight-GPU MI350X/MI355X platform with an eight-GPU NVIDIA B200 platform at 180GB and claims 1.6× the memory capacity. That is a capacity calculation, not an application benchmark. AMD’s marketed peak-performance multiples versus the B200 are a different comparison again, and Zyphra did not measure NVIDIA GPUs at all.
FP8 speeds the matrix products, not the softmax
FP8 is sold as a doubling of matrix throughput versus BF16. On attention, that doubling covers two products only: the scores, Q times K transposed, and the weighted values, P times V. Softmax still runs on the vector unit. FP8 does not give that unit a matching speedup.
Zyphra’s microbenchmark makes the mismatch concrete. One wave, 32 queries against 128 keys, head dimension 128. This is a cycle count, not the 32, 000-token throughput test. In BF16 they count 2, 048 matrix cycles against 768 vector cycles, so the matrix work is about 2.7× the softmax. Exact FP8 cuts the matrix side to 1, 024 cycles. The vector side does not fall with it. They count 896 vector cycles for the exact FP8 probability path, more than the 768 in BF16. “Same speed” here means the vector unit was not accelerated. The two precisions still do different amounts of vector work. Once the matrix side drops to 1, 024 cycles and the probability path still takes 896, the expensive units and the softmax are close enough that the matrix pipe waits.
Do not merge that picture with the 31.7% idle figure. The 31.7% comes from Zyphra’s thread traces of AITER. A separate count of Zyphra’s own exact-probability schedule puts the matrix pipe busy 60% of steady-state cycles, about 40% idle. Different code, different measurement. Averaging them would invent a third bottleneck that nobody reported.
AITER is still the baseline that matters. It is AMD’s AI Tensor Engine for ROCm, the high-performance operator library. An AMD writeup on vLLM 0.9.x says its kernels come from Triton, Composable Kernel, HIP, and assembly. Beating a toy kernel would not mean much. Beating the library AMD expects you to run might, if the comparison is the one your workload actually hits.
Three scheduling ideas, in plain terms
Zyphra’s account of the chip is that it does not track matrix-instruction completion in hardware, so overlap has to be created by instruction order. The FP8 kernels are hand-scheduled assembly emitted by a Python generator. The BF16 kernels are ordinary HIP. In Zyphra’s explainer, the HIP version needs 43-46% more cycles than the hand-ordered assembly. This is specialist code, not a portable patch you drop into every PyTorch box.
Lean P approximates the probability instead of computing a full exponential. Each FP8 probability is written with one multiply-add and one integer conversion: 6 vector cycles per score instead of 14. On the same microbenchmark, that cuts softmax vector cycles from 896 to 384, so matrix work is the long pole again. Zyphra puts the approximated value within 7.5% of the true probability p, against 5.9% for exact rounding into FP8. Treat that as a local bound on the stored probability, not as the model-quality gap measured later.
A two-wave ping-pong keeps two groups of threads busy, because the chip does not signal when the matrix unit finishes. Two waves share each SIMD. One issues matrix instructions while the other builds probabilities, then they swap. It is a software relay race.
Per-block scaling stops tiny scores from rounding to zero. In Mistral-7B layer 20, Zyphra found that 91% of row-and-64-key-block pairs sat more than eight binades, meaning factor-of-two steps, below the row maximum, where e4m3 goes subnormal or collapses to zero. Each 64-key block of probabilities is scaled by a power of two before rounding. Zyphra says a CDNA4 scaled matrix instruction removes that factor at no extra cost. One extra matrix instruction then sums the rounded probabilities, about 6% more matrix work, and the kernel divides by that sum. The row is normalized to what was actually stored, not to an ideal sum that never survived the format.
With Lean P, Zyphra reports the matrix pipe busy 86% of steady-state cycles, against 60% with an exact probability. The exact-probability phase is 1, 810 cycles. The Lean P phase is 1, 261. Those busy rates are steady-state claims. They are not 1, 024 divided by the full phase length, and they are not the AITER trace.
Same vocabulary, different operation: in AMD’s vLLM notes, block-scaled FP8 matrix multiplies for DeepSeek models raised request throughput by up to 19.4% and output-token throughput by up to 18.6%. Those scale blocks inside linear layers. ZYFA scales 64-key chunks of attention probabilities.
Synthetic gaps, then the real-activation shrink
On Gaussian inputs at 32, 000 tokens, kernel-only FP8 throughput is where the large ratios live. Grouped-query attention, GQA, means several query heads share one key/value head. The MLA rows use the 192/128 shape above.
- Head dimension 128: ZYFA 2, 672 TFLOP/s, AITER 1, 944, about 1.37×.
- GQA, head dimension 128: 2, 563 versus 2, 027, about 1.26×.
- MLA 192/128: 2, 464 versus 1, 237, about 1.99×.
- GQA, MLA 192/128: 2, 374 versus 1, 262, about 1.88×.
On the MLA shape, AITER’s default kernel is Composable Kernel. The near-2× gap is partly a weak baseline, not proof that ZYFA doubled attention in general.
Captured model activations, which Zyphra prefers, are quieter. These are 16, 000-token runs at the clocks the kernels actually sustained. Sustained does not mean matched. Clock rate moves with how many operand bits flip, so two kernels on the same layer need not land on the same megahertz.
- Mistral-7B layer 20: 2, 537 TFLOP/s versus AITER’s 2, 191, a 1.16× gap.
- Qwen3-8B layer 18: 2, 495 versus 2, 130, a 1.17× gap.
- DeepSeek-V2-Lite layer 13: 2, 337 versus 1, 326, a 1.76× gap.
Absolute rate is highest on Mistral, not on the MLA case with the flashy ratio. DeepSeek looks like a blowout because AITER is slow there, on the shape where its default is Composable Kernel, while ZYFA itself is slower in raw TFLOP/s than on Mistral.
Clocks explain part of the shrink on the ordinary models. On the Mistral-7B layer, Zyphra measured ZYFA at 1, 628 MHz and AITER at 2, 108 MHz. The 1.16× throughput edge is not a same-clock comparison, and it understates work per clock. Divide that throughput edge by the clock ratio, 1, 628 over 2, 108, and ZYFA did about 1.5× the work per clock. That 1.5× is arithmetic from Zyphra’s two figures, not a metric they published on its own. Their account of the clock drop is that AITER’s treatment of tiny probabilities flips fewer operand bits and lets the chip clock higher, while ZYFA keeps more numeric work alive and runs slower in megahertz.
BF16 only ties AITER, at 1.01-1.02×. The win is specific to the FP8 scheduling problem. If you are already matrix-bound, a cleverer softmax does not buy you much. On the captured Mistral-7B and Qwen3-8B layers, Zyphra also reports 1.94 TFLOP/J versus 1.60, and 1.91 versus 1.56. No power readings were published, so those efficiency ratios cannot be checked independently.
Kernel error falls more than full-model error
The error story depends on the yardstick, and the yardsticks do not describe the same gap. Relative RMS error of a single FP8 attention call against fp64, mean over layers on WikiText-2 at 4, 000 tokens:
- Mistral-7B: 0.0120 for ZYFA versus 0.0641 for AITER. Several-fold lower on this kernel metric.
- Qwen3-8B: 0.00911 versus 0.0317. Several-fold again.
- DeepSeek-V2-Lite: 0.00836 versus 0.0102. A modest gap. ZYFA’s error is 0.82× AITER’s.
Full-model KL divergence from fp64 on Mistral-7B is a third, narrower comparison: 3.4×10⁻³ with ZYFA and 5.9×10⁻³ with AITER. Zyphra says most full-model FP8 error comes from quantizing Q, K, and V, not from Lean P. That is why a large kernel-error gap can shrink once the whole model is scored. Task accuracy and perplexity were not measured. A Hadamard rotation of Q and K roughly halved the input-quantization error and added about 0.1% to attention time on Mistral-7B. Useful, cheap, and easy to forget when a slide only shows kernel error.
Zyphra also places its fp64 error within a few percent of Composable Kernel and Triton, so the large gaps above are against AITER on two of the three models, not against every careful kernel. Outputs were bitwise identical across 7, 200 launches. That is a strong check on the cases they ran. It is not a certificate for every supported shape and input.
What this does not yet change on a serving bill
Long-context prefill is a real line item. An automation job that ingests a contract before it answers, or an agent that cold-starts on a long ticket and policy packet, pays for that pass every time the cache is empty. The MI355X’s 288GB is why that workload is even plausible on one GPU. Memory capacity is the relevant AMD fact here, not a theoretical peak versus another vendor.
Kernel TFLOP/s at 32, 000 tokens still does not establish tokens per second, time to first token, or cost per request. A faster prefill kernel that cannot decode, and cannot use a paged KV cache, does not slot into a multi-tenant agent stack by itself. Once the cache is warm, many conversational bills are decode bills.
If AITER later learns the same overlap, a hand-scheduled assembly kernel becomes a temporary edge. Until then, a team already on this chip can swap the prefill attention op, time it on captured activations, and print the clock next to the FLOPs. The useful residue is not a 1.76× sticker. On this FP8 path, attention’s bottleneck moved from the matrix units to the softmax, and a serving stack still has to pay for decode, cache layout, and the prompts users actually send.
Questions worth asking before this hits a roadmap
-
Is FP8 attention on the MI355X limited by matrix throughput?
Not in Zyphra’s accounting. FP8 cuts their counted matrix cycles from 2, 048 to 1, 024, while the exact FP8 probability path still takes 896 vector cycles. A separate trace of AITER shows the matrix pipe idle 31.7% of cycles. That is not the same measurement as ZYFA’s own 60% steady-state busy rate.
-
How much faster is ZYFA on real models, not synthetic noise?
Zyphra reports 1.16× on a Mistral-7B layer, 1.17× on Qwen3-8B, and 1.76× on DeepSeek-V2-Lite. The roughly 2× Gaussian gap is an MLA shape where AITER defaults to Composable Kernel, and ZYFA’s absolute rate there is lower than on Mistral. On Mistral, ZYFA ran at 1, 628 MHz against AITER’s 2, 108 MHz, so the 1.16× understates work per clock.
-
Is the numerical error actually lower?
On the kernel RMS metric against fp64, yes: several-fold on Mistral-7B (0.0120 versus 0.0641) and Qwen3-8B (0.00911 versus 0.0317), and only modestly on DeepSeek-V2-Lite (0.00836 versus 0.0102). Full-model KL on Mistral-7B is a smaller gap, 3.4×10⁻³ versus 5.9×10⁻³, and task accuracy was not measured.
-
Does this lower the cost of AI agents or long-document automation?
Not on the evidence published. There is no end-to-end latency, throughput, or cost result, and the kernel is prefill only: no decode, no paged KV cache, MI355X only.
-
Can you treat a percent-of-peak figure as an AMD spec?
No. Zyphra has framed throughput against a 5.0 PFLOP/s FP8 peak that is not an AMD datasheet figure established next to these results, and AMD’s marketed peaks include sparsity where it applies. Score dense attention on your own activations, with the clock written down.
The rotation behind the error cut
Kernel RMS and full-model KL are not the same gap. The citation Zyphra leans on for rotating Q and K is the one to read before treating a Hadamard as a free accuracy patch.