Zyphra Converts ZAYA1-8B MoE to Block Diffusion, Delivering 4.6×–7.7× Inference Speedups

ZAYA1-8B Block Diffusion: Converting MoE LLMs for Major Inference Speedups

TL;DR

Zyphra converted an autoregressive Mixture‑of‑Experts (MoE) LLM, ZAYA1‑8B, into a discrete diffusion‑style model using a TiDAR mid‑training recipe and reports large inference speedups (≈4.6× lossless, ≈7.7× aggressive) on AMD hardware.
The converted model generates blocks of 16 tokens at once and uses two sampler strategies—one that preserves evaluation metrics and one that trades some quality for throughput.
Practical upside: lower cost and latency for serving and especially for expensive on‑policy RL rollouts. Caveat: diffusion tooling and real‑world integration remain immature and results are reported on a pre‑RL checkpoint using pass@ metrics.

What Zyphra did (plain summary)

Zyphra took an existing autoregressive MoE model (ZAYA1‑8B) and converted it mid‑training to a discrete diffusion language model using a TiDAR conversion recipe. TiDAR here refers to a process that blends autoregressive pretraining with diffusion‑style objectives during additional training. The conversion added about 1.1 trillion extra training tokens total—≈600B for diffusion mid‑training and ≈500B for context extension plus diffusion supervised fine‑tuning. The converted model decodes blocks of 16 tokens at a time via a single mask→token step, then uses samplers inspired by speculative decoding to accept a subset of proposed tokens.

What this means: rather than predicting one token per pass, the model proposes many tokens in parallel and accepts some of them, drastically reducing per‑token overhead and improving GPU utilization for many workloads.

Why block diffusion helps: the KV‑cache bottleneck and compute vs memory

Autoregressive decoding stores key/value (KV) caches per request so the model can attend to prior context. Loading each request’s KV cache every decode step eats memory bandwidth and leaves compute units idle when bandwidth is the limiter. Block diffusion generates multiple tokens per forward pass, allowing the same KV cache to be reused across those tokens and shifting the workload from memory‑bandwidth bound toward compute‑bound. On hardware where compute is plentiful relative to memory bandwidth, this increases throughput.

“Autoregressive generation can become memory‑bandwidth bound because each user’s KV‑cache must be loaded separately, leading to idle compute units.”

What this means: for deployments and RL rollouts where KV cache loads dominate, block decoding can give outsized savings in latency and cost because fewer heavy cache loads are needed per token produced.

How the samplers work (lossless vs logit‑mixing)

Zyphra uses two samplers to convert block proposals into final tokens.

Lossless sampler: A conservative acceptance rule based on speculative decoding—accept a proposed token when its probability under the target model is close to the proposal distribution; otherwise resample from the residual probability mass. This preserves evaluation metrics and yields about a 4.6× speedup in Zyphra’s reported tests.
Logit‑mixing sampler: An aggressive approach that mixes diffusion logits with autoregressive logits to increase acceptance rates and throughput, trading some sample quality for higher speed (reported ≈7.7×).

“The lossless sampler uses a speculative acceptance criterion and samples from the residual distribution when proposals are rejected.”

Plainly: the conservative sampler is like a bouncer only letting tokens in that match the model’s tastes; the aggressive sampler broadens the guest list to boost throughput but accepts a few more mismatches.

Hardware and architecture choices

Key hardware and model microarchitecture choices materially affected realized speedups. Zyphra trained and evaluated on AMD MI300x and MI355x accelerators. On MI300x they could fit about three block‑sized proposals per forward pass; on MI355x (bf16) about five.

Architecture choices included Zyphra’s CCA attention (a cache‑efficient attention variant) and CCGQA (a compressed cross‑granularity query‑attention with a 4:1 query‑to‑key head ratio and 2× compression). Zyphra avoided MLA (Multi‑Head Latent Attention) because its arithmetic intensity conflicted with their hardware goals.

What this means: hardware/model co‑design matters. The conversion payoff depends on how many block proposals the accelerator can evaluate per pass and how attention primitives reduce prefill FLOPs.

Results, evaluation and caveats

Headline numbers reported by Zyphra:

Block decode size: 16 tokens per block.
Lossless sampler: ~4.6× speedup with no systematic eval degradation (reported on pass@ metrics on a mid‑train checkpoint, not yet RL‑fine‑tuned).
Logit‑mixing sampler: ~7.7× speedup with some quality trade‑offs.
Training overhead: ≈1.1 trillion additional tokens of mid‑training and context extension.

Comparisons to alternatives: Zyphra reports higher net speedups vs multi‑token prediction (MTP) and several speculative/multi‑token methods (EAGLE3, dFlash) in their MoE tests, largely because converting an existing pretrained MoE reuses learned capabilities while changing the decoding regime.

Caveats: reported numbers come from a preview on a mid‑training checkpoint and use pass@ evaluation; RL fine‑tuning, broader benchmarks, and different prompt distributions may change acceptance rates and quality. Diffusion‑LM tooling (serving, monitoring, regression testing) is also less mature than autoregressive stacks, so operational integration requires nontrivial engineering.

Business implications: AI for business, AI automation and RL rollouts

Faster, more throughput‑efficient inference directly reduces the cost of on‑policy RL rollouts and can accelerate iteration cadence for agents and production models. For teams that pay for thousands of rollouts or serve high volumes of requests, a 4–8× decoder throughput gain amplifies experimental velocity and reduces operational spend.

Practical angle for leaders: treat inference cost as a first‑class optimization goal. Investing in hardware‑aware model designs (attention variants like CCA/CCGQA) and experimenting with speculative/diffusion decoding could unlock orders‑of‑magnitude improvements in effective throughput per GPU.

Hypothetical ROI example (illustrative)

Use this simple formula: break‑even rollouts = conversion_compute_cost / savings_per_rollout.

Hypothetical numbers (for illustration only): if converting costs the equivalent of 10,000 GPU‑hours and each rollout’s inference cost falls by 0.1 GPU‑hour after conversion, break‑even occurs after 100,000 rollouts. Adjust the inputs for your pricing and rollout volumes to estimate whether conversion pays off.

Actionable next steps and checklist for teams

Benchmark a converted checkpoint on your real prompts and RL loops. Track per‑prompt latency, acceptance rate, pass@, and end‑to‑end reward stability.
Run sensitivity sweeps: block sizes (4/8/16/32), sampler thresholds, and prompt lengths.
Estimate engineering cost to add block generation to your inference stack: serving changes, token streaming, monitoring, and regression suites.
Validate safety and policy enforcement under block decoding—test hard constraints and auditability for rejected/accepted token paths.
Decide which sampler to use by workload: conservative (lossless) for high‑assurance production, aggressive (logit‑mixing) for experimental throughput where minor quality variation is acceptable.

Limitations and open questions

Portability: how well do these speedups transfer to non‑AMD hardware (NVIDIA, other accelerators)?
RL fine‑tuning effects: acceptance behavior and downstream reward stability after RL remain to be seen.
Prompt sensitivity: acceptance rates and quality tradeoffs may vary across longer chat histories, chain‑of‑thought prompts, or constrained generation tasks.
Operational maturity: tooling for diffusion LMs—streaming, monitoring, and per‑token constraints—is still early stage compared with autoregressive ecosystems.

For infra engineers (microarchitecture notes)

Definitions: MoE = Mixture‑of‑Experts; KV‑cache = key/value cache used for attention; TiDAR = conversion recipe blending TiDAR‑style objectives (mid‑training diffusion); CCA = cache‑efficient attention variant; CCGQA = compressed cross‑granularity query attention; MLA = Multi‑Head Latent Attention.

Practical tips:

Measure arithmetic intensity and memory bandwidth usage before conversion; block diffusion helps most when memory bandwidth is the bottleneck.
Profile how many block proposals your hardware can evaluate per forward pass—this directly scales realized throughput.
Plan for added pretraining cost (here ≈1.1T tokens) as an upfront engineering investment; monitor acceptance rates to tune sampler parameters post‑conversion.
Expect to add new monitoring signals: acceptance rate, residual sampling distribution checks, and block‑level perplexity.

Practical benchmarking recipe you can run today

Select 1–2 representative prompt sets (single‑turn, multi‑turn, chain‑of‑thought) and an RL rollout workload if applicable.
Measure baseline: end‑to‑end latency, GPU utilization, and cost per rollout under your autoregressive stack.
Convert a checkpoint (or use Zyphra’s preview if available), run with lossless sampler, and record the same metrics plus acceptance rate and pass@.
Repeat with logit‑mixing sampler and chosen block sizes. Track quality drift with human eval or downstream RL reward.
Decide based on business metrics: latency targets, cost per rollout, and acceptable quality delta.

Limitations, safety and auditability

Block decoding changes how intermediate token proposals are generated and accepted. That affects deterministic editing, constrained generation, and fine‑grained policy enforcement. Build regression tests that enforce safety constraints at the block level and ensure audit trails record both proposals and finalized tokens where necessary.

FAQ

Can an autoregressive MoE be converted without losing quality?

Zyphra reports the conversion can be done with “no systematic loss” in evaluation performance using a conservative (lossless) sampler on their mid‑train checkpoint. That claim is promising but needs broader validation across RL‑fine‑tuned checkpoints and diverse prompt distributions.

How large is the extra training effort?

The conversion used about 1.1 trillion additional tokens total: roughly 600B tokens for diffusion mid‑training and ~500B tokens for context extension and diffusion SFT.

Which sampler should I use?

Use the lossless sampler for production where quality is critical; the logit‑mixing sampler can be useful for experimental settings where throughput trumps small quality losses.

Will this replace autoregressive decoding everywhere?

Not immediately. Diffusion tooling is less mature, and practical adoption depends on RL effects, prompt sensitivity, and engineering investment to adapt serving stacks. But for compute/memory–imbalanced deployments, conversion is a compelling option to explore.