Decoding Order Matters in Autoregressive Speech Synthesis

arXiv · 2026 · Preprint

Minghui Zhao and Anton Ragni (University of Sheffield) · → Paper · Demo: ? · Code: ?

Uses a masked diffusion framework to systematically compare speech synthesis decoding orders, showing that the near-universal left-to-right convention is suboptimal, adaptive confidence-guided ordering performs better, and even 1-bit scalar quantization of mel-spectrograms can support intelligible speech.

Problem

Autoregressive speech synthesis has almost universally used left-to-right decoding, from early systems like WaveNet and Tacotron 2 through more recent language-model-based approaches, generating each frame conditioned only on previously produced output. But the paper argues left-to-right order is not obviously optimal from a modelling perspective: speech dependencies extend beyond a simple causal chain (pauses and emphasis depend on global context; coarticulation reflects interactions between past and future phones), so even when a model is conditioned on the full utterance (e.g. the full phone sequence), the order in which frames are actually decoded can affect how well these dependencies are captured. Testing this directly with conventional autoregressive models would require training separate models under different fixed orders, which is intractable given there are T! possible orderings for a length-T sequence and prevents direct comparison within a single framework.

Method

The paper uses a masked diffusion model (MDM) as an order-agnostic autoregressive framework: during training, the model learns to predict randomly masked frames from visible ones (an ELBO-based, order-agnostic objective evaluated in parallel across masked positions, rather than requiring full sequential autoregression), and at inference any decoding order can be chosen and frames unmasked one at a time conditioned on what has already been revealed. To make comparisons directly analogous to conventional autoregressive TTS, the model is constrained to decode one frame at a time (update size k=1) without replacement in most experiments. The backbone architecture is based on Grad-TTS (text encoder, duration predictor, and a decoder reused here as the MDM), trained and evaluated on LJSpeech using the same split as Grad-TTS.

Because MDMs require discrete inputs, Mel-spectrograms are quantized with a simple linear scalar quantizer applied independently per frequency bin (not a learned VQ-style tokenizer), producing Q quantization levels per bin; the decoder predicts a mixture-of-5-logistics distribution per bin, sampled via the Gumbel-Max trick at inference, and HiFi-GAN (used off-the-shelf, without retraining) vocodes the dequantized mel output. Several decoding strategies are compared: fixed left-to-right (l2r) and right-to-left (r2l) orders; a fully random default order (uniformly sampled permutation); randomness-controlled orders that interpolate between l2r and fully random via a swap-based parameter β; adaptive Top-K decoding, which at each step scores all undecoded positions by summed maximum log-probability across frequency bins and decodes the highest-confidence position(s) next (top1 always selects the most probable value, top1* samples from the predicted distribution); and a proposed duration-guided decoding scheme, motivated by an observation that Top-K decoding tends to select contiguous frames, which uses the duration predictor to define candidate segments, picks the segment with highest average confidence, then decodes its frames one by one in random order.

Key Results

Preliminary quantization experiments (public HiFi-GAN checkpoint, no retraining) show 10-class quantization preserves high mel-spectrogram quality, and even 2-class (1-bit) quantization yields partially intelligible speech with recognizable words; training and evaluating a full model on 1-bit representations achieves an average MCD of 4.21 and average log F0 of 0.211, motivating the paper’s adoption of 100-class quantization as a practical balance. Varying decoding-order randomness (β) shows metric-dependent, sometimes conflicting effects: as randomness increases, MCD improves while UTMOS degrades, and log F0 is minimized at an intermediate β where order is neither strictly sequential nor fully random. Comparing decoding strategies directly, right-to-left decoding outperforms left-to-right on UTMOS and MCD with similar log F0, demonstrating left-to-right is not optimal even among fixed orders; duration-guided decoding and top1* (sampling-based adaptive decoding) perform best overall on automatic metrics and MOS, with top1* ranking highest among all tested MDM configurations after the vocoded reference itself. The Grad-TTS baselines (fixed 100-step and length-based decoding) achieve the highest UTMOS among all models but the worst MCD, and rank lower than top1*/duration-guided on subjective MOS despite their strong automatic UTMOS scores, a discrepancy the authors relate to their higher MCD. Deterministic top1 decoding (always selecting the most probable value) underperforms top1* on most metrics, which the authors attribute to over-smoothed outputs. Increasing the number of simultaneously decoded frames K in Top-K decoding improves MCD and log F0 (via shared context across jointly decoded frames) but reduces UTMOS, indicating a further multi-frame-update trade-off distinct from the single-frame ordering trade-offs.

Novelty Assessment

The core contribution is methodological: using masked diffusion’s order-agnostic training as a controlled experimental instrument to directly compare decoding orders within a single trained model, something conventional autoregressive TTS training cannot easily support (it would require training separate models per fixed order, which is intractable at scale). This lets the paper make a genuinely falsifiable and specific claim, left-to-right is suboptimal, with a controlled comparison (r2l vs. l2r, then several adaptive strategies) rather than an anecdotal or purely theoretical argument. The scalar-quantization finding is a secondary but notable contribution: showing that HiFi-GAN, trained on continuous mel-spectrograms, can vocode heavily quantized (even 1-bit) mel input without retraining suggests exact mel values may be less important to neural vocoders than their relative distribution, though the paper is explicit this specific claim is untested for other vocoder architectures. The duration-guided decoding scheme is a reasonable but relatively simple heuristic response to an observed pattern (Top-K tends to decode contiguous frames) rather than a fully principled solution; the authors themselves suggest reinforcement learning as a more principled way to learn an optimal joint order-and-update-size schedule, framing their proposed heuristics as a starting point rather than a final answer.

Field Significance

Moderate, the paper provides a well-controlled empirical challenge to a near-universal but rarely questioned modelling convention (left-to-right decoding) in autoregressive speech synthesis, with a genuinely useful secondary finding about the redundancy of precise mel-spectrogram values for neural vocoding. Its scope is narrow (single speaker, single English dataset, a Grad-TTS-scale model rather than a large modern TTS system), so the practical impact on current large-scale codec-token-based AR TTS systems (which this paper explicitly does not use) remains an open question the paper itself flags rather than resolves.

Claims

• contradicts: Left-to-right decoding order, the dominant convention in autoregressive speech synthesis, is not the optimal generation order for producing high-quality speech.

  Evidence: Under a masked-diffusion framework enabling controlled decoding-order comparison, right-to-left decoding outperforms left-to-right decoding on UTMOS and MCD with similar log F0, and adaptive strategies (top1*, duration-guided) consistently outperform both fixed orders across automatic and subjective (MOS) metrics. (§4.3, Figure 4, Figure 5)

• supports: Adaptive, confidence-guided decoding order selection outperforms both fixed left-to-right/right-to-left decoding and fully random decoding order in autoregressive speech synthesis.

  Evidence: Sampling-based adaptive decoding (top1*) and duration-guided segment-wise decoding achieve the best or near-best scores across MCD, log F0, UTMOS, and MOS, while the fully random (“default”) decoding-order baseline underperforms these adaptive strategies. (§4.3, §4.4, Figure 4, Figure 5, Figure 6)

• complicates: The degree of randomness in decoding order has metric-dependent, sometimes conflicting effects on synthesized speech quality, so no single randomness level uniformly optimizes all quality dimensions.

  Evidence: As decoding-order randomness increases via a swap-based interpolation parameter, Mel-Cepstral Distortion improves while UTMOS degrades, and log F0 error is minimized at an intermediate randomness level rather than at either extreme. (§4.2, Figure 3)

• supports: Simple, untrained scalar quantization of Mel-spectrograms, without a learned vector-quantized tokenizer, can produce discrete acoustic tokens that an off-the-shelf neural vocoder reconstructs into intelligible speech even at extremely low bit depths.

  Evidence: Vocoding mel-spectrograms quantized to just 2 classes (1 bit) with an off-the-shelf HiFi-GAN checkpoint (no retraining) yields partially intelligible speech with recognizable words, and a full model trained and evaluated on 1-bit representations achieves an average MCD of 4.21 and average log F0 of 0.211. (§4.1, Figure 2)

Limitations and Open Questions

• All experiments use LJSpeech, a single-speaker, single-language (English) corpus; generalization of the decoding-order findings to multi-speaker, multilingual, or zero-shot TTS settings is untested.
• The backbone is a Grad-TTS-scale model rather than a large modern codec-token-based AR TTS system, so it is unclear whether the left-to-right suboptimality finding transfers to the discrete-codec-token AR TTS systems (e.g. VALL-E-style models) that dominate current practice.
• The authors explicitly note that identifying a jointly optimal decoding order and update size (K) likely requires a learned approach such as reinforcement learning, rather than the fixed heuristics (Top-K, duration-guided) evaluated in this paper.
• Whether the finding that HiFi-GAN can vocode heavily quantized mel-spectrograms without retraining generalizes to other neural vocoder architectures is explicitly flagged by the authors as unexplored.

Wiki Connections

• Diffusion TTS — uses a masked diffusion model as an order-agnostic training and inference framework to systematically study decoding order in speech synthesis, rather than as a standard denoising-based generative model.
• Neural Audio Codec — deliberately avoids learned vector-quantized speech tokenizers in favor of simple scalar quantization of Mel-spectrograms, finding an off-the-shelf vocoder can still reconstruct intelligible speech from heavily quantized input.
• Subjective Evaluation — collects Mean Opinion Score ratings from 10 Amazon Mechanical Turk Master Workers per audio sample to validate decoding-strategy comparisons.
• WaveNet — cited as an early autoregressive speech synthesis system exemplifying the left-to-right decoding convention this paper investigates.
• Tacotron 2 — cited alongside WaveNet as an early autoregressive TTS system following the left-to-right decoding convention.
• HiFi-GAN — used as the vocoder throughout, including off-the-shelf (without retraining) to test how much quantization of mel-spectrogram input it can tolerate.
• VALL-E — cited as representative of recent autoregressive TTS approaches that model discretized acoustic features with a language model, part of the broader left-to-right convention this paper challenges.
• DiTAR — discussed in related work as a semi-autoregressive diffusion-transformer approach to speech generation, distinguished from this paper’s fully order-agnostic masked-diffusion framing.
• Finite Scalar Quantization — discussed in related work as an alternative scalar-quantization tokenization scheme for images that, unlike this paper’s per-bin sampling, maps quantized vectors to single token IDs, which the authors note would cause a combinatorial vocabulary explosion in their per-frame mel setting.