arXiv · 2024 · Preprint
Han et al. (Microsoft) · → Paper · Demo: ✓ · Code: ?
VALL-E R addresses two persistent weaknesses of neural codec language model TTS: robustness failures (omissions, repetitions, typos) and slow autoregressive inference, by introducing phoneme monotonic alignment during decoding and a codec-merging approach that halves the first-layer RVQ sampling rate, achieving WER near ground-truth levels and over 60% inference speedup.
Problem
Decoder-only autoregressive language models applied to TTS, exemplified by VALL-E, achieve strong naturalness and zero-shot speaker cloning but suffer from two structural problems. First, the absence of hard monotonic constraints on phoneme-to-audio alignment causes attention collapse in long sequences, leading to word skipping, repetition, and mis-readings — especially in cross-sentence (zero-shot) scenarios where the content differs from the acoustic prompt. Second, the high temporal resolution of EnCodec tokens (75 Hz) means that generating 10 seconds of speech requires approximately 750 autoregressive steps in the first RVQ layer alone, creating substantial inference latency. Prior robustness fixes either interleave phoneme tokens into the autoregressive stream (ELLA-V), which paradoxically increases latency, or combine the model with a Transducer architecture (VALL-T), which substantially slows training. No prior work addressed inference efficiency within the neural codec LM paradigm.
Method
VALL-E R retains the two-stage AR/NAR structure of VALL-E: a 12-layer Transformer autoregressive model generates first-layer RVQ tokens, followed by a 12-layer non-autoregressive model that produces RVQ layers 2-8 in parallel. Two innovations are layered onto this foundation.
The codec-merging approach modifies the EnCodec inference forward pass — without retraining the codec — to downsample the first-layer residual representation by 2x via average pooling before quantisation, then repeating each resulting code twice to restore the original sequence length. This ensures every pair of consecutive frames shares the same discrete token, halving the number of distinct codes the AR model must generate. Because Transformer self-attention has quadratic complexity, halving the sequence length yields a speedup exceeding 2x. Applying merging only to the first RVQ layer (rather than multiple layers) preserves audio quality: PESQ and STOI drop negligibly when only layer 1 is merged at 2x, while merging four or eight layers causes significant degradation.
The phoneme monotonic alignment strategy integrates phoneme prediction into AR training. The model is trained to jointly generate the first-layer acoustic token and its aligned phoneme at each timestep using teacher forcing, with a forced aligner (MFA) providing ground-truth phoneme-to-frame alignments. At inference, a monotonic alignment (MA) constraint operates on the predicted phoneme logits: at each step, the phoneme pointer can either stay on the current phoneme or advance to the next one, sampled via a Bernoulli decision. This guarantees three properties by construction: locality (each phoneme maps to at least one acoustic token), monotonicity (phoneme ordering is preserved), and completeness (no phoneme is skipped). Unlike encoder-decoder monotonic attention methods, this approach is compatible with the decoder-only Transformer architecture.
Because VALL-E R explicitly tracks phoneme progress during inference, it also supports prosody control: replacing the self-predicted phoneme sequence with a preset aligned phoneme sequence from a reference utterance allows timbre and prosody to be controlled independently, enabling a form of voice conversion.
Key Results
On LibriSpeech test-clean under the continuation task (3-second prompt, same sentence), VALL-E R achieves WER of 1.58%, compared to VALL-E’s 2.37% and ELLA-V’s 2.10%. The 1.58% WER is essentially at the EnCodec reconstruction upper bound (1.62%), demonstrating near-perfect robustness. Speaker similarity (Spk-Sim) is 0.876, matching VALL-E (0.875).
Under the cross-sentence task (different target sentence, harder zero-shot case), VALL-E R achieves WER of 3.18%, outperforming VALL-E (5.48%) and ELLA-V (7.15%). Spk-Sim is 0.974, on par with VALL-E and ELLA-V (both 0.975).
Subjectively (Table 2), VALL-E R scores QMOS 4.02 vs. VALL-E’s 3.96, SMOS 3.89 vs. 3.84, and CMOS +0.07 over VALL-E.
For inference efficiency (Table 4), VALL-E R (2x merging) generates 10 seconds of speech in 3.67 seconds (375 AR steps), compared to VALL-E’s 10.27 seconds (750 AR steps) — a 64% reduction. This also outpaces flow-matching-based VoiceBox (6.4 seconds) and CLaM-TTS (4.15 seconds).
Ablation results (Table 6) confirm that codec-merging alone does not harm WER or Spk-Sim, and removing monotonic alignment reverts performance to VALL-E-level WER.
Novelty Assessment
The two contributions are genuine and complementary. The codec-merging approach is an elegant, training-free inference acceleration: modifying the codec’s quantisation forward pass to enforce frame grouping requires no codec retraining and imposes almost no quality cost when applied to a single RVQ layer. This is a practical engineering insight that is non-obvious and immediately reusable by other neural codec LM systems. The phoneme monotonic alignment strategy adapts a well-studied idea from encoder-decoder TTS (monotonic attention) to the decoder-only Transformer setting, which required rethinking the alignment mechanism as an explicit inference-time pointer rather than an attention constraint. The combination of the two within a single system that simultaneously improves robustness and speed is the paper’s main architectural contribution. That said, neither idea is paradigm-breaking in isolation: codec-merging is a straightforward bitrate reduction, and phoneme-guided monotonic decoding builds directly on established forced alignment and CTC-style pointer logic. The evaluation is conducted under controlled, same-architecture conditions and is fair; however, the test domain is limited to clean English read speech (LibriSpeech), leaving robustness in noisy or expressive conditions undemonstrated.
Field Significance
Moderate — VALL-E R identifies and simultaneously addresses both major practical failure modes of neural codec LM-based TTS (robustness and latency) without requiring codec retraining or architectural changes to the LM backbone. Its codec-merging insight — that frame-level RVQ token grouping can be imposed at inference without quality loss — provides a reusable efficiency strategy for any autoregressive codec TTS system. The phoneme monotonic alignment approach demonstrates that decoder-only LMs can achieve near-codec-ceiling robustness when phoneme control is made explicit during inference.
Claims
- Phoneme monotonic alignment in decoder-only autoregressive TTS can close most of the robustness gap caused by unconstrained attention, achieving near-ground-truth WER without encoder-decoder architectural changes. (§3.2, Table 1)
- Downsampling only the first RVQ layer of a neural codec at inference time reduces autoregressive steps and latency by more than half, with negligible impact on PESQ and STOI. (§3.1, Table 5)
- Robustness improvements that route additional phoneme tokens through the autoregressive stream (as in ELLA-V) improve WER but increase inference time, illustrating a robustness-efficiency trade-off in codec LM TTS. (§5.3, Table 4)
- Explicit phoneme-level alignment in a codec LM enables independent control of prosody and timbre by substituting preset phoneme sequences at inference, enabling a form of voice conversion. (§3.2.3, Table 3)
Limitations and Open Questions
Warning
All evaluations use LibriSpeech (clean English read speech). Robustness gains from monotonic alignment and codec-merging quality preservation have not been tested on noisy, expressive, or multilingual speech.
The model size is not explicitly reported, though the architecture (12-layer Transformer, 1024-dim hidden, 16 heads) matches the VALL-E reference scale. Code and model weights are not publicly released, limiting reproducibility. The prosody control evaluation uses MCD-DTW-SL, a proxy metric; perceptual validation of prosody cloning quality is absent. The merging rate of 2x is validated by reconstruction metrics but its downstream effect on naturalness under diverse speaker and content conditions is not fully explored. RALL-E (chain-of-thought prompting for robustness) is included only in the efficiency comparison, not in the WER robustness comparison, making head-to-head robustness assessment with concurrent work incomplete.
Wiki Connections
- autoregressive-codec-tts — extends VALL-E’s AR/NAR codec LM structure with phoneme monotonic alignment and codec-merging
- zero-shot-tts — zero-shot speaker cloning from a 3-second reference prompt, evaluated on LibriSpeech test-clean
- neural-codec — EnCodec 75Hz with first-layer 2x merging applied at inference without codec retraining
- prosody-control — phoneme alignment enables prosody cloning by substituting preset phoneme sequences at inference
- speaker-adaptation — cross-sentence zero-shot voice cloning from a reference utterance differing in content
- 2301.02111 (VALL-E) — direct predecessor; VALL-E R inherits the AR/NAR two-stage structure and adds robustness/efficiency fixes
- 2210.13438 (EnCodec) — underlying codec with codec-merging applied to the first RVQ layer
- 2306.00814 (Vocos) — vocoder used to improve audio quality over raw EnCodec decoding
- 2401.07333 (ELLA-V) — concurrent robustness work compared on WER and inference efficiency
- 2404.03204 (RALL-E) — concurrent efficiency work compared on inference speed
- 2303.03926 (VALL-E X) — multilingual extension of VALL-E; cited for context