arXiv · 2024 · Preprint
Song et al. · → Paper · Demo: ✓ · Code: ?
ELLA-V introduces interleaved phoneme-acoustic token sequences for autoregressive codec language model TTS, using forced-alignment supervision at training time to produce a self-aligned generation process that eliminates the infinite-silence and repetition failures endemic to VALL-E-style models.
Problem
VALL-E and its successors train an autoregressive language model on sequences where all phoneme tokens are concatenated ahead of all acoustic tokens. The model must learn monotonic phoneme-to-audio alignment purely through self-attention, with no explicit structural constraint. In practice this leads to three failure modes: repetitions, transpositions, and omissions (phonemes are skipped or duplicated), and catastrophic infinite-silence generation where the model loops on silence tokens indefinitely. The problem worsens substantially as the sampling temperature drops toward greedy decoding, making deterministic or near-deterministic inference strategies largely unusable for VALL-E.
Method
ELLA-V reorders the hybrid token sequence so that phonemes are interleaved with their corresponding acoustic tokens rather than prepended en bloc. Using the Montreal Forced Aligner (MFA), the system extracts forced alignment between text transcriptions and EnCodec (24 kHz, 75 Hz, 8 RVQ layers, codebook size 1024) acoustic tokens at training time. For each phoneme, the corresponding acoustic frames are sandwiched between the phoneme token and a special EndOfPhone (EOP) marker. A global-advance prefix, consisting of the full phoneme sequence, is prepended before the BOS token so the model has complete phoneme context from the start.
The first-layer RVQ codes are generated by a Generalized Autoregressive (GAR) codec language model, a 12-layer Transformer decoder with embedding and hidden dimension 1024 and FFN dimension 4096 (154.3M parameters). The training objective computes loss only on acoustic tokens, EOP, and EOS, excluding phoneme tokens and BOS. This design means the GAR model effectively generates audio phoneme-by-phoneme: at inference, whenever the model predicts EOP for the current phoneme, the next phoneme token is appended directly to the sequence, guaranteeing the model always knows which phoneme it is currently synthesizing. This structural self-alignment allows forced truncation of anomalously long phoneme segments (threshold 0.4 s), preventing infinite-silence failure.
An additional local-advance variant shifts the EOP token and the next phoneme token a few frames earlier in the sequence, allowing the model to exploit local context about the upcoming phoneme while predicting the current one’s final frames. A second-stage NAR model (also 154.3M parameters, same architecture) predicts RVQ layers 2-8 in parallel using the full hybrid sequence from the GAR stage.
Key Results
On the zero-shot TTS continuation task using LibriSpeech test-clean, ELLA-V achieves a WER of 2.28% versus 5.00% for a comparable VALL-E baseline trained on the same 960-hour LibriSpeech corpus (Table 2). Speaker similarity (SPK-SIM: 0.87 vs. 0.868) and SMOS (3.56 for both) are approximately matched, while CMOS shows a +0.10 naturalness advantage for ELLA-V. Both models use 154.3M parameters per sub-model; the baseline was reproduced and re-trained on LibriSpeech to ensure a fair comparison.
On 100 hard-case sentences designed with challenging phonetic patterns, ELLA-V reduces WER from 28.39% (VALL-E) to 12.79%, with all error categories lower (substitution: 17.79% vs. 7.76%; deletion: 5.36% vs. 3.40%; insertion: 5.24% vs. 1.63%) (Table 3). The stability advantage is especially pronounced under near-greedy decoding: VALL-E’s infinite-silence rate (INF%) reaches 87% at greedy decoding, while ELLA-V’s forced-truncation rate (CUT%) remains low across the entire top-p range.
Ablation (Table 4) confirms that both global advance (full phoneme prefix) and local phoneme interleaving contribute. Removing the global prefix (ELLA-V-noglobal) degrades WER to 5.00%, matching the VALL-E baseline; removing in-sequence phoneme tokens (ELLA-V-nophn) yields WER 3.51%, confirming both components are needed.
Novelty Assessment
The core contribution is a structural change to the training sequence rather than a new architecture or a new loss function. Using MFA-derived forced alignment to interleave phoneme tokens with their corresponding acoustic frames is a direct and elegant solution to the well-documented alignment fragility of VALL-E-style models. The GAR inference loop (EOP triggers phoneme append) follows naturally from this sequence structure. The idea that explicit local alignment constraints should be baked into the sequence order, rather than learned implicitly through attention, is the genuine insight.
The approach is incremental relative to VALL-E in the sense that the backbone architecture, codec, and training paradigm are unchanged; the contribution is the sequence reordering strategy. However, the stability gains are substantial and the design is principled. The reliance on MFA at training time adds a preprocessing dependency (forced alignment is not always available or accurate), which is a practical limitation the paper does not deeply analyze.
Field Significance
Moderate — ELLA-V provides a targeted and effective solution to the alignment-stability problem in autoregressive codec LM TTS, demonstrating that explicit phoneme-acoustic interleaving during training can largely eliminate the infinite-silence and repetition pathologies of VALL-E without sacrificing speaker similarity or naturalness. The method introduces a useful design principle (sequence-level alignment encoding) that subsequent work in the codec-LM TTS line can adopt without changing training infrastructure beyond the addition of an MFA preprocessing step.
Claims
- Interleaving phoneme tokens with their corresponding acoustic frames in the training sequence substantially reduces phoneme-level alignment errors in autoregressive codec LM TTS, including repetitions, transpositions, and omissions. (§3.2, §4.2, Table 3)
- Autoregressive codec language models that concatenate all phoneme tokens ahead of all acoustic tokens are prone to infinite-silence generation, with failure rates exceeding 80% under greedy decoding. (§1, Table 1)
- Explicit forced-alignment supervision at training time enables fine-grained phoneme-level control at inference, allowing deterministic truncation of abnormal synthesis and making greedy decoding viable. (§3.2.1, §3.3, Figure 5)
- Structural alignment constraints in the token sequence provide larger accuracy gains than naturalness or speaker similarity improvements, suggesting that intelligibility and speaker identity are relatively easy to preserve while alignment robustness remains the primary challenge. (§4.2, Table 2)
Limitations and Open Questions
Warning
Results are compared only against a VALL-E baseline reproduced on LibriSpeech 960h, not against the original VALL-E trained on 60k hours of LibriLight. The restricted training data limits direct comparison to the published VALL-E numbers and leaves open whether the gains hold at scale.
MFA forced alignment is a preprocessing dependency that may fail or produce noisy alignments for short utterances, spontaneous speech, or languages without reliable MFA models. The paper does not analyze alignment quality or its effect on synthesis when MFA makes errors.
The evaluation is limited to LibriSpeech, a clean read-speech corpus in English. Generalization to noisy, spontaneous, or non-English settings is untested. The hard-case set of 100 sentences is internally designed rather than a standard benchmark, limiting reproducibility of the cross-speaker evaluation.
The local-advance hyperparameter (how many frames ahead to shift EOP and the next phoneme) is evaluated informally; systematic tuning across speech rates and phoneme types is left for future work.
Wiki Connections
- autoregressive-codec-tts — ELLA-V directly extends the VALL-E framework by introducing alignment-guided sequence reordering as a solution to the stability failures of standard AR codec LM TTS.
- zero-shot-tts — the system targets zero-shot voice cloning, synthesizing unseen speakers from a 3-second acoustic prompt without speaker-specific training.
- neural-codec — EnCodec (24 kHz, 75 Hz, 8 RVQ layers) is the discrete representation backbone; the interleaving strategy is tightly coupled to the per-layer RVQ structure.
- 2301.02111 (VALL-E) — ELLA-V directly extends VALL-E’s architecture and evaluates against a reproduced baseline, addressing VALL-E’s alignment fragility and infinite-silence failure modes.
- 2303.03926 (VALL-E-X) — cited as a multilingual extension of the same codec LM framework; ELLA-V’s sequence reordering could in principle be applied to VALL-E-X’s cross-lingual setting.
- 2304.09116 (NaturalSpeech 2) — cited as a diffusion-based alternative for zero-shot TTS; represents the competing paradigm to the AR codec LM approach.
- 2306.12925 (AudioPaLM) — cited as a related large-scale audio language model system in the broader context of AR codec generation.