arXiv · 2023 · Preprint
Zhang et al. (Microsoft) · → Paper · Demo: ✓ · Code: ?
VALL-E X extends the VALL-E codec language model to cross-lingual speech synthesis, enabling zero-shot transfer of a speaker’s voice, emotion, and acoustic environment across languages from a single source utterance.
Problem
Cross-lingual TTS has long struggled with two compounding problems: the scarcity of bilingual speech data from the same speakers, and the inability of conventional encoder-decoder models to transfer rich voice characteristics across languages. Prior systems like YourTTS rely on speaker embeddings extracted from reference recordings, which flatten voice identity into a fixed vector and discard fine-grained acoustic detail. As a result, these models produce synthesised speech that sounds foreign (L2 accent) and poorly preserves the source speaker’s timbre, emotion, and background acoustics. Zero-shot capability, where a model handles an entirely unseen speaker with no fine-tuning, has been entirely absent from prior cross-lingual TTS work.
Method
VALL-E X treats cross-lingual speech synthesis as a conditional multilingual language modelling task over discrete acoustic tokens, extending the VALL-E architecture to handle bilingual prompts. The codec backbone is EnCodec at 75 Hz with 8 residual vector quantisation layers (L=8), producing 1024-entry codebooks per layer.
The model consists of two components. A multilingual autoregressive codec language model (φ_MAR) is a 12-layer Transformer decoder (attention dimension 1024, FFN dimension 4096) that generates first-layer acoustic tokens from a concatenated prompt of source-language phonemes, target-language phonemes, and source first-layer acoustic tokens. A multilingual non-autoregressive codec language model (φ_MNAR), a parallel 12-layer Transformer, then iteratively fills layers 2 through 8 using greedy decoding, conditioned on the target phonemes and all source acoustic tokens.
A language ID module embeds a target-language identifier into the acoustic token embeddings of φ_MAR. This provides a controllable signal to guide the model toward native-sounding target speech and suppress the L2 accent tendency that emerges from multilingual training. At inference, the source acoustic tokens serve as a prefix that seeds the target generation with the speaker’s characteristic voice.
For speech-to-speech translation (S2ST), VALL-E X is paired with a speech recognition and translation front-end (VALL-E X Trans), derived from an improved SpeechUT model pre-trained on ASR and MT corpora. The front-end generates both source and target phoneme sequences from input speech, which are then fed to VALL-E X for acoustic generation.
Training uses approximately 70K hours of multilingual speech: ~60K hours of English audiobook speech from LibriLight (with pseudo-transcripts from a Kaldi ASR model) and 10K+ hours of Chinese multi-domain speech from WenetSpeech. Both φ_MAR and φ_MNAR are trained independently on 32 V100 GPUs for 800K steps.
Key Results
On zero-shot cross-lingual TTS (English generation prompted by Chinese speech), VALL-E X substantially outperforms YourTTS across all metrics: speaker similarity improves from 0.30 to 0.36 ASV-Score, WER drops from 8.53 to 4.07, and NISQA naturalness rises from 3.36 to 3.54 (Table 2). Human evaluation on 50 English test sentences confirms the gap: SMOS improves from 3.42 to 4.00, and CMOS is +0.24 in favour of VALL-E X (Table 3).
On bidirectional zero-shot S2ST (EMIME dataset, 350 utterances), VALL-E X Trans achieves SMOS of 4.12 vs. 3.06 for the cascaded baseline on Chinese-to-English, a gain of more than 1 point (Table 5). ASR-BLEU for VALL-E X Trans reaches 30.66 vs. 27.49 for the baseline on Chinese-to-English, and the upper-bound oracle-text condition yields 86.78 BLEU, demonstrating the codec language model itself generates highly intelligible speech given correct phonemes.
Ablation of the language ID module confirms its importance: removing it drops accent scores from 4.10 to 2.98 (scale 1-5, higher is more native-sounding) and degrades translation quality by roughly 1.5 BLEU (Table 6). Counterintuitively, removing the language ID modestly increases ASV-Score, indicating a trade-off between accent control and pure voice fidelity.
Novelty Assessment
The architectural contribution is a clean and consequential extension of VALL-E to the cross-lingual setting. The two-stage AR/NAR design, the bilingual phoneme prompt, and the language ID module are all adaptations of existing ideas (in-context learning, multilingual conditioning, token prefix prompting) rather than structural inventions. The genuine novelty lies in showing that codec language modelling scales naturally to cross-lingual transfer without requiring paired bilingual data from the same speakers, a constraint that had limited all prior work. The integration of a speech recognition and translation front-end for S2ST is engineering integration rather than a new architecture. The training data scale (70K hours across two languages) is large for the period and is itself a meaningful contributor to the results.
Field Significance
Tip
High — VALL-E X demonstrates that the in-context learning capability of codec language models transfers to cross-lingual settings, resolving the L2 accent problem and enabling zero-shot voice preservation without paired bilingual speaker data. It extends the VALL-E paradigm to multilingual and speech-to-speech translation tasks, providing a blueprint for subsequent cross-lingual codec TTS systems and establishing autoregressive codec LMs as a viable architecture for cross-lingual voice transfer.
Claims
- Large-scale multilingual codec language models can transfer speaker identity, emotion, and acoustic environment across languages from a single source utterance without paired bilingual data. (§3, §5.3, Table 3)
- Language ID conditioning is essential for native-sounding accent in cross-lingual codec TTS: removing it significantly degrades accent scores even while modestly improving speaker similarity. (§5.5, Table 6)
- In-context learning with acoustic token prompts provides stronger cross-lingual voice preservation than speaker embedding approaches across both TTS and speech-to-speech translation tasks. (§5.3, §5.4, Tables 3, 5)
- The AR/NAR two-stage codec language model architecture extends naturally to cross-lingual generation by treating bilingual phoneme sequences as concatenated prompts. (§3.2, §3.4)
Limitations and Open Questions
Warning
Evaluations cover only English and Chinese, and test sets are small (40 speakers, 1373 samples for TTS; 14 speakers, 350 utterances for S2ST on EMIME). Generalisation to typologically distant language pairs, more than two languages, or lower-resource settings is untested.
The EMIME dataset is bilingual by construction (same speakers recorded in both languages), making the ASV upper bound (tgt vs. src) unusually informative but also artificially favourable for a cross-lingual system. The ASV gap between the model and the upper bound for English-to-Chinese (0.48 vs. 0.58) suggests substantial room for improvement in voice transferability in that direction.
The language ID ablation reveals an inherent tension: stronger accent control comes at the cost of voice similarity. The right operating point depends on application, and the paper does not explore soft or learnable trade-off mechanisms. No model size figures are reported, making cost comparison difficult. S2ST quality is bottlenecked by translation quality; the paper does not disentangle acoustic and linguistic error sources beyond the oracle-text upper bound analysis.
Wiki Connections
This paper is a direct extension of 2301.02111 (VALL-E), adopting its AR/NAR codec language model architecture and in-context learning approach for cross-lingual transfer. The codec backbone is 2210.13438 (EnCodec). It draws on 2209.03143 (AudioLM) for the broader framework of treating audio generation as language modelling over discrete tokens.
Relevant concept pages: autoregressive-codec-tts, zero-shot-tts, multilingual-tts, neural-codec, speech-to-speech.
2306.12925 — AudioPaLM: A Large Language Model That Can Speak and Listen 2310.00704 — UniAudio: An Audio Foundation Model Toward Universal Audio Generation 2402.01912 — Natural language guidance of high-fidelity text-to-speech with synthetic annotations 2402.08093 — BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data 2403.16973 — VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild 2406.04904 — XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model 2407.08551 — Autoregressive Speech Synthesis without Vector Quantization