workshop · 2025 · Workshop
Yaping Liu et al. (Kunming University of Science and Technology) · → Paper · Demo: ? · Code: ✗
Extends VALL-E X for Lao-English code-switched TTS by projecting Lao and English phoneme representations into a shared latent space via dot-product attention and adding per-token language ID embeddings, overcoming the speaker-data bottleneck of bilingual corpus construction.
Problem
Lao is an extremely low-resource language with no publicly available bilingual (Lao-English) speech corpora recorded by a single speaker, making it difficult to train a code-switched TTS system. The VALL-E X multilingual codec LM, while pre-trained on English, struggles with cross-lingual phoneme transfer for Lao because phoneme sets differ and no shared representational space exists between the two languages. Simply concatenating monolingual corpora (VITS Joint) introduces phoneme confusion; training separate models (VITS Separate) yields prosodic discontinuities at language boundaries.
Method
The system builds on VALL-E X — an autoregressive + non-autoregressive Transformer decoder pair that operates on EnCodec tokens. Two modifications are introduced:
1. Cross-lingual phoneme latent space. Frame-level speech features (extracted from a pretrained encoder) are averaged within phoneme boundaries to produce phoneme-level vectors. These serve as queries in a scaled dot-product attention module, with the pretrained encoder outputs as keys and values. The resulting latent representations H for Lao and English phonemes are fused into a shared space and concatenated with the phoneme sequence fed to the codec LM. This maps both languages’ phonetic characteristics into a common representation, reducing the mismatch at language-switch boundaries.
2. Language ID embeddings. Each input token is assigned a language identifier (LID) from a predefined dictionary. The LID is embedded into a dense vector and added to the acoustic token embeddings, providing explicit conditioning on which language’s phonetic rules govern each segment. For mixed-language inputs, per-token LIDs are assigned individually.
Training follows VALL-E X’s autoregressive (AR) and non-autoregressive (NAR) scheme. Training data: LibriTTS (~585h English) + 100h proprietary Lao corpus (380 speakers, 15,965 clips, mono 24kHz). No real code-switched audio is used — the model learns cross-lingual synthesis entirely from monolingual sources.
Key Results
From Table 2 (CMOS vs VALL-E X baseline, lower is better when negative; proposed model closer to 0):
| Model | En-speaker → En text | Lo-speaker → Lo text | En-speaker → Lo text | Lo-speaker → En text |
|---|---|---|---|---|
| VALL-E X | CMOS –0.11 | CMOS –0.97 | CMOS –1.21 | CMOS –0.78 |
| Proposed | CMOS –0.09 | CMOS –0.27 | CMOS –0.10 | CMOS –0.30 |
The largest gain is for Lao speakers generating English (+0.70 CMOS) and English speakers generating Lao (+1.11 CMOS). MOS of the proposed model peaks at 3.69 (full model) versus 3.29–3.54 across VALL-E X conditions. AB preference: 75% of 10 bilingual judges preferred the proposed model; 5% preferred VALL-E X baseline.
Ablation (Table 4): removing the phoneme latent space (PC) drops MOS by 0.10 and CMOS to –1.26; removing LID drops MOS by 0.06 and CMOS to –0.54. The phoneme space contributes more than the LID module.
RMSE also improves: 52.25 (proposed) vs 55.43 (VALL-E X) for English; similar improvement on Lao channel.
VITS baselines (Table 3) score substantially lower (CMOS –0.42 to –1.50), confirming that codec LM-based approaches outperform seq2seq VAE architectures for code-switching.
Novelty Assessment
Primarily engineering/application. The contribution is the integration of two well-known components — cross-lingual phoneme alignment via attention and language ID conditioning — into VALL-E X for a specific underserved language pair (Lao-English). Neither the phoneme latent space construction nor LID conditioning is novel in isolation. The novelty is: (a) applying this to Lao, a truly low-resource language, without any real code-switched training data, and (b) demonstrating that this combination beats VITS-based architectures decisively. The proprietary Lao corpus is not released.
Field Significance
Moderate — This paper demonstrates that a phoneme-level shared latent space combined with per-token language ID conditioning can enable code-switched TTS for a low-resource language without any bilingual training data. Its contribution is primarily application-focused, extending an established codec LM architecture to an underserved language pair. It provides useful evidence that the VALL-E X framework generalises to low-resource cross-lingual scenarios, though the proprietary dataset limits reproducibility.
Claims
- Projecting phoneme representations from two languages into a shared latent space reduces cross-lingual phoneme confusion and improves naturalness in code-switched speech synthesis. (§3.2, Table 2)
- Per-token language ID conditioning helps a multilingual codec LM distinguish phonetic characteristics across languages in code-switched synthesis, though its impact is smaller than that of shared phoneme representations. (§3.3, Table 4)
- Codec language model architectures outperform VAE-based seq2seq systems for code-switched TTS when trained on monolingual data only. (§4.3.2, Table 3)
- Code-switched TTS systems can be trained effectively from monolingual corpora alone, without requiring real bilingual or code-switched training audio. (§3.1, §4.1)
Limitations and Open Questions
- Proprietary Lao dataset is not publicly available, limiting reproducibility.
- Evaluation uses a small number of listeners (10 Lao, 10 English), raising statistical concerns.
- RMSE is used as the sole objective metric; no WER, CER, or SPK-SIM reported.
- The approach does not generalize beyond Lao-English without new proprietary data per language.
- No streaming or real-time inference analysis.
- How the model handles intra-word code-switching (vs. inter-sentence) is not evaluated.
Wiki Connections
- autoregressive-codec-tts — the base VALL-E X architecture that this system extends
- multilingual-tts — this paper’s core setting: cross-lingual synthesis for a low-resource language pair
- neural-codec — EnCodec is the acoustic representation backbone, inherited from VALL-E X
- zero-shot-tts — the approach achieves cross-lingual synthesis without bilingual speakers, connecting to zero-shot transfer goals