ACL · 2025 · Conference
Thi Vu et al. (Movian AI) · → Paper · Demo: ? · Code: ?
This paper presents PhoAudiobook, a 941-hour high-quality Vietnamese audiobook dataset with speaker identity labels, and benchmarks three zero-shot TTS systems (VALL-E, VoiceCraft, XTTS-v2) trained on it against the prior viVoice-trained baseline.
Problem
Low-resource languages such as Vietnamese lack large-scale, high-quality speech datasets suitable for training zero-shot TTS models. Existing Vietnamese datasets (VinBigData, VietnamCeleb, VLSP, BUD500, viVoice) are either small (101–462 hours), lack speaker identity information, consist only of very short segments (typically under 10 seconds), suffer from poor audio quality, or lack text normalization. These deficiencies limit the ability of codec language model architectures — which benefit from long-form training audio and accurate transcriptions — to be applied to Vietnamese.
Method
The authors construct PhoAudiobook via an automated pipeline applied to ~23K hours of Vietnamese audiobook audio scraped from sachnoiviet.net. The pipeline uses Demucs for source separation (vocal extraction), Whisper-large-v3 for transcription and timestamping, and PhoWhisper-large as a quality filter (only samples where both ASR systems agree exactly are retained). Short segments are concatenated to produce 10–20 second samples. Multi-speaker filtering is applied with a wav2vec2-BARTpho model. Audio is volume-normalized via sox. Transcriptions are normalized using an mbart-large-50 fine-tuned sequence-to-sequence model. Each speaker is capped at 4 hours. The final dataset has 941 hours from 735 speakers at 16 kHz, with explicit speaker IDs and normalized text — properties absent or incomplete in prior Vietnamese corpora.
Three zero-shot TTS models are adapted to Vietnamese by fine-tuning on the 1,494-hour augmented set (940h long-form + 554h short-clip augmentation):
- VALL-E: An autoregressive codec language model operating on EnCodec discrete tokens at the phoneme level. A Vietnamese Wav2Vec 2.0 model is fine-tuned for dialect classification to select the correct phonemizer per speaker. The AR model uses 12 Transformer-decoder layers, 1024 hidden units, 16 heads, trained on 8×A100-40GB GPUs.
- VoiceCraft: A token-rearrangement-based neural codec language model (830M_TTSEnhanced checkpoint). Fine-tuned with AdamW (lr 1e-5) on 4×A100-40GB GPUs for 16 epochs. Vietnamese phonemes are appended to the existing English vocabulary.
- XTTS-v2: A multilingual voice-cloning model built on Tortoise, using BPE text encoding. Fine-tuned from the 16-language public checkpoint with AdamW (lr 5e-6) on a single A100-40GB GPU for 18 epochs.
The baseline is viXTTS, which is fine-tuned from pretrained XTTS-v2 on the viVoice dataset.
Evaluation uses four test sets: PhoAudiobook seen (PAB-S), PhoAudiobook unseen (PAB-U), VIVOS (short sentences, 19 speakers), and a held-out viVoice subset. Metrics are WER (via PhoWhisper-large), Mel-Cepstral Distortion (MCD), F0-RMSE (via Amphion), MOS, and SMOS (10–20 native-speaker listeners, blind).
Key Results
Results are from Table 2. XTTS-v2PAB consistently outperforms the viXTTS baseline across all four test sets on most metrics. On the viVoice test set: XTTS-v2PAB WER = 8.32 vs. viXTTS WER = 12.54; XTTS-v2PAB SMOS = 3.39 vs. viXTTS SMOS = 3.11; XTTS-v2PAB F0-RMSE = 228.81 vs. viXTTS 238.05. On PAB-S: MOS 4.20 (XTTS-v2PAB) vs. 4.05 (viXTTS), SMOS 3.55 vs. 2.88.
However, on the VIVOS short-sentence test set, XTTS-v2PAB and viXTTS perform markedly worse than VALL-EPAB and VoiceCraftPAB. XTTS-v2PAB achieves WER = 9.85 on VIVOS, while VALL-EPAB reaches 10.13 and VoiceCraftPAB 10.27 — but MOS for XTTS-v2PAB on VIVOS drops sharply to 2.79, versus 3.44 (VALL-EPAB) and 3.85 (VoiceCraftPAB). Manual inspection reveals that XTTS-v2-based models generate rambling or redundant speech on short inputs, which the authors attribute to an architectural issue in XTTS-v2 rather than a data problem. Comparisons are conducted under fair conditions; viXTTS on viVoice is evaluated on its own training distribution, yet still underperforms XTTS-v2PAB on that set.
Novelty Assessment
The primary contribution is a new data resource (PhoAudiobook) rather than a novel architecture or training method. The pipeline is an application of existing tools (Whisper, wav2vec2, mbart, Demucs, sox) and the three TTS models are fine-tuned from public checkpoints. The novelty lies in: (1) the dataset’s combination of scale (~941h), long-form segments (10–20s), explicit speaker identity, high SI-SNR, and normalized transcripts for Vietnamese, (2) the systematic cross-model benchmark in a low-resource setting, and (3) the empirical finding that model architecture interacts strongly with input length — a practically important finding for Vietnamese TTS deployment. The contribution is primarily data-scale and empirical rather than architectural.
Field Significance
Moderate — PhoAudiobook provides the first large-scale Vietnamese corpus with explicit speaker identity, long-form segments, and normalized transcripts, filling a gap that previously prevented codec LM architectures from being applied to the language. The cross-model benchmark contributes comparative evidence about how existing zero-shot TTS architectures handle tonal, low-resource languages, and the finding that XTTS-v2 degrades on short sentences can inform model selection for Vietnamese deployment.
Claims
- Long-form training audio (10–20 seconds per segment) with explicit speaker identities improves zero-shot TTS quality for low-resource tonal languages compared to training on short-segment corpora. (§3.1, §4)
- Multilingual voice-cloning models such as XTTS-v2 exhibit architectural failure modes on short input sequences that are not corrected by data augmentation with short clips. (§4)
- Autoregressive codec language models (VALL-E, VoiceCraft) generalize better than Tortoise-based models to short-sentence inputs in low-resource language fine-tuning. (§4, Table 2)
- A dataset curation pipeline based on dual-ASR agreement filtering produces higher-quality transcriptions for audiobook audio than single-model transcription alone, enabling more reliable TTS training. (§2.1)
Limitations and Open Questions
The paper does not evaluate code-switching scenarios (mixed Vietnamese-English input), which is relevant in practice. The dataset is audiobook domain only, so speaking style coverage is narrower than general-purpose datasets. All models are fine-tuned rather than trained from scratch, which means performance is bounded by the pre-trained model’s multilingual capacity. The architecture issue observed with XTTS-v2 on short sentences is identified but not resolved. The dataset is released for non-commercial use only, which limits industrial adoption. It is also unclear how the system handles tonal phonology beyond phonemizer outputs, and no ablation on augmented data proportion is presented.
Wiki Connections
This paper advances the zero-shot-tts paradigm in a low-resource language setting. The use of EnCodec discrete tokens in VALL-E connects to neural-codec and autoregressive-codec-tts. The multi-model comparative study contributes to evaluation-metrics and multilingual-tts. Fine-tuning pretrained models to new languages relates to speaker-adaptation.